<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
In the branch I introduce at https: //gitlab. com/petsc/petsc/-/merge_requests/7658 On src/ksp/ksp/tutorials/ex45. c on a good memory bandwidth Intel system using the options OMP_NUM_THREADS=1-64 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner0h6f7gi" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner0h6f7gi" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner0h6f7gi" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner0h6f7gi" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html><head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner0h6f7gi { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton0h6f7gi:hover, .pfptPrimaryButton0h6f7gi:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton0h6f7gi:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> In the branch I introduce at <a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7658__;!!G_uCfscf7eWS!bDIUaHqh4wl1qOCJcAguXJHWFoOm4VTVCLhaVCpC9UKNvTShW_jjtq_DiWBWpbj0cSsF0wPyRAJm2hzppiiRwaE$">https://gitlab.com/petsc/petsc/-/merge_requests/7658</a><div><br></div><div>On src/ksp/ksp/tutorials/ex45.c on a good memory bandwidth Intel system using the options </div><div><br></div><div>OMP_NUM_THREADS=1-64 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view </div><div><br></div><div>I get pretty good performance improvement with speedups of 24 MatMult, 15 VecMdot, 27 VecMAXPY</div><div><br></div><div>I also get good speedups on an M2 Mac.</div><div><br></div><div>I am trying, slowly, to improve the experience of OpenMP users of PETSc (with or without MPI).</div><div><br></div><div><div>$ OMP_NUM_THREADS=1 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 9.6804e-01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 28 17 0 0 0 28 17 0 0 0 1263</div><div>VecMDot 100 1.0 8.6170e-01 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 38 0 0 0 25 38 0 0 0 3072</div><div>VecMAXPY 104 1.0 9.4225e-01 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 27 40 0 0 0 27 40 0 0 0 3003</div><div><br></div><div>$ OMP_NUM_THREADS=2 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 5.2966e-01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 17 0 0 0 25 17 0 0 0 2308</div><div>VecMDot 100 1.0 4.8504e-01 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 23 38 0 0 0 23 38 0 0 0 5457</div><div>VecMAXPY 104 1.0 4.9861e-01 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 23 40 0 0 0 23 40 0 0 0 5674</div><div><br></div><div><br></div><div>$ OMP_NUM_THREADS=4 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 2.9545e-01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 20 17 0 0 0 20 17 0 0 0 4137</div><div>VecMDot 100 1.0 2.7871e-01 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 18 38 0 0 0 18 38 0 0 0 9496</div><div>VecMAXPY 104 1.0 2.9648e-01 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 20 40 0 0 0 20 40 0 0 0 9543</div><div><br></div><div>$ OMP_NUM_THREADS=8 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 1.6195e-01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 17 0 0 0 15 17 0 0 0 7547</div><div>VecMDot 100 1.0 1.4976e-01 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 14 38 0 0 0 14 38 0 0 0 17674</div><div>VecMAXPY 104 1.0 1.5640e-01 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 14 40 0 0 0 14 40 0 0 0 18090</div><div><br></div><div>$ OMP_NUM_THREADS=16 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 9.0148e-02 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 11 17 0 0 0 11 17 0 0 0 13558</div><div>VecMDot 100 1.0 6.9346e-02 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 8 38 0 0 0 8 38 0 0 0 38167</div><div>VecMAXPY 104 1.0 6.9274e-02 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 8 40 0 0 0 8 40 0 0 0 40842</div><div><br></div><div>$ OMP_NUM_THREADS=24 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 7.4300e-02 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 8 17 0 0 0 8 17 0 0 0 16450</div><div>VecMDot 100 1.0 5.7918e-02 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 7 38 0 0 0 7 38 0 0 0 45698</div><div>VecMAXPY 104 1.0 5.2788e-02 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 6 40 0 0 0 6 40 0 0 0 53597</div><div><br></div><div>$ OMP_NUM_THREADS=64 ./ex45 -pc_type none -da_refine 4 -ksp_max_it 100 -ksp_monitor -log_view -ksp_gmres_preallocate true -blas_view -vec_maxpy_use_gemv -ksp_converged_reason -ksp_view | egrep -e MatMult -e MAXPY -e MDot</div><div>MatMult 104 1.0 6.4348e-02 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 8 17 0 0 0 8 17 0 0 0 18994</div><div>VecMDot 100 1.0 5.8682e-02 1.0 2.65e+09 1.0 0.0e+00 0.0e+00 0.0e+00 8 38 0 0 0 8 38 0 0 0 45103</div><div>VecMAXPY 104 1.0 3.4760e-02 1.0 2.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 4 40 0 0 0 4 40 0 0 0 81394</div><div><br></div><div><br></div><div><br><blockquote type="cite"><div>On Jun 28, 2024, at 2:33 PM, Junchao Zhang <junchao.zhang@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div dir="ltr" id="pfptBannerqfqjdw6" style="color-scheme: revert !important; writing-mode: revert !important; -webkit-ruby-position: revert !important; color: revert !important; font-family: revert !important; font-feature-settings: revert !important; font-kerning: revert !important; font-optical-sizing: revert !important; font-palette: revert !important; font-size: revert !important; font-size-adjust: revert !important; font-stretch: revert !important; font-style: revert !important; font-synthesis-small-caps: revert !important; font-synthesis-style: revert !important; font-synthesis-weight: revert !important; font-variant-alternates: revert !important; font-variant-caps: revert !important; font-variant-east-asian: revert !important; font-variant-emoji: revert !important; font-variant-ligatures: revert !important; font-variant-numeric: revert !important; font-variant-position: revert !important; font-variation-settings: revert !important; font-weight: revert !important; text-orientation: revert !important; text-rendering: revert !important; zoom: revert !important; -webkit-font-smoothing: revert !important; -webkit-locale: revert !important; -webkit-text-size-adjust: revert !important; -webkit-text-zoom: revert !important; letter-spacing: revert !important; text-autospace: revert !important; text-spacing-trim: revert !important; accent-color: revert !important; align-content: revert !important; align-items: revert !important; align-self: revert !important; alignment-baseline: revert !important; alt: revert !important; animation-composition: revert !important; animation: revert !important; animation-timeline: revert !important; appearance: revert !important; aspect-ratio: revert !important; backface-visibility: revert !important; background-attachment: revert !important; background-blend-mode: revert !important; background-image: revert !important; background-position: revert !important; background-repeat: revert !important; background-size: revert !important; baseline-shift: revert !important; block-step-insert: revert !important; block-step-size: revert !important; border-collapse: revert !important; border-image: revert !important; box-sizing: revert !important; break-after: revert !important; break-before: revert !important; break-inside: revert !important; buffered-rendering: revert !important; caption-side: revert !important; caret-color: revert !important; clear: revert !important; clip: revert !important; clip-path: revert !important; clip-rule: revert !important; color-interpolation: revert !important; color-interpolation-filters: revert !important; column-count: revert !important; column-fill: revert !important; column-gap: revert !important; column-rule-color: revert !important; column-rule-style: revert !important; column-rule-width: revert !important; column-span: revert !important; column-width: revert !important; contain: revert !important; container-name: revert !important; container-type: revert !important; content: revert !important; content-visibility: revert !important; counter-increment: revert !important; counter-reset: revert !important; counter-set: revert !important; cursor: revert !important; cx: revert !important; cy: revert !important; dominant-baseline: revert !important; empty-cells: revert !important; fill: revert !important; fill-opacity: revert !important; fill-rule: revert !important; filter: revert !important; flex: revert !important; flex-flow: revert !important; float: revert !important; flood-color: revert !important; flood-opacity: revert !important; glyph-orientation-horizontal: revert !important; glyph-orientation-vertical: revert !important; grid: revert !important; grid-column-end: revert !important; grid-column-start: revert !important; grid-row-end: revert !important; grid-row-start: revert !important; hanging-punctuation: revert !important; hyphenate-character: revert !important; hyphens: revert !important; image-orientation: revert !important; image-rendering: revert !important; input-security: revert !important; isolation: revert !important; justify-content: revert !important; justify-items: revert !important; justify-self: revert !important; kerning: revert !important; lighting-color: revert !important; line-break: revert !important; line-height: revert !important; list-style: revert !important; margin-trim: revert !important; marker: revert !important; mask: revert !important; mask-size: revert !important; mask-type: revert !important; masonry-auto-flow: revert !important; math-style: revert !important; mix-blend-mode: revert !important; object-fit: revert !important; object-position: revert !important; offset-anchor: revert !important; offset-distance: revert !important; offset-path: revert !important; offset-position: revert !important; offset-rotate: revert !important; order: revert !important; orphans: revert !important; outline: revert !important; outline-offset: revert !important; overflow-anchor: revert !important; overflow-wrap: revert !important; overflow: revert !important; page: revert !important; paint-order: revert !important; perspective: revert !important; perspective-origin: revert !important; pointer-events: revert !important; position: revert !important; print-color-adjust: revert !important; quotes: revert !important; r: revert !important; resize: revert !important; rotate: revert !important; row-gap: revert !important; rx: revert !important; ry: revert !important; scale: revert !important; scroll-behavior: revert !important; scroll-snap-align: revert !important; scroll-snap-stop: revert !important; scroll-snap-type: revert !important; scroll-timeline: revert !important; scrollbar-color: revert !important; scrollbar-gutter: revert !important; scrollbar-width: revert !important; shape-image-threshold: revert !important; shape-margin: revert !important; shape-outside: revert !important; shape-rendering: revert !important; size: revert !important; speak-as: revert !important; stop-color: revert !important; stop-opacity: revert !important; stroke: revert !important; stroke-color: revert !important; stroke-dasharray: revert !important; stroke-dashoffset: revert !important; stroke-linecap: revert !important; stroke-linejoin: revert !important; stroke-miterlimit: revert !important; stroke-opacity: revert !important; stroke-width: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: left !important; text-align-last: revert !important; text-anchor: revert !important; text-box-edge: revert !important; text-box-trim: revert !important; text-decoration-color: revert !important; text-decoration: revert !important; text-decoration-skip-ink: revert !important; text-decoration-style: revert !important; text-decoration-thickness: revert !important; text-emphasis-color: revert !important; text-emphasis-position: revert !important; text-emphasis-style: revert !important; text-group-align: revert !important; text-indent: revert !important; text-justify: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-underline-offset: revert !important; text-underline-position: revert !important; text-wrap: revert !important; touch-action: revert !important; transform: revert !important; transform-box: revert !important; transform-origin: revert !important; transform-style: revert !important; transition: revert !important; translate: revert !important; vector-effect: revert !important; vertical-align: revert !important; view-timeline: revert !important; view-transition-name: revert !important; white-space: revert !important; widows: revert !important; will-change: revert !important; word-break: revert !important; x: revert !important; y: revert !important; z-index: revert !important; -apple-color-filter: revert !important; -apple-pay-button-style: revert !important; -apple-pay-button-type: revert !important; -internal-text-autosizing-status: revert !important; border-spacing: revert !important; -webkit-box-align: revert !important; -webkit-box-decoration-break: revert !important; -webkit-box-direction: revert !important; -webkit-box-flex: revert !important; -webkit-box-flex-group: revert !important; -webkit-box-lines: revert !important; -webkit-box-ordinal-group: revert !important; -webkit-box-orient: revert !important; -webkit-box-pack: revert !important; -webkit-box-reflect: revert !important; -webkit-column-axis: revert !important; -webkit-column-progression: revert !important; -webkit-cursor-visibility: revert !important; -webkit-font-size-delta: revert !important; -webkit-hyphenate-limit-after: revert !important; -webkit-hyphenate-limit-before: revert !important; -webkit-hyphenate-limit-lines: revert !important; -webkit-initial-letter: revert !important; -webkit-line-align: revert !important; -webkit-line-box-contain: revert !important; -webkit-line-clamp: revert !important; -webkit-line-grid: revert !important; -webkit-line-snap: revert !important; -webkit-marquee-direction: revert !important; -webkit-marquee-increment: revert !important; -webkit-marquee-repetition: revert !important; -webkit-marquee-speed: revert !important; -webkit-marquee-style: revert !important; -webkit-nbsp-mode: revert !important; -webkit-rtl-ordering: revert !important; -webkit-text-decorations-in-effect: revert !important; -webkit-text-fill-color: revert !important; -webkit-text-security: revert !important; -webkit-text-stroke-color: revert !important; -webkit-text-stroke-width: revert !important; -webkit-user-drag: revert !important; -webkit-user-modify: revert !important; -webkit-user-select: revert !important; word-spacing: revert !important; backdrop-filter: revert !important; background-clip: revert !important; background-origin: revert !important; box-shadow: revert !important; mask-composite: revert !important; mask-mode: revert !important; text-combine-upright: revert !important; -webkit-backdrop-filter: revert !important; -webkit-background-clip: revert !important; -webkit-background-origin: revert !important; -webkit-box-shadow: revert !important; -webkit-mask: revert !important; -webkit-mask-composite: revert !important; -webkit-mask-source-type: revert !important; -webkit-text-combine: revert !important; display: block !important; visibility: revert !important; opacity: revert !important; background-color: rgb(208, 216, 220) !important; block-size: revert !important; border-block: revert !important; border-bottom-color: revert !important; border-bottom-style: revert !important; border-bottom-width: revert !important; border-end-end-radius: revert !important; border-end-start-radius: revert !important; border-inline: revert !important; border-left-color: revert !important; border-left-style: revert !important; border-left-width: revert !important; border-right-color: revert !important; border-right-style: revert !important; border-right-width: revert !important; border-start-end-radius: revert !important; border-start-start-radius: revert !important; inset: revert !important; contain-intrinsic-block-size: revert !important; contain-intrinsic-size: revert !important; contain-intrinsic-inline-size: revert !important; height: revert !important; inline-size: revert !important; inset-block: revert !important; inset-inline: revert !important; margin-block: revert !important; margin-inline: revert !important; max-block-size: revert !important; max-height: revert !important; max-inline-size: revert !important; max-width: revert !important; min-block-size: revert !important; min-height: revert !important; min-inline-size: revert !important; overscroll-behavior-block: revert !important; overscroll-behavior-inline: revert !important; overscroll-behavior: revert !important; padding-block: revert !important; padding-inline: revert !important; scroll-margin-block: revert !important; scroll-margin: revert !important; scroll-margin-inline: revert !important; scroll-padding-block: revert !important; scroll-padding: revert !important; scroll-padding-inline: revert !important; width: revert !important; margin: 16px 0px !important; padding: 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; border-top-width: 4px !important; border-top-style: solid !important; border-top-color: rgb(144, 164, 174) !important;"><div id="pfptBannerqfqjdw6" style="color-scheme: unset !important; writing-mode: unset !important; -webkit-ruby-position: unset !important; color: unset !important; font-family: unset !important; font-feature-settings: unset !important; font-kerning: unset !important; font-optical-sizing: unset !important; font-palette: unset !important; font-size: unset !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-synthesis-small-caps: unset !important; font-synthesis-style: unset !important; font-synthesis-weight: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-emoji: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-variant-position: unset !important; font-variation-settings: unset !important; font-weight: unset !important; text-orientation: unset !important; text-rendering: unset !important; zoom: unset !important; -webkit-font-smoothing: unset !important; -webkit-locale: unset !important; -webkit-text-size-adjust: unset !important; -webkit-text-zoom: unset !important; letter-spacing: unset !important; text-autospace: unset !important; text-spacing-trim: unset !important; accent-color: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; alignment-baseline: unset !important; alt: unset !important; animation-composition: unset !important; animation: unset !important; animation-timeline: unset !important; appearance: unset !important; aspect-ratio: unset !important; backface-visibility: unset !important; background: unset !important; background-blend-mode: unset !important; baseline-shift: unset !important; block-step-insert: unset !important; block-step-size: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; buffered-rendering: unset !important; caption-side: unset !important; caret-color: unset !important; clear: unset !important; clip: unset !important; clip-path: unset !important; clip-rule: unset !important; color-interpolation: unset !important; color-interpolation-filters: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; contain: unset !important; container-name: unset !important; container-type: unset !important; content: unset !important; content-visibility: unset !important; counter-increment: unset !important; counter-reset: unset !important; counter-set: unset !important; cursor: unset !important; cx: unset !important; cy: unset !important; dominant-baseline: unset !important; empty-cells: unset !important; fill: unset !important; fill-opacity: unset !important; fill-rule: unset !important; filter: unset !important; flex: unset !important; flex-flow: unset !important; float: left !important; flood-color: unset !important; flood-opacity: unset !important; glyph-orientation-horizontal: unset !important; glyph-orientation-vertical: unset !important; grid: unset !important; grid-column-end: unset !important; grid-column-start: unset !important; grid-row-end: unset !important; grid-row-start: unset !important; hanging-punctuation: unset !important; hyphenate-character: unset !important; hyphens: unset !important; image-orientation: unset !important; image-rendering: unset !important; input-security: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; justify-self: unset !important; kerning: unset !important; lighting-color: unset !important; line-break: unset !important; line-height: unset !important; list-style: unset !important; margin-trim: unset !important; marker: unset !important; mask: unset !important; mask-size: unset !important; mask-type: unset !important; masonry-auto-flow: unset !important; math-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; offset-anchor: unset !important; offset-distance: unset !important; offset-path: unset !important; offset-position: unset !important; offset-rotate: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; outline-offset: unset !important; overflow-anchor: unset !important; overflow-wrap: unset !important; overflow: unset !important; page: unset !important; paint-order: unset !important; perspective: unset !important; perspective-origin: unset !important; pointer-events: unset !important; position: unset !important; print-color-adjust: unset !important; quotes: unset !important; r: unset !important; resize: unset !important; rotate: unset !important; row-gap: unset !important; rx: unset !important; ry: unset !important; scale: unset !important; scroll-behavior: unset !important; scroll-snap-align: unset !important; scroll-snap-stop: unset !important; scroll-snap-type: unset !important; scroll-timeline: unset !important; scrollbar-color: unset !important; scrollbar-gutter: unset !important; scrollbar-width: unset !important; shape-image-threshold: unset !important; shape-margin: unset !important; shape-outside: unset !important; shape-rendering: unset !important; size: unset !important; speak-as: unset !important; stop-color: unset !important; stop-opacity: unset !important; stroke: unset !important; stroke-color: unset !important; stroke-dasharray: unset !important; stroke-dashoffset: unset !important; stroke-linecap: unset !important; stroke-linejoin: unset !important; stroke-miterlimit: unset !important; stroke-opacity: unset !important; stroke-width: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-align-last: unset !important; text-anchor: unset !important; text-box-edge: unset !important; text-box-trim: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-skip-ink: unset !important; text-decoration-style: unset !important; text-decoration-thickness: unset !important; text-emphasis-color: unset !important; text-emphasis-position: unset !important; text-emphasis-style: unset !important; text-group-align: unset !important; text-indent: unset !important; text-justify: unset !important; text-overflow: unset !important; text-shadow: unset !important; text-transform: unset !important; text-underline-offset: unset !important; text-underline-position: unset !important; text-wrap: unset !important; touch-action: unset !important; transform: unset !important; transform-box: unset !important; transform-origin: unset !important; transform-style: unset !important; transition: unset !important; translate: unset !important; vector-effect: unset !important; vertical-align: unset !important; view-timeline: unset !important; view-transition-name: unset !important; white-space: unset !important; widows: unset !important; will-change: unset !important; word-break: unset !important; x: unset !important; y: unset !important; z-index: unset !important; -apple-color-filter: unset !important; -apple-pay-button-style: unset !important; -apple-pay-button-type: unset !important; -internal-text-autosizing-status: unset !important; border-spacing: unset !important; -webkit-box-align: unset !important; -webkit-box-decoration-break: unset !important; -webkit-box-direction: unset !important; -webkit-box-flex: unset !important; -webkit-box-flex-group: unset !important; -webkit-box-lines: unset !important; -webkit-box-ordinal-group: unset !important; -webkit-box-orient: unset !important; -webkit-box-pack: unset !important; -webkit-box-reflect: unset !important; -webkit-column-axis: unset !important; -webkit-column-progression: unset !important; -webkit-cursor-visibility: unset !important; -webkit-font-size-delta: unset !important; -webkit-hyphenate-limit-after: unset !important; -webkit-hyphenate-limit-before: unset !important; -webkit-hyphenate-limit-lines: unset !important; -webkit-initial-letter: unset !important; -webkit-line-align: unset !important; -webkit-line-box-contain: unset !important; -webkit-line-clamp: unset !important; -webkit-line-grid: unset !important; -webkit-line-snap: unset !important; -webkit-marquee-direction: unset !important; -webkit-marquee-increment: unset !important; -webkit-marquee-repetition: unset !important; -webkit-marquee-speed: unset !important; -webkit-marquee-style: unset !important; -webkit-nbsp-mode: unset !important; -webkit-rtl-ordering: unset !important; -webkit-text-decorations-in-effect: unset !important; -webkit-text-fill-color: unset !important; -webkit-text-security: unset !important; -webkit-text-stroke-color: unset !important; -webkit-text-stroke-width: unset !important; -webkit-user-drag: unset !important; -webkit-user-modify: unset !important; -webkit-user-select: unset !important; word-spacing: unset !important; backdrop-filter: unset !important; box-shadow: unset !important; mask-composite: unset !important; mask-mode: unset !important; text-combine-upright: unset !important; -webkit-backdrop-filter: unset !important; -webkit-background-clip: unset !important; -webkit-background-origin: unset !important; -webkit-box-shadow: unset !important; -webkit-mask: unset !important; -webkit-mask-composite: unset !important; -webkit-mask-source-type: unset !important; -webkit-text-combine: unset !important; display: block !important; visibility: unset !important; opacity: unset !important; block-size: unset !important; border-block: unset !important; border-radius: unset !important; border-end-end-radius: unset !important; border-end-start-radius: unset !important; border-inline: unset !important; border-start-end-radius: unset !important; border-start-start-radius: unset !important; inset: unset !important; contain-intrinsic-block-size: unset !important; contain-intrinsic-size: unset !important; contain-intrinsic-inline-size: unset !important; height: unset !important; inline-size: unset !important; inset-block: unset !important; inset-inline: unset !important; margin-block: unset !important; margin-inline: unset !important; max-block-size: unset !important; max-height: unset !important; max-inline-size: unset !important; min-block-size: unset !important; min-height: unset !important; min-inline-size: unset !important; min-width: unset !important; overscroll-behavior-block: unset !important; overscroll-behavior-inline: unset !important; overscroll-behavior: unset !important; padding-block: unset !important; padding: unset !important; padding-inline: unset !important; scroll-margin-block: unset !important; scroll-margin: unset !important; scroll-margin-inline: unset !important; scroll-padding-block: unset !important; scroll-padding: unset !important; scroll-padding-inline: unset !important; width: unset !important; margin: 0px 0px 1px !important; max-width: 600px !important;"><div id="pfptBannerqfqjdw6" style="color-scheme: unset !important; writing-mode: unset !important; -webkit-ruby-position: unset !important; font-family: Arial, sans-serif !important; font-feature-settings: unset !important; font-kerning: unset !important; font-optical-sizing: unset !important; font-palette: unset !important; font-size: 14px !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-synthesis-small-caps: unset !important; font-synthesis-style: unset !important; font-synthesis-weight: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-variant-position: unset !important; font-variation-settings: unset !important; font-weight: bold !important; text-orientation: unset !important; text-rendering: unset !important; zoom: unset !important; -webkit-font-smoothing: unset !important; -webkit-locale: unset !important; -webkit-text-zoom: unset !important; letter-spacing: unset !important; accent-color: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; alignment-baseline: unset !important; alt: unset !important; animation-composition: unset !important; animation: unset !important; appearance: unset !important; aspect-ratio: unset !important; backface-visibility: unset !important; background-attachment: unset !important; background-blend-mode: unset !important; background-image: unset !important; background-position: unset !important; background-repeat: unset !important; background-size: unset !important; baseline-shift: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; buffered-rendering: unset !important; caption-side: unset !important; caret-color: unset !important; clear: unset !important; clip: unset !important; clip-path: unset !important; clip-rule: unset !important; color-interpolation: unset !important; color-interpolation-filters: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; contain: unset !important; container-name: unset !important; container-type: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; counter-set: unset !important; cursor: unset !important; cx: unset !important; cy: unset !important; dominant-baseline: unset !important; empty-cells: unset !important; fill: unset !important; fill-opacity: unset !important; fill-rule: unset !important; filter: unset !important; flex: unset !important; flex-flow: unset !important; float: unset !important; flood-color: unset !important; flood-opacity: unset !important; glyph-orientation-horizontal: unset !important; glyph-orientation-vertical: unset !important; grid: unset !important; grid-column-end: unset !important; grid-column-start: unset !important; grid-row-end: unset !important; grid-row-start: unset !important; hanging-punctuation: unset !important; hyphenate-character: unset !important; hyphens: unset !important; image-orientation: unset !important; image-rendering: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; justify-self: unset !important; kerning: unset !important; lighting-color: unset !important; line-break: unset !important; line-height: 18px !important; list-style: unset !important; margin-trim: unset !important; marker: unset !important; mask: unset !important; mask-type: unset !important; math-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; offset-anchor: unset !important; offset-distance: unset !important; offset-path: unset !important; offset-position: unset !important; offset-rotate: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; outline-offset: unset !important; overflow-wrap: unset !important; overflow: unset !important; page: unset !important; paint-order: unset !important; perspective: unset !important; perspective-origin: unset !important; pointer-events: unset !important; position: unset !important; print-color-adjust: unset !important; quotes: unset !important; r: unset !important; resize: unset !important; rotate: unset !important; row-gap: unset !important; rx: unset !important; ry: unset !important; scale: unset !important; scroll-behavior: unset !important; scroll-snap-align: unset !important; scroll-snap-stop: unset !important; scroll-snap-type: unset !important; shape-image-threshold: unset !important; shape-margin: unset !important; shape-outside: unset !important; shape-rendering: unset !important; size: unset !important; speak-as: unset !important; stop-color: unset !important; stop-opacity: unset !important; stroke: unset !important; stroke-color: unset !important; stroke-dasharray: unset !important; stroke-dashoffset: unset !important; stroke-linecap: unset !important; stroke-linejoin: unset !important; stroke-miterlimit: unset !important; stroke-opacity: unset !important; stroke-width: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-align-last: unset !important; text-anchor: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-skip-ink: unset !important; text-decoration-style: unset !important; text-decoration-thickness: unset !important; text-emphasis-color: unset !important; text-emphasis-position: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-shadow: unset !important; text-transform: unset !important; text-underline-offset: unset !important; text-underline-position: unset !important; text-wrap: unset !important; touch-action: unset !important; transform: unset !important; transform-box: unset !important; transform-origin: unset !important; transform-style: unset !important; transition: unset !important; translate: unset !important; vector-effect: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; will-change: unset !important; word-break: unset !important; x: unset !important; y: unset !important; z-index: unset !important; -apple-color-filter: unset !important; -apple-pay-button-style: unset !important; -apple-pay-button-type: unset !important; border-spacing: unset !important; -webkit-box-align: unset !important; -webkit-box-decoration-break: unset !important; -webkit-box-direction: unset !important; -webkit-box-flex: unset !important; -webkit-box-flex-group: unset !important; -webkit-box-lines: unset !important; -webkit-box-ordinal-group: unset !important; -webkit-box-orient: unset !important; -webkit-box-pack: unset !important; -webkit-box-reflect: unset !important; -webkit-column-axis: unset !important; -webkit-column-progression: unset !important; -webkit-cursor-visibility: unset !important; -webkit-hyphenate-limit-after: unset !important; -webkit-hyphenate-limit-before: unset !important; -webkit-hyphenate-limit-lines: unset !important; -webkit-initial-letter: unset !important; -webkit-line-align: unset !important; -webkit-line-box-contain: unset !important; -webkit-line-clamp: unset !important; -webkit-line-grid: unset !important; -webkit-line-snap: unset !important; -webkit-nbsp-mode: unset !important; -webkit-rtl-ordering: unset !important; -webkit-text-decorations-in-effect: unset !important; -webkit-text-fill-color: unset !important; -webkit-text-security: unset !important; -webkit-text-stroke-color: unset !important; -webkit-text-stroke-width: unset !important; -webkit-user-drag: unset !important; -webkit-user-modify: unset !important; -webkit-user-select: unset !important; word-spacing: unset !important; background-clip: unset !important; background-origin: unset !important; box-shadow: unset !important; mask-composite: unset !important; mask-mode: unset !important; text-combine-upright: unset !important; -webkit-backdrop-filter: unset !important; -webkit-background-clip: unset !important; -webkit-background-origin: unset !important; -webkit-box-shadow: unset !important; mask-size: unset !important; -webkit-mask: unset !important; -webkit-mask-composite: unset !important; -webkit-mask-source-type: unset !important; -webkit-text-combine: unset !important; display: block !important; visibility: visible !important; opacity: unset !important; background-color: rgb(208, 216, 220) !important; block-size: unset !important; border-block: unset !important; border-radius: unset !important; border-end-end-radius: unset !important; border-end-start-radius: unset !important; border-inline: unset !important; border-start-end-radius: unset !important; border-start-start-radius: unset !important; inset: unset !important; contain-intrinsic-block-size: unset !important; contain-intrinsic-size: unset !important; contain-intrinsic-inline-size: unset !important; height: unset !important; inline-size: unset !important; inset-block: unset !important; inset-inline: unset !important; margin-block: unset !important; margin: unset !important; margin-inline: unset !important; max-block-size: unset !important; max-height: unset !important; max-inline-size: unset !important; max-width: unset !important; min-block-size: unset !important; min-height: unset !important; min-inline-size: unset !important; min-width: unset !important; overscroll-behavior-block: unset !important; overscroll-behavior-inline: unset !important; overscroll-behavior: unset !important; padding-block: unset !important; padding: unset !important; padding-inline: unset !important; scroll-margin-block: unset !important; scroll-margin: unset !important; scroll-margin-inline: unset !important; scroll-padding-block: unset !important; scroll-padding: unset !important; scroll-padding-inline: unset !important; width: unset !important;">This Message Is From an External Sender</div><div id="pfptBannerqfqjdw6" style="color-scheme: unset !important; writing-mode: unset !important; -webkit-ruby-position: unset !important; font-family: Arial, sans-serif !important; font-feature-settings: unset !important; font-kerning: unset !important; font-optical-sizing: unset !important; font-palette: unset !important; font-size: 12px !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-synthesis-small-caps: unset !important; font-synthesis-style: unset !important; font-synthesis-weight: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-variant-position: unset !important; font-variation-settings: unset !important; font-weight: unset !important; text-orientation: unset !important; text-rendering: unset !important; zoom: unset !important; -webkit-font-smoothing: unset !important; -webkit-locale: unset !important; -webkit-text-zoom: unset !important; letter-spacing: unset !important; accent-color: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; alignment-baseline: unset !important; alt: unset !important; animation-composition: unset !important; animation: unset !important; appearance: unset !important; aspect-ratio: unset !important; backface-visibility: unset !important; background-attachment: unset !important; background-blend-mode: unset !important; background-image: unset !important; background-position: unset !important; background-repeat: unset !important; background-size: unset !important; baseline-shift: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; buffered-rendering: unset !important; caption-side: unset !important; caret-color: unset !important; clear: unset !important; clip: unset !important; clip-path: unset !important; clip-rule: unset !important; color-interpolation: unset !important; color-interpolation-filters: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; contain: unset !important; container-name: unset !important; container-type: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; counter-set: unset !important; cursor: unset !important; cx: unset !important; cy: unset !important; dominant-baseline: unset !important; empty-cells: unset !important; fill: unset !important; fill-opacity: unset !important; fill-rule: unset !important; filter: unset !important; flex: unset !important; flex-flow: unset !important; float: unset !important; flood-color: unset !important; flood-opacity: unset !important; glyph-orientation-horizontal: unset !important; glyph-orientation-vertical: unset !important; grid: unset !important; grid-column-end: unset !important; grid-column-start: unset !important; grid-row-end: unset !important; grid-row-start: unset !important; hanging-punctuation: unset !important; hyphenate-character: unset !important; hyphens: unset !important; image-orientation: unset !important; image-rendering: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; justify-self: unset !important; kerning: unset !important; lighting-color: unset !important; line-break: unset !important; line-height: 18px !important; list-style: unset !important; margin-trim: unset !important; marker: unset !important; mask: unset !important; mask-type: unset !important; math-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; offset-anchor: unset !important; offset-distance: unset !important; offset-path: unset !important; offset-position: unset !important; offset-rotate: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; outline-offset: unset !important; overflow-wrap: unset !important; overflow: unset !important; page: unset !important; paint-order: unset !important; perspective: unset !important; perspective-origin: unset !important; pointer-events: unset !important; position: unset !important; print-color-adjust: unset !important; quotes: unset !important; r: unset !important; resize: unset !important; rotate: unset !important; row-gap: unset !important; rx: unset !important; ry: unset !important; scale: unset !important; scroll-behavior: unset !important; scroll-snap-align: unset !important; scroll-snap-stop: unset !important; scroll-snap-type: unset !important; shape-image-threshold: unset !important; shape-margin: unset !important; shape-outside: unset !important; shape-rendering: unset !important; size: unset !important; speak-as: unset !important; stop-color: unset !important; stop-opacity: unset !important; stroke: unset !important; stroke-color: unset !important; stroke-dasharray: unset !important; stroke-dashoffset: unset !important; stroke-linecap: unset !important; stroke-linejoin: unset !important; stroke-miterlimit: unset !important; stroke-opacity: unset !important; stroke-width: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-align-last: unset !important; text-anchor: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-skip-ink: unset !important; text-decoration-style: unset !important; text-decoration-thickness: unset !important; text-emphasis-color: unset !important; text-emphasis-position: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-shadow: unset !important; text-transform: unset !important; text-underline-offset: unset !important; text-underline-position: unset !important; text-wrap: unset !important; touch-action: unset !important; transform: unset !important; transform-box: unset !important; transform-origin: unset !important; transform-style: unset !important; transition: unset !important; translate: unset !important; vector-effect: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; will-change: unset !important; word-break: unset !important; x: unset !important; y: unset !important; z-index: unset !important; -apple-color-filter: unset !important; -apple-pay-button-style: unset !important; -apple-pay-button-type: unset !important; border-spacing: unset !important; -webkit-box-align: unset !important; -webkit-box-decoration-break: unset !important; -webkit-box-direction: unset !important; -webkit-box-flex: unset !important; -webkit-box-flex-group: unset !important; -webkit-box-lines: unset !important; -webkit-box-ordinal-group: unset !important; -webkit-box-orient: unset !important; -webkit-box-pack: unset !important; -webkit-box-reflect: unset !important; -webkit-column-axis: unset !important; -webkit-column-progression: unset !important; -webkit-cursor-visibility: unset !important; -webkit-hyphenate-limit-after: unset !important; -webkit-hyphenate-limit-before: unset !important; -webkit-hyphenate-limit-lines: unset !important; -webkit-initial-letter: unset !important; -webkit-line-align: unset !important; -webkit-line-box-contain: unset !important; -webkit-line-clamp: unset !important; -webkit-line-grid: unset !important; -webkit-line-snap: unset !important; -webkit-nbsp-mode: unset !important; -webkit-rtl-ordering: unset !important; -webkit-text-decorations-in-effect: unset !important; -webkit-text-fill-color: unset !important; -webkit-text-security: unset !important; -webkit-text-stroke-color: unset !important; -webkit-text-stroke-width: unset !important; -webkit-user-drag: unset !important; -webkit-user-modify: unset !important; -webkit-user-select: unset !important; word-spacing: unset !important; background-clip: unset !important; background-origin: unset !important; box-shadow: unset !important; mask-composite: unset !important; mask-mode: unset !important; text-combine-upright: unset !important; -webkit-backdrop-filter: unset !important; -webkit-background-clip: unset !important; -webkit-background-origin: unset !important; -webkit-box-shadow: unset !important; mask-size: unset !important; -webkit-mask: unset !important; -webkit-mask-composite: unset !important; -webkit-mask-source-type: unset !important; -webkit-text-combine: unset !important; display: block !important; visibility: visible !important; opacity: unset !important; background-color: rgb(208, 216, 220) !important; block-size: unset !important; border-block: unset !important; border-radius: unset !important; border-end-end-radius: unset !important; border-end-start-radius: unset !important; border-inline: unset !important; border-start-end-radius: unset !important; border-start-start-radius: unset !important; inset: unset !important; contain-intrinsic-block-size: unset !important; contain-intrinsic-size: unset !important; contain-intrinsic-inline-size: unset !important; height: unset !important; inline-size: unset !important; inset-block: unset !important; inset-inline: unset !important; margin-block: unset !important; margin-bottom: unset !important; margin-inline: unset !important; margin-left: unset !important; margin-right: unset !important; max-block-size: unset !important; max-height: unset !important; max-inline-size: unset !important; max-width: unset !important; min-block-size: unset !important; min-height: unset !important; min-inline-size: unset !important; min-width: unset !important; overscroll-behavior-block: unset !important; overscroll-behavior-inline: unset !important; overscroll-behavior: unset !important; padding-block: unset !important; padding: unset !important; padding-inline: unset !important; scroll-margin-block: unset !important; scroll-margin: unset !important; scroll-margin-inline: unset !important; scroll-padding-block: unset !important; scroll-padding: unset !important; scroll-padding-inline: unset !important; width: unset !important; margin-top: 2px !important;">This message came from outside your organization.</div></div><div style="height: 0px; clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important;"></div></div><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">OK, then you need <font> </font><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre; color: rgb(163, 21, 21);">'--with-mkl_pardiso-dir='</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre;">+os.environ[</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre; color: rgb(163, 21, 21);">'MKLROOT'</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; white-space: pre;">] </span>in petsc configure<div><font face="Menlo, Monaco, Courier New, monospace"><span style="font-size: 14px; white-space: pre;"><br></span></font><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div dir="ltr" class="gmail_attr">On Fri, Jun 28, 2024 at 1:05 PM Pierre Jolivet <<a href="mailto:pierre.jolivet@lip6.fr">pierre.jolivet@lip6.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><br id="m_-3762595706006288767lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 28 Jun 2024, at 7:20 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:</div><br><div><div dir="ltr" id="m_-3762595706006288767pfptBannerl0fkpcl" style="writing-mode: revert; color: revert; font-family: revert; font-feature-settings: revert; font-kerning: revert; font-size: revert; font-stretch: revert; font-variant-alternates: revert; font-variant-caps: revert; font-variant-east-asian: revert; font-variant-ligatures: revert; font-variant-numeric: revert; font-weight: revert; text-orientation: revert; zoom: revert; letter-spacing: revert; background-blend-mode: revert; background-image: revert; background-position: revert; background-repeat: revert; background-size: revert; border-collapse: revert; box-sizing: revert; break-after: revert; break-before: revert; break-inside: revert; caption-side: revert; clear: revert; column-width: revert; column-count: revert; column-fill: revert; column-gap: revert; column-rule-width: revert; column-rule-style: revert; column-rule-color: revert; column-span: revert; empty-cells: revert; float: revert; image-orientation: revert; isolation: revert; line-break: revert; line-height: revert; list-style: revert; mix-blend-mode: revert; object-fit: revert; object-position: revert; outline: revert; overflow: revert; quotes: revert; table-layout: revert; text-align: left; text-align-last: revert; text-decoration: revert; text-emphasis-style: revert; text-emphasis-color: revert; text-indent: revert; text-overflow: revert; text-transform: revert; text-underline-position: revert; vertical-align: revert; white-space: revert; word-break: revert; border-spacing: revert; word-spacing: revert; background-clip: revert; background-origin: revert; text-combine-upright: revert; display: block; opacity: revert; background-color: rgb(208, 216, 220); border-bottom-width: revert; border-bottom-style: revert; border-bottom-color: revert; border-left-width: revert; border-left-style: revert; border-left-color: revert; border-right-width: revert; border-right-style: revert; border-right-color: revert; height: revert; max-height: revert; max-width: revert; min-height: revert; width: revert; margin: 16px 0px; padding: 8px 16px; border-radius: 4px; min-width: 200px; border-top-width: 4px; border-top-style: solid; border-top-color: rgb(144, 164, 174);"><div id="m_-3762595706006288767pfptBannerl0fkpcl" style="writing-mode: unset; color: unset; font-family: unset; font-feature-settings: unset; font-kerning: unset; font-size: unset; font-stretch: unset; font-variant-alternates: unset; font-variant-caps: unset; font-variant-east-asian: unset; font-variant-ligatures: unset; font-variant-numeric: unset; font-weight: unset; text-orientation: unset; zoom: unset; letter-spacing: unset; background: unset; background-blend-mode: unset; border-collapse: unset; border: unset; box-sizing: unset; break-after: unset; break-before: unset; break-inside: unset; caption-side: unset; clear: unset; column-width: unset; column-count: unset; column-fill: unset; column-gap: unset; column-rule-width: unset; column-rule-style: unset; column-rule-color: unset; column-span: unset; empty-cells: unset; float: left; image-orientation: unset; isolation: unset; line-break: unset; line-height: unset; list-style: unset; mix-blend-mode: unset; object-fit: unset; object-position: unset; outline: unset; overflow: unset; quotes: unset; table-layout: unset; text-align: unset; text-align-last: unset; text-decoration: unset; text-emphasis-style: unset; text-emphasis-color: unset; text-indent: unset; text-overflow: unset; text-transform: unset; text-underline-position: unset; vertical-align: unset; white-space: unset; word-break: unset; border-spacing: unset; word-spacing: unset; text-combine-upright: unset; display: block; opacity: unset; border-radius: unset; height: unset; max-height: unset; min-height: unset; min-width: unset; padding: unset; width: unset; margin: 0px 0px 1px; max-width: 600px;"><div id="m_-3762595706006288767pfptBannerl0fkpcl" style="writing-mode: unset; font-family: Arial, sans-serif; font-feature-settings: unset; font-kerning: unset; font-size: 14px; font-stretch: unset; font-variant-alternates: unset; font-variant-caps: unset; font-variant-east-asian: unset; font-variant-ligatures: unset; font-variant-numeric: unset; font-weight: bold; text-orientation: unset; zoom: unset; letter-spacing: unset; background-blend-mode: unset; background-image: unset; background-position: unset; background-repeat: unset; background-size: unset; border-collapse: unset; border: unset; box-sizing: unset; break-after: unset; break-before: unset; break-inside: unset; caption-side: unset; clear: unset; column-width: unset; column-count: unset; column-fill: unset; column-gap: unset; column-rule-width: unset; column-rule-style: unset; column-rule-color: unset; column-span: unset; empty-cells: unset; float: unset; image-orientation: unset; isolation: unset; line-break: unset; line-height: 18px; list-style: unset; mix-blend-mode: unset; object-fit: unset; object-position: unset; outline: unset; overflow: unset; quotes: unset; table-layout: unset; text-align: unset; text-align-last: unset; text-decoration: unset; text-emphasis-style: unset; text-emphasis-color: unset; text-indent: unset; text-overflow: unset; text-transform: unset; text-underline-position: unset; vertical-align: unset; white-space: unset; word-break: unset; border-spacing: unset; word-spacing: unset; background-clip: unset; background-origin: unset; text-combine-upright: unset; display: block; opacity: unset; background-color: rgb(208, 216, 220); border-radius: unset; height: unset; margin: unset; max-height: unset; max-width: unset; min-height: unset; min-width: unset; padding: unset; width: unset;">This Message Is From an External Sender</div><div id="m_-3762595706006288767pfptBannerl0fkpcl" style="writing-mode: unset; font-family: Arial, sans-serif; font-feature-settings: unset; font-kerning: unset; font-size: 12px; font-stretch: unset; font-variant-alternates: unset; font-variant-caps: unset; font-variant-east-asian: unset; font-variant-ligatures: unset; font-variant-numeric: unset; font-weight: unset; text-orientation: unset; zoom: unset; letter-spacing: unset; background-blend-mode: unset; background-image: unset; background-position: unset; background-repeat: unset; background-size: unset; border-collapse: unset; border: unset; box-sizing: unset; break-after: unset; break-before: unset; break-inside: unset; caption-side: unset; clear: unset; column-width: unset; column-count: unset; column-fill: unset; column-gap: unset; column-rule-width: unset; column-rule-style: unset; column-rule-color: unset; column-span: unset; empty-cells: unset; float: unset; image-orientation: unset; isolation: unset; line-break: unset; line-height: 18px; list-style: unset; mix-blend-mode: unset; object-fit: unset; object-position: unset; outline: unset; overflow: unset; quotes: unset; table-layout: unset; text-align: unset; text-align-last: unset; text-decoration: unset; text-emphasis-style: unset; text-emphasis-color: unset; text-indent: unset; text-overflow: unset; text-transform: unset; text-underline-position: unset; vertical-align: unset; white-space: unset; word-break: unset; border-spacing: unset; word-spacing: unset; background-clip: unset; background-origin: unset; text-combine-upright: unset; display: block; opacity: unset; background-color: rgb(208, 216, 220); border-radius: unset; height: unset; margin-bottom: unset; margin-left: unset; margin-right: unset; max-height: unset; max-width: unset; min-height: unset; min-width: unset; padding: unset; width: unset; margin-top: 2px;">This message came from outside your organization.</div></div><div style="height: 0px; clear: both; display: block; line-height: 0; font-size: 0.01px;"></div></div><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div>Hi, Yongzhong,</div><div> It is great to see you have made such good progress. Barry is right, you need -vec_maxpy_use_gemv 1. It's my mistake for not mentioning it earlier. But even with that, there are still problems.</div><div> petsc tries to optimize VecMDot/MAXPY with BLAS GEMV, with hope that vendors' BLAS library would be highly optimized on that. However, we found though they were good with VecMDot, but not with VecMAXPY. So by default in petsc, we disabled the GEMV optimization for VecMAXPY. One can use -vec_maxpy_use_gemv 1 to turn on it.</div><div> I turned it on and tested VecMAXPY with ex2k and MKL, but failed to see any improvement with multiple threads. I could not understand why MKL is so bad on it. You can try it yourself in your environment. </div><div> Without the GEMV optimization, VecMAXPY() is implemented by petsc with a batch of PetscKernelAXPY() kernels, which contain simple for loops but not OpenMP parallelized (since petsc does not support OpenMP outright) . I added "omp parallel for" pragma in PetscKernelAXPY() kernels, and tested ex2k again with now parallelized petsc. Here is the result.</div><div><br></div><font face="monospace"> $ OMP_PLACES=cores OMP_PROC_BIND=spread OMP_NUM_THREADS=1 ./ex2k -n 15 -m 2 -test_name VecMAXPY -vec_maxpy_use_gemv 0<br>Vector(N) VecMAXPY-1 VecMAXPY-3 VecMAXPY-8 VecMAXPY-30 (us)<br>--------------------------------------------------------------------------<br> 128 7.0 10.1 21.4 72.7<br> 256 7.9 12.9 29.5 101.0<br> 512 9.4 17.2 40.5 136.2<br> 1024 15.9 27.3 67.5 249.3<br> 2048 26.5 48.7 139.6 432.7<br> 4096 47.1 77.3 186.4 710.3<br> 8192 84.8 152.2 423.9 1580.6<br> 16384 154.9 298.5 792.1 2889.2<br> 32768 183.7 338.7 893.9 3436.2<br> 65536 639.1 1247.8 3219.1 12494.8<br> 131072 1125.2 1856.2 6843.0 23653.7<br> 262144 2603.2 4948.4 13259.4 51287.7<br> 524288 5093.6 10305.0 26451.7 96919.6<br> 1048576 5898.6 10947.2 45486.4 127352.8<br> 2097152 11845.4 21912.5 57999.6 331403.4<br><br>$ OMP_PLACES=cores OMP_PROC_BIND=spread OMP_NUM_THREADS=16 ./ex2k -n 15 -m 2 -test_name VecMAXPY -vec_maxpy_use_gemv 0<br>Vector(N) VecMAXPY-1 VecMAXPY-3 VecMAXPY-8 VecMAXPY-30 (us)<br>--------------------------------------------------------------------------<br> 128 17.0 16.1 31.5 112.9<br> 256 13.7 16.8 31.2 120.2<br> 512 14.5 18.1 33.9 129.9<br> 1024 16.5 21.0 38.5 150.4<br> 2048 18.5 22.1 41.8 171.4<br> 4096 21.0 25.4 55.3 212.3<br> 8192 27.0 30.3 68.6 251.9<br> 16384 32.2 44.5 93.3 350.5<br> 32768 45.8 65.0 149.8 558.8<br> 65536 59.7 102.8 247.1 946.0<br> 131072 100.7 186.4 485.3 1898.1<br> 262144 183.4 345.2 922.2 3567.0<br> 524288 339.6 676.8 1820.7 7530.4<br> 1048576 662.0 1364.7 3585.3 13969.1<br> 2097152 1379.7 2788.6 7414.0 28275.3<span> </span></font><div><div><br></div><div>We can see VecMAXPY() can be easily speeded up with multithreading. </div><div><br></div><div>For MatSolve, I checked petsc's aijmkl.c, and found we don't have interface to MKL's sparse solve.</div></div></div></div></blockquote><div><br></div><div>We do, it’s in src/mat/impls/aij/seq/mkl_pardiso, and it’s threaded (the distributed version is in src/mat/impls/aij/mpi/mkl_cpardiso).</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div><div> I checked <a href="https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-linux/2023-0/openmp-threaded-functions-and-problems.html__;!!G_uCfscf7eWS!bT8Fh0B1GB5nDS3DTpc--fcfGuqOeym0MPwCORXl6F2Sy8A0GFIbVFQUT0J54XZ5Ds7eG_kLdQ-s6tD0GVEQIgTsoHmt$" target="_blank">https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-linux/2023-0/openmp-threaded-functions-and-problems.html</a>, but confused with MKL's list of threaded function</div><ul id="m_-3762595706006288767gmail-GUID-E3F8A448-C29A-4370-AEA0-9031E5FE0889" style="box-sizing: border-box; margin-top: 0px; margin-bottom: 11px; color: rgb(38, 38, 38); font-family: intel-clear, tahoma, Helvetica, helvetica, Arial, sans-serif; font-size: 16px;"><li style="box-sizing: border-box;"><div style="box-sizing: border-box; margin: 0px; line-height: 1.25;">Direct sparse solver.</div></li><li style="box-sizing: border-box;"><div style="box-sizing: border-box; margin: 0px; line-height: 1.25;">All Level 3 BLAS and all Sparse BLAS routines except Level 2 Sparse Triangular solvers.</div></li></ul>I don't know whether MKL has threaded sparse solver.<div><br></div><div>--Junchao Zhang<br></div><br></div></div><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div dir="ltr" class="gmail_attr">On Fri, Jun 28, 2024 at 11:35 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><div><br></div> Are you running with -vec_maxpy_use_gemv ?<div><br id="m_-3762595706006288767m_-6213770648425822909lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jun 28, 2024, at 1:46 AM, Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank">yongzhong.li@mail.utoronto.ca</a>> wrote:</div><br><div><div style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Thanks all for your help!!!<br><br>I think I find the issues. I am compiling a large CMake project that relies on many external libraries (projects). Previously, I used OpenBLAS as the BLAS for all the dependencies including PETSc. After I switched to Intel MKL for PETSc, I still kept the OpenBLAS and use it as the BLAS for all the other dependencies. I think somehow even when I specify the blas-lapack-dir to the MKLROOT when PETSc is configured, the actual program still use OpenBLAS as the BLAS for some PETSc functions, such as VecMDot() and VecMAXPY(), so that’s why I didn’t see any MKL verbose during the KSPSolve(). Now I remove the OpenBLAS and use Intel MKL as the BLAS for all the dependencies. The issue is resolved, I can clearly see MKL routines are called when KSP GMRES is running.<br><br>Back to my original questions, my goal is to achieve good parallelization efficiency for KSP GMRES Solve. As I use multithreading-enabled MKL spmv routines, the wall time for MatMult/MatMultAdd() has been greatly reduced. However,the KSPGMRESOrthog and MatSolve in PCApply still take over 50% of solving time and can’t benefit from multithreading.<span> </span><b>After I fixed the issue I mentioned, I found I got around 15% time reduced because of more efficient VecMDot() calls</b>. I attach a petsc log comparison for your reference (same settings, only difference is whether use MKL BLAS or not), you can see the percentage of VecMDot() is reduced. However, here comes the interesting part,<span> </span><b>VecMAXPY() didn’t benefit from MKL BLAS, it still takes almost 40% of solution when I use 64 MKL Threads</b>, which is a lot for my program. And if I multiple this percentage with the actual wall time against different # of threads, it stays the same. Then I used ex2k benchmark to verify what I found. Here is the result,<br><br>$ MKL_NUM_THREADS=1 ./ex2k -n 15 -m 5 -test_name VecMAXPY<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Vector(N) VecMAXPY-1 VecMAXPY-3 VecMAXPY-8 VecMAXPY-30 (us)<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">--------------------------------------------------------------------------<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 128 0.4 0.9 2.4 8.8<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 256 0.3 1.1 3.5 13.3<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 512 0.5 4.4 6.7 26.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 1024 0.9 4.8 13.3 51.0<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 2048 3.5 12.3 37.1 94.7<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 4096 4.3 24.5 73.6 179.6<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 8192 6.3 48.7 98.9 380.8<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 16384 9.3 99.2 200.2 774.0<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 32768 30.6 155.4 421.2 1662.9<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 65536 101.2 269.4 827.4 3565.0<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 131072 206.9 551.0 1829.0 7580.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 262144 450.2 1251.9 3986.2 15525.6<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 524288 1322.1 2901.7 8567.1 31840.0<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 1048576 2788.6 6190.6 16394.7 63514.9<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 2097152 5534.8 12619.9 35427.4 130064.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">$ MKL_NUM_THREADS=8 ./ex2k -n 15 -m 5 -test_name VecMAXPY<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Vector(N) VecMAXPY-1 VecMAXPY-3 VecMAXPY-8 VecMAXPY-30 (us)<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">--------------------------------------------------------------------------<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 128 0.3 0.7 2.4 8.8<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 256 0.3 1.1 3.6 13.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 512 0.5 4.4 6.8 26.4<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 1024 0.9 4.8 13.6 50.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 2048 7.6 12.2 36.5 95.0<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 4096 8.5 25.7 72.4 182.6<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 8192 11.9 48.5 103.7 383.7<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 16384 12.8 97.7 203.7 785.0<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 32768 11.2 148.5 421.9 1681.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 65536 15.5 271.2 843.8 3613.7<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 131072 34.3 564.7 1905.2 7558.8<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 262144 106.4 1334.5 4002.8 15458.3<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 524288 217.2 2858.4 8407.9 31303.7<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 1048576 701.5 6060.6 16947.3 64118.5<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> 2097152 1769.7 13218.3 36347.3 131062.9<br><br>It stays the same, no benefit from multithreading BLAS!! Unlike what I found for VecMdot(), where I did see speed up for more #of threads. Then, I dig deeper.<span> </span><b>I learned that for VecMDot(), it calls ZGEMV while for VecMAXPY(), it calls ZAXPY. This observation seems to indicate that ZAXPY is not benefiting from MKL threads.</b><br><br>My question is<span> </span><b>do you know why ZAXPY is not multithreaded</b>? From my perspective, VecMDot() and VecMAXPY() are very similar operations, the only difference is whether we need to scale the vectors to be multiplied or not. I think you have mentioned that recently you did some optimization to these two routines<b>, from my above results and observations, are these aligned with your expectations</b>? Could we further optimize the codes to get more parallelization efficiency in my case?<br><br><b>And another question, can MatSolve() in KSPSolve be multithreaded? Would MUMPS help?<u></u><u></u></b></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br>Thank you and regards,<u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Yongzhong<br><br><u></u><u></u></div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; padding: 3pt 0cm 0cm;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"><span> </span>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;">junchao.zhang@gmail.com</a>><br><b>Sent:</b><span> </span>Thursday, June 27, 2024 11:10 AM<br><b>To:</b><span> </span>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;">yongzhong.li@mail.utoronto.ca</a>><br><b>Cc:</b><span> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: purple; text-decoration: underline;">bsmith@petsc.dev</a>>;<span> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;">petsc-users@mcs.anl.gov</a><br><b>Subject:</b><span> </span>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></span></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><u></u> <u></u></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">How big is the n when you call PetscCallBLAS("BLASgemv", BLASgemv_(trans, &n, &m, &one, yarray, &lda2, xarray, &ione, &zero, z + i, &ione))? n is the vector length in VecMDot.<u></u><u></u></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">it is strange with MKL_VERBOSE=1 you did not see <span style="font-family: "Courier New";">MKL_VERBOSE <b>ZGEMV..., </b></span>since the code did call gemv. Perhaps you need to double check your spelling etc. <u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><u></u> <u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">If you also use ex2k, and potentially modify Ms[] and Ns[] to match the sizes in your code, to see if there is a speedup with more threads.<u></u><u></u></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br clear="all"><u></u><u></u></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">--Junchao Zhang<u></u><u></u></div></div></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><u></u> <u></u></div></div></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><u></u> <u></u></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Thu, Jun 27, 2024 at 9:39<span style="font-family: Arial, sans-serif;"> </span>AM Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;">yongzhong.li@mail.utoronto.ca</a>> wrote:<u></u><u></u></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">Mostly 3, maximum 7, but definitely hit the point when m > 1, I can see the PetscCallBLAS("BLASgemv", BLASgemv_(trans, &n, &m, &one, yarray, &lda2, xarray, &ione, &zero, z + i, &ione)); is called multiple<span> </span><u></u><u></u></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart<u></u><u></u></span></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerb8mftbr" style="border-width: 3pt medium medium; border-style: solid none none; border-color: rgb(144, 164, 174) currentcolor currentcolor; padding: 6pt 0cm 0cm; margin-top: 12pt; margin-bottom: 12pt; border-radius: 4px; min-width: 200px;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerb8mftbr" style="margin-bottom: 0.75pt; float: left;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerb8mftbr"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender<u></u><u></u></span></b></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerb8mftbr" style="margin-top: 1.5pt;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.<u></u><u></u></span></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 0px; background: rgb(208, 216, 220);"><span style="font-size: 1pt;"> </span><span style="font-size: 1pt;"><u></u><u></u></span></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd<u></u><u></u></span></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Mostly 3, maximum 7, but definitely hit the point when m > 1,<br><br>I can see the PetscCallBLAS("BLASgemv", BLASgemv_(trans, &n, &m, &one, yarray, &lda2, xarray, &ione, &zero, z + i, &ione)); is called multiple times</span><u></u><u></u></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136mail-editor-reference-message-container"><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span>From:<span> </span></span></b><span>Barry Smith <</span><a href="mailto:bsmith@petsc.dev" target="_blank" style="color: purple; text-decoration: underline;">bsmith@petsc.dev</a><span>><br><b>Date:<span> </span></b>Thursday, June 27, 2024 at 1:12</span><span style="font-family: Arial, sans-serif;"> </span><span>AM<br><b>To:<span> </span></b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;">yongzhong.li@mail.utoronto.ca</a><span>><br><b>Cc:<span> </span></b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span><span> </span><</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span>><br><b>Subject:<span> </span></b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue</span><u></u><u></u></p></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>How big are the m's getting in your code?<u></u><u></u></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><u></u> <u></u></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Jun 27, 2024, at 12:40<span style="font-family: Arial, sans-serif;"> </span>AM, Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;">yongzhong.li@mail.utoronto.ca</a>> wrote:<u></u><u></u></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Hi Barry, I used gdb to debug my program, set a breakpoint to VecMultiDot_Seq_GEMV function. I did see when I debug this function, it will call BLAS (but not always, only if m > 1), as shown below. However, I still didn’t see any MKL outputs even if I set MKLK_VERBOSE=1.<br><br></span><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">550 PetscCall(VecRestoreArrayRead(yin[i], &yfirst));</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">553 m = j - i;</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">554 if (m > 1) {</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">555 PetscBLASInt ione = 1, lda2 = (PetscBLASInt)lda; // the cast is safe since we've screened out those lda > PETSC_BLAS_INT_MAX above</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">556 PetscScalar one = 1, zero = 0;</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">558 PetscCallBLAS("BLASgemv", BLASgemv_(trans, &n, &m, &one, yarray, &lda2, xarray, &ione, &zero, z + i, &ione));</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb) s</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">PetscMallocValidate (line=558, function=0x7ffff68a11a0 <__func__.18210> "VecMultiDot_Seq_GEMV",</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;"> file=0x7ffff68a1078 "/gpfs/s4h/scratch/t/triverio/modelics/workplace/rebel/build_debug/external/petsc-3.21.0/src/vec/vec/impls/seq/dvec2.c")</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;"> <span class="Apple-converted-space"> </span>at /gpfs/s4h/scratch/t/triverio/modelics/workplace/rebel/build_debug/external/petsc-3.21.0/src/sys/memory/mtr.c:106</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">106 if (!TRdebug) return PETSC_SUCCESS;</span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">(gdb)<span> </span></span></i><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span style="font-size: 11pt;">154 }</span></i><i><span lang="EN-US" style="font-size: 11pt;"><br><br></span></i><span lang="EN-US" style="font-size: 11pt;">Am I not using MKL BLAS, is that why I didn’t see multithreading speed up for KSPGMRESOrthog? What do you think could be the potential reasons? Is there any silent mode that will possibly affect the MKL Verbose.<span> </span><br><br>Thank you and best regards,</span><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"> </span><u></u><u></u></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From:<span> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">bsmith@petsc.dev</span></a>><br><b>Date:<span> </span></b>Wednesday, June 26, 2024 at 8:15<span style="font-family: Arial, sans-serif;"> </span>PM<br><b>To:<span> </span></b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>><br><b>Cc:<span> </span></b><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject:<span> </span></b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>if (m > 1) {<u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>PetscBLASInt ione = 1, lda2 = (PetscBLASInt)lda; // the cast is safe since we've screened out those lda > PETSC_BLAS_INT_MAX above<u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>PetscScalar one = 1, zero = 0;<u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>PetscCallBLAS("BLASgemv", BLASgemv_(trans, &n, &m, &one, yarray, &lda2, xarray, &ione, &zero, z + i, &ione));<u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>PetscCall(PetscLogFlops(PetscMax(m * (2.0 * n - 1), 0.0)));<u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">The call to BLAS above is where it uses MKL.<u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Jun 26, 2024, at 6:59<span style="font-family: Arial, sans-serif;"> </span>PM, Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Hi Barry, I am looking into the source codes of VecMultiDot_Seq_GEMV</span><span><span lang="EN-US"> </span></span><a href="https://urldefense.us/v3/__https://petsc.org/release/src/vec/vec/impls/seq/dvec2.c.html*VecMDot_Seq__;Iw!!G_uCfscf7eWS!ZshPGnAUymZ7rmZ8Cq0JR23FBhEioHOuAq-lFnn4iQn1bK8ioexLwIQVLSQNCfmBaWWExCcshZ6KphgTYR6kv18wg0MHEITtuVo$" target="_blank" style="color: purple; text-decoration: underline;"><span lang="EN-US" style="font-size: 11pt; color: purple;">https://petsc.org/release/src/vec/vec/impls/seq/dvec2.c.html#VecMDot_Seq</span></a><span lang="EN-US" style="font-size: 11pt;"><br>Can I ask which lines of codes suggest the use of intel mkl?<br><br>Thanks,</span><u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"> </span><u></u><u></u></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From:<span> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">bsmith@petsc.dev</span></a>><br><b>Date:<span> </span></b>Wednesday, June 26, 2024 at 10:30<span style="font-family: Arial, sans-serif;"> </span>AM<br><b>To:<span> </span></b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>><br><b>Cc:<span> </span></b><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject:<span> </span></b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> In a debug version of PETSc run your application in a debugger and put a break point in VecMultiDot_Seq_GEMV. Then next through the code from that point to see what decision it makes about using dgemv() to see why it is not getting into the Intel code.<u></u><u></u></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Jun 25, 2024, at 11:19<span style="font-family: Arial, sans-serif;"> </span>PM, Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerqgdldcn"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerqgdldcn"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerqgdldcn"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This Message Is From an External Sender</span><u></u><u></u></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136pfptBannerqgdldcn"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Hi Junchao, thank you for your help for these benchmarking test!<br><br>I check out to petsc/main and did a few things to verify from my side,<br><br>1. I ran the microbenchmark (vec/vec/tests/ex2k.c) test on my compute node. The results are as follow,</span><u></u><u></u></p></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">$ MKL_NUM_THREADS=64 ./ex2k -n 15 -m 4<br>Vector(N) VecMDot-1 VecMDot-3 VecMDot-8 VecMDot-30 (us)</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">--------------------------------------------------------------------------</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span class="Apple-converted-space"> </span>128 14.5 1.2 1.8 5.2</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 256 1.5 0.9 1.6 4.7</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 512 2.7 2.8 6.1 13.2</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 1024 4.0 4.0 9.3 16.4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 2048 7.4 7.3 11.3 39.3</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 4096 14.2 13.9 19.1 93.4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 8192 28.8 26.3 25.4 31.3</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 16384 54.1 25.8 26.7 33.8</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 32768 109.8 25.7 24.2 56.0</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 65536 220.2 24.4 26.5 89.0</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 131072 424.1 31.5 36.1 149.6</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 262144 898.1 37.1 53.9 286.1</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 524288 1754.6 48.7 100.3 1122.2</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 1048576 3645.8 86.5 347.9 2950.4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 2097152 7371.4 308.7 1440.6 6874.9</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">$ MKL_NUM_THREADS=1 ./ex2k -n 15 -m 4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Vector(N) VecMDot-1 VecMDot-3 VecMDot-8 VecMDot-30 (us)</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">--------------------------------------------------------------------------</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span class="Apple-converted-space"> </span>128 14.9 1.2 1.9 5.2</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 256 1.5 1.0 1.7 4.7</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 512 2.7 2.8 6.1 12.0</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 1024 3.9 4.0 9.3 16.8</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 2048 7.4 7.3 10.4 41.3</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 4096 14.0 13.8 18.6 84.2</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 8192 27.0 21.3 43.8 177.5</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 16384 54.1 34.1 89.1 330.4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 32768 110.4 82.1 203.5 781.1</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 65536 213.0 191.8 423.9 1696.4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 131072 428.7 360.2 934.0 4080.0</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 262144 883.4 723.2 1745.6 10120.7</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 524288 1817.5 1466.1 4751.4 23217.2</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 1048576 3611.0 3796.5 11814.9 48687.7</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> 2097152 7401.9 10592.0 27543.2 106565.4</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><br>I can see the speed up brought by more MKL threads, and if I set NKL_VERBOSE to 1, I can see something like</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><i><span lang="EN-US" style="font-size: 11pt;">MKL_VERBOSE ZGEMV(C,262144,8,0x7ffd375d6470,0x2ac76e7fb010,262144,0x16d0f40,1,0x7ffd375d6480,0x16435d0,1) 32.70us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:6 ca<br><br></span></i><span lang="EN-US" style="font-size: 11pt;">From my understanding, the VecMDot()/VecMAXPY() can benefit from more MKL threads in my compute node and is using ZGEMV MKL BLAS.<br><br>However, when I ran my own program and set MKL_VERBOSE to 1, it is very strange that I still can’t find any MKL outputs, though I can see from the PETSc log that VecMDot and VecMAXPY() are called.</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><br>I am wondering are VecMDot and VecMAXPY in KSPGMRESOrthog optimized in a way that is similar to ex2k test? Should I expect to see MKL outputs for whatever linear system I solve with KSPGMRES? Does it relate to if it is dense matrix or sparse matrix, although I am not really understand why VecMDot/MAXPY() have something to do with dense matrix-vector multiplication.<br><br>Thank you,</span><u></u><u></u></div></div></div></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></p></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From:<span> </span></b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>><br><b>Date:<span> </span></b>Tuesday, June 25, 2024 at 6:34<span style="font-family: Arial, sans-serif;"> </span>PM<br><b>To:<span> </span></b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">knepley@gmail.com</span></a>><br><b>Cc:<span> </span></b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>>, Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">pierre@joliv.et</span></a>>,<span> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject:<span> </span></b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Hi, Yongzhong,<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>Since the two kernels of KSPGMRESOrthog are VecMDot and VecMAXPY, if we can speed up the two with OpenMP threads, then we can speed up KSPGMRESOrthog. We recently added an optimization to do VecMDot/MAXPY() in dense matrix-vector multiplication (i.e., BLAS2 GEMV, with tall-and-skinny matrices ). So with MKL_VERBOSE=1, you should see something like "MKL_VERBOSE ZGEMV ..." in output. If not, could you try again with petsc/main?<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span class="Apple-converted-space"> </span>petsc has a microbenchmark (vec/vec/tests/ex2k.c) to test them. I ran VecMDot with multithreaded oneMKL (via setting MKL_NUM_THREADS), it was strange to see no speedup. I then configured petsc with openblas, I did see better performance with more threads<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: "Courier New";">$ OMP_PROC_BIND=spread OMP_NUM_THREADS=1 ./ex2k -n 15 -m 4<br>Vector(N) VecMDot-3 VecMDot-8 VecMDot-30 (us)<br>--------------------------------------------------------------------------<br> 128 2.0 2.5 6.1<span> </span><br> 256 1.8 2.7 7.0<span> </span><br> 512 2.1 3.1 8.6<span> </span><br> <span class="Apple-converted-space"> </span>1024 2.7 4.0 12.3<span> </span><br> <span class="Apple-converted-space"> </span>2048 3.8 6.3 28.0<span> </span><br> <span class="Apple-converted-space"> </span>4096 6.1 10.6 42.4<span> </span><br> <span class="Apple-converted-space"> </span>8192 10.9 21.8 79.5<span> </span><br> 16384 21.2 39.4 149.6<span> </span><br> 32768 45.9 75.7 224.6<span> </span><br> 65536 142.2 215.8 732.1<span> </span><br> <span class="Apple-converted-space"> </span>131072 169.1 233.2 1729.4<span> </span><br> <span class="Apple-converted-space"> </span>262144 367.5 830.0 4159.2<span> </span><br> <span class="Apple-converted-space"> </span>524288 999.2 1718.1 8538.5<span> </span><br> 1048576 2113.5 4082.1 18274.8<span> </span><br> 2097152 5392.6 10273.4 43273.4 </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: "Courier New";">$ OMP_PROC_BIND=spread OMP_NUM_THREADS=8 ./ex2k -n 15 -m 4<br>Vector(N) VecMDot-3 VecMDot-8 VecMDot-30 (us)<br>--------------------------------------------------------------------------<br> 128 2.0 2.5 6.0<span> </span><br> 256 1.8 2.7 15.0<span> </span><br> 512 2.1 9.0 16.6<span> </span><br> <span class="Apple-converted-space"> </span>1024 2.6 8.7 16.1<span> </span><br> <span class="Apple-converted-space"> </span>2048 7.7 10.3 20.5<span> </span><br> <span class="Apple-converted-space"> </span>4096 9.9 11.4 25.9<span> </span><br> <span class="Apple-converted-space"> </span>8192 14.5 22.1 39.6<span> </span><br> 16384 25.1 27.8 67.8<span> </span><br> 32768 44.7 95.7 91.5<span> </span><br> 65536 82.1 156.8 165.1<span> </span><br> <span class="Apple-converted-space"> </span>131072 194.0 335.1 341.5<span> </span><br> <span class="Apple-converted-space"> </span>262144 388.5 380.8 612.9<span> </span><br> <span class="Apple-converted-space"> </span>524288 1046.7 967.1 1653.3<span> </span><br> 1048576 1997.4 2169.0 4034.4<span> </span><br> 2097152 5502.9 5787.3 12608.1</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">The tall-and-skinny matrices in KSPGMRESOrthog vary in width. The average speedup depends on components. So I suggest you run ex2k to see in your environment whether oneMKL can speedup the kernels. <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">--Junchao Zhang<u></u><u></u></div></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Mon, Jun 24, 2024 at 11:35<span style="font-family: Arial, sans-serif;"> </span>AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>> wrote:<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Let me run some examples on our end to see whether the code calls expected functions. <u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br clear="all"><u></u><u></u></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">--Junchao Zhang<u></u><u></u></div></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Mon, Jun 24, 2024 at 10:46<span style="font-family: Arial, sans-serif;"> </span>AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">knepley@gmail.com</span></a>> wrote:<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">On Mon, Jun 24, 2024 at 11:</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><span style="font-size: 1pt; color: white;">21 AM Yongzhong Li <yongzhong.</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><span style="font-size: 1pt; color: white;">li@</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><span style="font-size: 1pt; color: white;">mail.</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><span style="font-size: 1pt; color: white;">utoronto.</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><span style="font-size: 1pt; color: white;">ca> wrote: Thank you Pierre for your information. Do we have a conclusion for my original question about the parallelization efficiency for different stages of</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart</span><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203pfptBanner0gnzkk2" style="margin-top: 12pt; margin-bottom: 12pt; border-width: medium; border-style: none; border-color: currentcolor; border-radius: 4px; min-width: 200px;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203pfptBanner0gnzkk2" style="margin-bottom: 0.75pt; float: left;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203pfptBanner0gnzkk2"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203pfptBanner0gnzkk2" style="margin-top: 1.5pt;"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; background: rgb(208, 216, 220);"><span style="font-size: 1pt;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Mon, Jun 24, 2024 at 11:21<span style="font-family: Arial, sans-serif;"> </span>AM Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div></div></div><div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">Thank you Pierre for your information. Do we have a conclusion for my original question about the parallelization efficiency for different stages of KSP Solve? Do we need to do more testing to figure out the issues? Thank you, Yongzhong From:</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart</span><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBanner7d6k4n9" style="margin-top: 12pt; margin-bottom: 12pt; border-width: medium; border-style: none; border-color: currentcolor; border-radius: 4px; min-width: 200px;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBanner7d6k4n9" style="margin-bottom: 0.75pt; float: left;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBanner7d6k4n9"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBanner7d6k4n9" style="margin-top: 1.5pt;"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; background: rgb(208, 216, 220);"><span style="font-size: 1pt;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Thank you Pierre for your information. Do we have a conclusion for my original question about the parallelization efficiency for different stages of KSP Solve? Do we need to do more testing to figure out the issues?</span><u></u><u></u></div></div></div></div></div></div></div></blockquote><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">We have an extended discussion of this here: <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/*what-kind-of-parallel-computers-or-clusters-are-needed-to-use-petsc-or-why-do-i-get-little-speedup__;Iw!!G_uCfscf7eWS!aQJpmm5W6l6FUiumnIPmkouzwzNUfx-Dyq04i1O2KS_InQGk6qjI7wUir0Hx6QEUQE2AMiJDsez3x4zRO7V_$" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-are-needed-to-use-petsc-or-why-do-i-get-little-speedup</span></a><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">The kinds of operations you are talking about (SpMV, VecDot, VecAXPY, etc) are memory bandwidth limited. If there is no more bandwidth to be marshalled on your board, then adding more processes does nothing at all. This is why people were asking about how many "nodes" you are running on, because that is the unit of memory bandwidth, not "cores" which make little difference.<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span> </span>Thanks,<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> Matt<u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Thank you,</span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"> </span><u></u><u></u></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From:<span> </span></b>Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">pierre@joliv.et</span></a>><br><b>Date:<span> </span></b>Sunday, June 23, 2024 at 12:41<span style="font-family: Arial, sans-serif;"> </span>AM<br><b>To:<span> </span></b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>><br><b>Cc:<span> </span></b><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject:<span> </span></b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On 23 Jun 2024, at 4:07<span style="font-family: Arial, sans-serif;"> </span>AM, Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div><div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBannerfa2qtuo"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBannerfa2qtuo"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBannerfa2qtuo"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This Message Is From an External Sender</span><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261pfptBannerfa2qtuo"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yeah, I ran my program again using -mat_view::ascii_info and set MKL_VERBOSE to be 1, then I noticed the outputs suggested that the matrix to be seqaijmkl type (I’ve attached a few as below)<br><br>--> Setting up matrix-vector products...</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Mat Object: 1 MPI process</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>type: seqaijmkl</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>rows=16490, cols=35937</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>total: nonzeros=128496, allocated nonzeros=128496</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>total number of mallocs used during MatSetValues calls=0</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>not using I-node routines</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Mat Object: 1 MPI process</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>type: seqaijmkl</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>rows=16490, cols=35937</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>total: nonzeros=128496, allocated nonzeros=128496</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>total number of mallocs used during MatSetValues calls=0</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <span> </span>not using I-node routines</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">--> Solving the system...</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Excitation 1 of 1...</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">================================================</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Iterative solve completed in 7435 ms.</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">CONVERGED: rtol.</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Iterations: 72</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Final relative residual norm: 9.22287e-07</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">================================================</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">[CPU TIME] System solution: 2.27160000e+02 s.</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">[WALL TIME] System solution: 7.44387218e+00 s.<br><br>However, it seems to me that there were still no MKL outputs even I set MKL_VERBOSE to be 1. Although, I think it should be many spmv operations when doing KSPSolve(). Do you see the possible reasons?</span><u></u><u></u></div></div></div></div></div></div></div></blockquote><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">SPMV are not reported with MKL_VERBOSE (last I checked), only dense BLAS is.<u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Thanks,<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Pierre<u></u><u></u></div></div></div></div></div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Thanks,</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From: </b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">knepley@gmail.com</span></a>><br><b>Date: </b>Saturday, June 22, 2024 at 5:56<span style="font-family: Arial, sans-serif;"> </span>PM<br><b>To: </b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>><br><b>Cc: </b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>>, Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">pierre@joliv.et</span></a>>, <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject: </b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><table border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width: 792.219px; display: table; float: none;"><tbody><tr><td style="background: rgb(166, 166, 166); padding: 5.25pt 1.5pt;"></td><td width="100%" style="width: 758.219px; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt 5.25pt 11.25pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">你通常不会收到来自</span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span><a href="mailto:knepley@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: purple;">knepley@gmail.com</span></a></span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">的电子邮件。</span><span><a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fVvbGldqcUV5ju4jpu5oGmt-VjITi5JpCJzhHxpbgsERLVYZzglpxKOOyrBRGxjRxp7vWHwt3SnINFOQErR1Z8kcDcf3qwbYRxM$" target="_blank" style="color: purple; text-decoration: underline;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: purple;">了解这一点为什么很重要</span></a></span><span><u></u><u></u></span></div></div></div></div></div></div></td><td width="75" style="width: 56.25pt; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt;"></td></tr></tbody></table><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Sat, Jun 22, 2024 at 5:03<span style="font-family: Arial, sans-serif;"> </span>PM Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div></div></div></div><div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">MKL_VERBOSE=1 ./ex1 matrix nonzeros = 100, allocated nonzeros = 100 MKL_VERBOSE Intel(R) MKL 2019.</span><span style="font-size: 1pt; font-family: Arial, sans-serif; color: white;"> </span><span style="font-size: 1pt; color: white;">0 Update 4 Product build 20190411 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512) with support of Vector</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart</span><u></u><u></u></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322pfptBanner3e1hrxr" style="margin-top: 12pt; margin-bottom: 12pt; border-color: currentcolor; border-radius: 4px; min-width: 200px;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322pfptBanner3e1hrxr" style="margin-bottom: 0.75pt; float: left;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322pfptBanner3e1hrxr"><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><u></u><u></u></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322pfptBanner3e1hrxr" style="margin-top: 1.5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; background: rgb(208, 216, 220);"><span style="font-size: 1pt;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: "Courier New";">MKL_VERBOSE=1 ./ex1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><br></span><span style="font-size: 11pt;">matrix nonzeros = 100, allocated nonzeros = 100</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE Intel(R) MKL 2019.0 Update 4 Product build 20190411 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512) with support of Vector Neural Network Instructions enabled processors, Lnx 2.50GHz lp64 gnu_thread</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEMV(N,10,10,0x7ffd9d7078f0,0x187eb20,10,0x187f7c0,1,0x7ffd9d707900,0x187ff70,1) 167.34ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRF(L,10,0x1894b50,10,0x1893df0,0x7ffd9d7078c0,-1,0) 77.19ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRF(L,10,0x1894b50,10,0x1893df0,0x1894490,10,0) 83.97ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRS(L,10,1,0x1894b50,10,0x1893df0,0x1880720,10,0) 44.94ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(10,0x7ffd9d7078f0,0x187f7c0,1,0x1880720,1) 20.72us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRS(L,10,2,0x1894b50,10,0x1893df0,0x187d2a0,10,0) 4.22us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEMM(N,N,10,2,10,0x7ffd9d707790,0x187eb20,10,0x187d2a0,10,0x7ffd9d7077a0,0x1896a70,10) 1.41ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(20,0x7ffd9d7078a0,0x1896a70,1,0x187b650,1) 381ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRF(L,10,0x1894b50,10,0x1893df0,0x7ffd9d707840,-1,0) 742ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRF(L,10,0x1894b50,10,0x1893df0,0x18951a0,10,0) 4.20us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZSYTRS(L,10,1,0x1894b50,10,0x1893df0,0x1880720,10,0) 2.94us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(10,0x7ffd9d7078f0,0x187f7c0,1,0x1880720,1) 292ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEMV(N,10,10,0x7ffd9d7078f0,0x187eb20,10,0x187f7c0,1,0x7ffd9d707900,0x187ff70,1) 1.17us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGETRF(10,10,0x1894b50,10,0x1893df0,0) 202.48ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGETRS(N,10,1,0x1894b50,10,0x1893df0,0x1880720,10,0) 20.78ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(10,0x7ffd9d7078f0,0x187f7c0,1,0x1880720,1) 954ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGETRS(N,10,2,0x1894b50,10,0x1893df0,0x187d2a0,10,0) 30.74ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEMM(N,N,10,2,10,0x7ffd9d707790,0x187eb20,10,0x187d2a0,10,0x7ffd9d7077a0,0x18969c0,10) 3.95us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(20,0x7ffd9d7078a0,0x18969c0,1,0x187b650,1) 995ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGETRF(10,10,0x1894b50,10,0x1893df0,0) 4.09us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGETRS(N,10,1,0x1894b50,10,0x1893df0,0x1880720,10,0) 3.92us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(10,0x7ffd9d7078f0,0x187f7c0,1,0x1880720,1) 274ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEMV(N,15,10,0x7ffd9d7078f0,0x187ec70,15,0x187fc30,1,0x7ffd9d707900,0x1880400,1) 1.59us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEQRF(15,10,0x1894b40,15,0x1894550,0x7ffd9d707900,-1,0) 47.07us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEQRF(15,10,0x1894b40,15,0x1894550,0x1895cb0,10,0) 26.62us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZUNMQR(L,C,15,1,10,0x1894b40,15,0x1894550,0x1895b00,15,0x7ffd9d7078b0,-1,0) 35.32us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZUNMQR(L,C,15,1,10,0x1894b40,15,0x1894550,0x1895b00,15,0x1895cb0,10,0) 42.33ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZTRTRS(U,N,N,10,1,0x1894b40,15,0x1895b00,15,0) 16.11us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(10,0x7ffd9d7078f0,0x187fc30,1,0x1880c70,1) 395ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEMM(N,N,15,2,10,0x7ffd9d707790,0x187ec70,15,0x187d310,10,0x7ffd9d7077a0,0x187b5b0,15) 3.22us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZUNMQR(L,C,15,2,10,0x1894b40,15,0x1894550,0x1897760,15,0x7ffd9d7078c0,-1,0) 730ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZUNMQR(L,C,15,2,10,0x1894b40,15,0x1894550,0x1897760,15,0x1895cb0,10,0) 4.42us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZTRTRS(U,N,N,10,2,0x1894b40,15,0x1897760,15,0) 5.96us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(20,0x7ffd9d7078a0,0x187d310,1,0x1897610,1) 222ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEQRF(15,10,0x1894b40,15,0x18954b0,0x7ffd9d707820,-1,0) 685ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZGEQRF(15,10,0x1894b40,15,0x18954b0,0x1895d60,10,0) 6.11us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZUNMQR(L,C,15,1,10,0x1894b40,15,0x18954b0,0x1895bb0,15,0x7ffd9d7078b0,-1,0) 390ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZUNMQR(L,C,15,1,10,0x1894b40,15,0x18954b0,0x1895bb0,15,0x1895d60,10,0) 3.09us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZTRTRS(U,N,N,10,1,0x1894b40,15,0x1895bb0,15,0) 1.05us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;">MKL_VERBOSE ZAXPY(10,0x7ffd9d7078f0,0x187fc30,1,0x1880c70,1) 257ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</span><span lang="EN-US" style="font-size: 11pt;"><br><br>Yes, for petsc example, there are MKL outputs, but for my own program. All I did is to change the matrix type from MATAIJ to MATAIJMKL to get optimized performance for spmv from MKL. Should I expect to see any MKL outputs in this case?</span><u></u><u></u></div></div></div></div></div></div></div></div></blockquote><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Are you sure that the type changed? You can MatView() the matrix with format ascii_info to see.<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <span> </span>Thanks,<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> Matt<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Thanks,</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From: </b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>><br><b>Date: </b>Saturday, June 22, 2024 at 9:40<span style="font-family: Arial, sans-serif;"> </span>AM<br><b>To: </b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>><br><b>Cc: </b>Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">pierre@joliv.et</span></a>>, <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject: </b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">No, you don't. It is strange. Perhaps you can you run a petsc example first and see if MKL is really used<u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: "Courier New";">$ cd src/mat/tests</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: "Courier New";">$ make ex1</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: "Courier New";">$ MKL_VERBOSE=1 ./ex1</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br clear="all"><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">--Junchao Zhang<u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On Fri, Jun 21, 2024 at 4:03<span style="font-family: Arial, sans-serif;"> </span>PM Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div></div></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">I am using<br><br>export MKL_VERBOSE=1</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">./xx<br><br>in the bash file, do I have to use -</span><span lang="EN-US"> </span>ksp_converged_reason<span lang="EN-US">?<br><br>Thanks,</span><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US">Yongzhong</span><u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From: </b>Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">pierre@joliv.et</span></a>><br><b>Date: </b>Friday, June 21, 2024 at 1:47<span style="font-family: Arial, sans-serif;"> </span>PM<br><b>To: </b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>><br><b>Cc: </b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>>, <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject: </b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><table border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width: 761.438px; display: table; float: none;"><tbody><tr><td style="background: rgb(166, 166, 166); padding: 5.25pt 1.5pt;"></td><td width="100%" style="width: 727.438px; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt 5.25pt 11.25pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">你通常不会收到来自</span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span><a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: purple;">pierre@joliv.et</span></a></span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">的电子邮件。</span><span><a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!flsZMI97ne0yyxHhLda3hROB9qsgstuZS-jPinxGIzFCCSdn1ujdoMR8dyz-5_kVqqMM-12Lt0dTdjKrx3wXhHZmBhNydvFQeSY$" target="_blank" style="color: purple; text-decoration: underline;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: purple;">了解这一点为什么很重要</span></a></span><span><u></u><u></u></span></div></div></div></div></div></div></td><td width="75" style="width: 56.25pt; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt;"></td></tr></tbody></table><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">How do you set the variable?<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">$ MKL_VERBOSE=1 ./ex1 -ksp_converged_reason<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">MKL_VERBOSE oneMKL 2024.0 Update 1 Product build 20240215 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Lnx 2.80GHz lp64 intel_thread<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">MKL_VERBOSE DDOT(10,0x22127c0,1,0x22127c0,1) 2.02ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">MKL_VERBOSE DSCAL(10,0x7ffc9fb4ff08,0x22127c0,1) 12.67us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">MKL_VERBOSE DDOT(10,0x22127c0,1,0x2212840,1) 1.52us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">MKL_VERBOSE DDOT(10,0x2212840,1,0x2212840,1) 167ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">[...]<u></u><u></u></div></div></div></div></div></div><div><div style="margin-bottom: 12pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On 21 Jun 2024, at 7:37<span style="font-family: Arial, sans-serif;"> </span>PM, Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>> wrote:<u></u><u></u></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div><div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBannermu5fqka"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBannermu5fqka"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBannermu5fqka"><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This Message Is From an External Sender </span><u></u><u></u></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBannermu5fqka"><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Hello all,<br><br>I set MKL_VERBOSE = 1, but observed no print output specific to the use of MKL. Does PETSc enable this verbose output?<br><br>Best,</span><u></u><u></u></div></div></div></div></div></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong</span><u></u><u></u></p></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"> </span><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b>From: </b>Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">pierre@joliv.et</span></a>><br><b>Date: </b>Friday, June 21, 2024 at 1:36<span style="font-family: Arial, sans-serif;"> </span>AM<br><b>To: </b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>><br><b>Cc: </b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">yongzhong.li@mail.utoronto.ca</span></a>>, <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">petsc-users@mcs.anl.gov</span></a>><br><b>Subject: </b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<u></u><u></u></p></div><table border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width: 681.438px; display: table; float: none;"><tbody><tr><td style="background: rgb(166, 166, 166); padding: 5.25pt 1.5pt;"></td><td width="100%" style="width: 647.438px; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt 5.25pt 11.25pt;"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">你通常不会收到来自</span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span><a href="mailto:pierre@joliv.et" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: purple;">pierre@joliv.et</span></a></span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">的电子邮件。</span><span><a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!eXBeeIXo9Yqgp2nypqwKYimLnGBZXnF4dXxgLM1UoOIO6n8nt3XlfgjVWLPWJh4UOa5NNpx-nrJb_H828XRQKUREfR2m69oCbxI$" target="_blank" style="color: purple; text-decoration: underline;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: purple;">了解这一点为什么很重要</span></a></span><span><u></u><u></u></span></div></div></div></div></div></div></div></td><td width="75" style="width: 56.25pt; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt;"></td></tr></tbody></table><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div style="margin-bottom: 12pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On 21 Jun 2024, at 6:42<span style="font-family: Arial, sans-serif;"> </span>AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">junchao.zhang@gmail.com</span></a>> wrote:<u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBanners71iuax"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBanners71iuax"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBanners71iuax"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This Message Is From an External Sender</span><u></u><u></u></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580pfptBanners71iuax"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">I remember there are some MKL env vars to print MKL routines called. </span><u></u><u></u></div></div></div></div></div></div></div></div></blockquote><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">The environment variable is MKL_VERBOSE<u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Thanks,<u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Pierre<u></u><u></u></div></div></div></div></div></div></div><div><div style="margin-bottom: 12pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Maybe we can try it to see what MKL routines are really used and then we can understand why some petsc functions did not speed up </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"><br clear="all"></span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">--Junchao Zhang</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">On Thu, Jun 20, 2024 at 10:39 PM Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">> wrote:</span><u></u><u></u></div></div></div></div></div></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerqdehbik" style="margin-top: 12pt; margin-bottom: 12pt; border-width: initial; border-style: initial; border-color: currentcolor; border-radius: 4px; min-width: 200px;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerqdehbik" style="margin-bottom: 0.75pt; float: left;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerqdehbik"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><u></u><u></u></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerqdehbik" style="margin-top: 1.5pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; background: rgb(208, 216, 220);"><span style="font-size: 1pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;">Hi Barry, sorry for my last results. I didn’t fully understand the stage profiling and logging in PETSc, now I only record KSPSolve() stage of my program. Some sample codes are as follow,<br><br> // Static variable to keep track of the stage counter</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> static int stageCounter = 1;</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> // Generate a unique stage name</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> std::ostringstream oss;</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> oss << "Stage " << stageCounter << " of Code";</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> std::string stageName = oss.str();</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> // Register the stage</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> PetscLogStage stagenum;</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> PetscLogStageRegister(stageName.c_str(), &stagenum);</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> PetscLogStagePush(stagenum);</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> <b>KSPSolve(*ksp_ptr, b, x);</b></span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> PetscLogStagePop();</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> stageCounter++;<br><br>I have attached my new logging results, there are 1 main stage and 4 other stages where each one is KSPSolve() call.<br><br>To provide some additional backgrounds, if you recall, I have been trying to get efficient iterative solution using multithreading. I found out by compiling PETSc with Intel MKL library instead of OpenBLAS, I am able to perform sparse matrix-vector multiplication faster, I am using MATSEQAIJMKL. This makes the shell matrix vector product in each iteration scale well with the #of threads. However, I found out the total GMERS solve time (~KSPSolve() time) is not scaling well the #of threads.<br><br>From the logging results I learned that when performing KSPSolve(), there are some CPU overheads in PCApply() and KSPGMERSOrthog(). I ran my programs using different number of threads and plotted the time consumption for PCApply() and KSPGMERSOrthog() against #of thread. I found out these two operations are not scaling with the threads at all! My results are attached as the pdf to give you a clear view.<br><br>My questions is,<br><br>From my understanding, in PCApply, MatSolve() is involved, KSPGMERSOrthog() will have many vector operations, so why these two parts can’t scale well with the # of threads when the intel MKL library is linked?<br><br>Thank you,<br>Yongzhong</span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">From: </span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Barry Smith <</span><a href="mailto:bsmith@petsc.dev" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">bsmith@petsc.dev</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Date: </b>Friday, June 14, 2024 at 11:36</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">AM<br><b>To: </b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Cc: </b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">>, </span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> <</span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">>, Piero Triverio <</span><a href="mailto:piero.triverio@utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">piero.triverio@utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Subject: </b>Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue</span><u></u><u></u></p></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> I am a bit confused. Without the initial guess computation, there are still a bunch of events I don't understand </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatTranspose 79 1.0 4.0598e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatMatMultSym 110 1.0 1.7419e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatMatMultNum 90 1.0 1.2640e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatMatMatMultSym 20 1.0 1.3049e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatRARtSym 25 1.0 1.2492e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatMatTrnMultSym 25 1.0 8.8265e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatMatTrnMultNum 25 1.0 2.4820e+02 1.0 6.83e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 275</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatTrnMatMultSym 10 1.0 7.2984e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatTrnMatMultNum 10 1.0 9.3128e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">in addition there are many more VecMAXPY then VecMDot (in GMRES they are each done the same number of times)</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">VecMDot 5588 1.0 1.7183e+03 1.0 2.06e+13 1.0 0.0e+00 0.0e+00 0.0e+00 8 10 0 0 0 8 10 0 0 0 12016</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">VecMAXPY 22412 1.0 8.4898e+03 1.0 4.17e+13 1.0 0.0e+00 0.0e+00 0.0e+00 39 20 0 0 0 39 20 0 0 0 4913</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Finally there are a huge number of </span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">MatMultAdd 258048 1.0 1.4178e+03 1.0 6.10e+13 1.0 0.0e+00 0.0e+00 0.0e+00 7 29 0 0 0 7 29 0 0 0 43025</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Are you making calls to all these routines? Are you doing this inside your MatMult() or before you call KSPSolve?</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">The reason I wanted you to make a simpler run without the initial guess code is that your events are far more complicated than would be produced by GMRES alone so it is not possible to understand the behavior you are seeing without fully understanding all the events happening in the code.</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> Barry</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div><div><div style="margin-bottom: 12pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">On Jun 14, 2024, at 1:19</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">AM, Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">> wrote:</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;">Thanks, I have attached the results without using any KSPGuess. At low frequency, the iteration steps are quite close to the one with KSPGuess, specifically <br><br> KSPGuess Object: 1 MPI process</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> type: fischer</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;"> Model 1, size 200<br><br>However, I found at higher frequency, the # of iteration steps are significant higher than the one with KSPGuess, I have attahced both of the results for your reference.<br><br>Moreover, could I ask why the one without the KSPGuess options can be used for a baseline comparsion? What are we comparing here? How does it relate to the performance issue/bottleneck I found? “</span><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">I have noticed that the time taken by </span></b><b><span style="font-size: 9pt; font-family: "Courier New";">KSPSolve</span></b><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> is </span></b><b><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">almost two times </span></b><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">greater than the CPU time for matrix-vector product multiplied by the number of iteration</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;">” <br><br>Thank you!<br>Yongzhong</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">From: </span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Barry Smith <</span><a href="mailto:bsmith@petsc.dev" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">bsmith@petsc.dev</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Date: </b>Thursday, June 13, 2024 at 2:14</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">PM<br><b>To: </b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Cc: </b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">>, </span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> <</span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">>, Piero Triverio <</span><a href="mailto:piero.triverio@utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">piero.triverio@utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Subject: </b>Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue</span><u></u><u></u></p></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> Can you please run the same thing without the KSPGuess option(s) for a baseline comparison?</span><u></u><u></u></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> Thanks</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> Barry</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div style="margin-bottom: 12pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">On Jun 13, 2024, at 1:27</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">PM, Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">> wrote:</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div><div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerxs2204y"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerxs2204y"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerxs2204y"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span><u></u><u></u></div></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987pfptBannerxs2204y"><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica, sans-serif;">Hi Matt,<br><br>I have rerun the program with the keys you provided. The system output when performing ksp solve and the final petsc log output were stored in a .txt file attached for your reference.<br><br>Thanks!<br>Yongzhong</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987mail-editor-reference-message-container"><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">From: </span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Matthew Knepley <</span><a href="mailto:knepley@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">knepley@gmail.com</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Date: </b>Wednesday, June 12, 2024 at 6:46</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">PM<br><b>To: </b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Cc: </b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">>, </span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> <</span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">>, Piero Triverio <</span><a href="mailto:piero.triverio@utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">piero.triverio@utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">><br><b>Subject: </b>Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue</span><u></u><u></u></p></div><table border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width: 426.047px; display: table; float: none;"><tbody><tr><td style="background: rgb(166, 166, 166); padding: 5.25pt 1.5pt;"></td><td width="100%" style="width: 392.047px; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt 5.25pt 11.25pt;"><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">你通常不会收到来自</span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span><a href="mailto:knepley@gmail.com" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: purple;">knepley@gmail.com</span></a></span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">的电子邮件。</span><span><a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!djGfJnEhNJROfsMsBJy5u_KoRKbug55xZ64oHKUFnH2cWku_Th1hwt4TDdoMd8pWYVDzJeqJslMNZwpO3y0Et94d31qk-oCEwo4$" target="_blank" style="color: purple; text-decoration: underline;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: purple;">了解这一点为什么很重要</span></a></span><span><u></u><u></u></span></div></div></div></div></div></div></div></div></div></td><td width="75" style="width: 56.25pt; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt;"></td></tr></tbody></table><div><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">On Wed, Jun 12, 2024 at 6:36</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">PM Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">> wrote:</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; font-family: Helvetica, sans-serif; color: white;">Dear PETSc’s developers, I hope this email finds you well. I am currently working on a project using PETSc and have encountered a performance issue with the KSPSolve function. Specifically, I have noticed that the time taken by KSPSolve is </span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; font-family: Helvetica, sans-serif; color: white;">ZjQcmQRYFpfptBannerStart</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww" style="margin-top: 12pt; margin-bottom: 12pt; border-width: initial; border-style: initial; border-color: currentcolor; border-radius: 4px; min-width: 200px;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww" style="margin-bottom: 0.75pt; float: left;"><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww"><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><u></u><u></u></div></div></div></div></div></div></div></div></div><div id="m_-3762595706006288767m_-6213770648425822909m_-5045176430152657136m_-7305460844614688919m_2643581689528628203m_6159968039748304261m_1825240119000957322m_-6075196701903926580m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww" style="margin-top: 1.5pt;"><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><u></u><u></u></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; background: rgb(208, 216, 220);"><span style="font-size: 1pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; font-family: Helvetica, sans-serif; color: white;">ZjQcmQRYFpfptBannerEnd</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Dear </span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">PETSc’s developers</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">,</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">I hope this email finds you well.</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">I am currently working on a project using PETSc and have encountered a performance issue with the </span><span style="font-size: 9pt; font-family: "Courier New";">KSPSolve</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> function. Specifically, <b>I have noticed that the time taken by </b></span><b><span style="font-size: 9pt; font-family: "Courier New";">KSPSolve</span></b><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> is </span></b><b><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">almost two times </span></b><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">greater than the CPU time for matrix-vector product multiplied by the number of iteration steps</span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">. </span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">I use C++ chrono to record CPU time.</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">For context, I am using a shell system matrix</span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;"> A</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">. Despite my efforts to </span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">parallelize</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> the matrix-vector product</span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;"> (Ax)</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">, the overall solve time remains higher</span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;"> than the matrix vector product per iteration indicates when multiple threads were used</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">. Here are a few details of my setup:</span><u></u><u></u></div></div></div></div></div></div></div></div><ul type="disc" style="margin-bottom: 0cm;"><li class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Matrix Type</span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">: Shell system matrix</span><u></u><u></u></li><li class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Preconditioner</span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">: </span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">Shell PC</span><u></u><u></u></li><li class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Parallel Environment</span></b><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">: </span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">Using Intel MKL as PETSc’s BLAS/LAPACK library, multithreading is enabled</span><u></u><u></u></li></ul><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">I have considered several potential reasons, such as preconditioner setup, additional solver operations, and the inherent overhead of using a shell system matrix. <b>However, since KSPSolve is a high-level API, I have been unable to pinpoint the exact cause of the increased solve time.</b></span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">Have you observed the same issue? </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Could you</span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;"> please </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">provide some </span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">experience </span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">on how to diagnose and address this performance discrepancy? Any insights or recommendations you could offer would be greatly appreciated.</span><u></u><u></u></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">For any performance question like this, we need to see the output of your code run with</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason -log_view</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> Thanks,</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> Matt</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Thank you for your time and assistance.</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">Best</span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;"> regards,</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica, sans-serif;">Yongzhong</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">-----------------------------------------------------------</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Yongzhong Li</span></b><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">PhD student | Electromagnetics Group</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Department of Electrical & Computer Engineering</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">University of Toronto</span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.us/v3/__http://www.modelics.org__;!!G_uCfscf7eWS!cuLttMJEcegaqu461Bt4QLsO4fASfLM5vjRbtyNhWJQiInbjgNwkGNdkFE1ebSbFjOUatYB0-jd2yQWMWzqkDFFjwMvNl3ZKAr8$" target="_blank" style="color: purple; text-decoration: underline;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(5, 99, 193);">http://www.modelics.org</span></a><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div></div></div></blockquote></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"><br clear="all"></span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">-- </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</span><u></u><u></u></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!djGfJnEhNJROfsMsBJy5u_KoRKbug55xZ64oHKUFnH2cWku_Th1hwt4TDdoMd8pWYVDzJeqJslMNZwpO3y0Et94d31qkNOuenGA$" target="_blank" style="color: purple; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;">https://www.cse.buffalo.edu/~knepley/</span></a><u></u><u></u></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"><ksp_petsc_log.txt></span><u></u><u></u></div></div></div></div></div></div></div></div></blockquote></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"> </span><u></u><u></u></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;"><ksp_petsc_log.txt><ksp_petsc_log_noguess.txt></span><u></u><u></u></div></div></div></div></div></div></div></blockquote></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></div></blockquote></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div></div></div></div></div></div></blockquote></div></div></div></div></div></div></blockquote></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br clear="all"><u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">-- <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<u></u><u></u></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fVvbGldqcUV5ju4jpu5oGmt-VjITi5JpCJzhHxpbgsERLVYZzglpxKOOyrBRGxjRxp7vWHwt3SnINFOQErR1Z8kcDcf3cNeD9Gw$" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">https://www.cse.buffalo.edu/~knepley/</span></a><u></u><u></u></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div></div></div></blockquote></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br clear="all"><u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span>--</span><span> </span><u></u><u></u></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<u></u><u></u></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aQJpmm5W6l6FUiumnIPmkouzwzNUfx-Dyq04i1O2KS_InQGk6qjI7wUir0Hx6QEUQE2AMiJDsez3x2Os2C2d$" target="_blank" style="color: purple; text-decoration: underline;"><span style="color: purple;">https://www.cse.buffalo.edu/~knepley/</span></a><u></u><u></u></div></div></div></div></div></div></blockquote></div></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"> <u></u><u></u></div></div></div></div></div></div></div></blockquote></div></div><span id="m_-3762595706006288767m_-6213770648425822909cid:31A29931-173E-4898-AC38-E27564FEFC0B"><petsc_log_comparison.txt></span></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div><br></div></body></html>