<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
How do you set the variable? $ MKL_VERBOSE=1 ./ex1 -ksp_converged_reason MKL_VERBOSE oneMKL 2024. 0 Update 1 Product build 20240215 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Lnx 2. 80GHz
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerncw9aqb" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerncw9aqb" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerncw9aqb" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerncw9aqb" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html><head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerncw9aqb { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonncw9aqb:hover, .pfptPrimaryButtonncw9aqb:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonncw9aqb:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">How do you set the variable?<div><br></div><div><div>$ MKL_VERBOSE=1 ./ex1 -ksp_converged_reason</div><div>MKL_VERBOSE oneMKL 2024.0 Update 1 Product build 20240215 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Lnx 2.80GHz lp64 intel_thread</div><div>MKL_VERBOSE DDOT(10,0x22127c0,1,0x22127c0,1) 2.02ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</div><div>MKL_VERBOSE DSCAL(10,0x7ffc9fb4ff08,0x22127c0,1) 12.67us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</div><div>MKL_VERBOSE DDOT(10,0x22127c0,1,0x2212840,1) 1.52us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</div><div>MKL_VERBOSE DDOT(10,0x2212840,1,0x2212840,1) 167ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1</div><div>[...]</div><div><br><blockquote type="cite"><div>On 21 Jun 2024, at 7:37 PM, Yongzhong Li <yongzhong.li@mail.utoronto.ca> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div dir="ltr" id="pfptBannermu5fqka" style="color-scheme: revert !important; writing-mode: revert !important; -webkit-ruby-position: revert !important; color: revert !important; font-family: revert !important; font-feature-settings: revert !important; font-kerning: revert !important; font-optical-sizing: revert !important; font-palette: revert !important; font-size: revert !important; font-size-adjust: revert !important; font-stretch: revert !important; font-style: revert !important; font-synthesis-small-caps: revert !important; font-synthesis-style: revert !important; font-synthesis-weight: revert !important; font-variant-alternates: revert !important; font-variant-caps: revert !important; font-variant-east-asian: revert !important; font-variant-emoji: revert !important; font-variant-ligatures: revert !important; font-variant-numeric: revert !important; font-variant-position: revert !important; font-variation-settings: revert !important; font-weight: revert !important; text-orientation: revert !important; text-rendering: revert !important; zoom: revert !important; -webkit-font-smoothing: revert !important; -webkit-locale: revert !important; -webkit-text-size-adjust: revert !important; -webkit-text-zoom: revert !important; letter-spacing: revert !important; text-autospace: revert !important; text-spacing-trim: revert !important; accent-color: revert !important; align-content: revert !important; align-items: revert !important; align-self: revert !important; alignment-baseline: revert !important; alt: revert !important; animation-composition: revert !important; animation: revert !important; animation-timeline: revert !important; appearance: revert !important; aspect-ratio: revert !important; backface-visibility: revert !important; background-attachment: revert !important; background-blend-mode: revert !important; background-image: revert !important; background-position: revert !important; background-repeat: revert !important; background-size: revert !important; baseline-shift: revert !important; block-step-insert: revert !important; block-step-size: revert !important; border-collapse: revert !important; border-image: revert !important; box-sizing: revert !important; break-after: revert !important; break-before: revert !important; break-inside: revert !important; buffered-rendering: revert !important; caption-side: revert !important; caret-color: revert !important; clear: revert !important; clip: revert !important; clip-path: revert !important; clip-rule: revert !important; color-interpolation: revert !important; color-interpolation-filters: revert !important; column-count: revert !important; column-fill: revert !important; column-gap: revert !important; column-rule-color: revert !important; column-rule-style: revert !important; column-rule-width: revert !important; column-span: revert !important; column-width: revert !important; contain: revert !important; container-name: revert !important; container-type: revert !important; content: revert !important; content-visibility: revert !important; counter-increment: revert !important; counter-reset: revert !important; counter-set: revert !important; cursor: revert !important; cx: revert !important; cy: revert !important; dominant-baseline: revert !important; empty-cells: revert !important; fill: revert !important; fill-opacity: revert !important; fill-rule: revert !important; filter: revert !important; flex: revert !important; flex-flow: revert !important; float: revert !important; flood-color: revert !important; flood-opacity: revert !important; glyph-orientation-horizontal: revert !important; glyph-orientation-vertical: revert !important; grid: revert !important; grid-column-end: revert !important; grid-column-start: revert !important; grid-row-end: revert !important; grid-row-start: revert !important; hanging-punctuation: revert !important; hyphenate-character: revert !important; hyphens: revert !important; image-orientation: revert !important; image-rendering: revert !important; input-security: revert !important; isolation: revert !important; justify-content: revert !important; justify-items: revert !important; justify-self: revert !important; kerning: revert !important; lighting-color: revert !important; line-break: revert !important; line-height: revert !important; list-style: revert !important; margin-trim: revert !important; marker: revert !important; mask: revert !important; mask-size: revert !important; mask-type: revert !important; masonry-auto-flow: revert !important; math-style: revert !important; mix-blend-mode: revert !important; object-fit: revert !important; object-position: revert !important; offset-anchor: revert !important; offset-distance: revert !important; offset-path: revert !important; offset-position: revert !important; offset-rotate: revert !important; order: revert !important; orphans: revert !important; outline: revert !important; outline-offset: revert !important; overflow-anchor: revert !important; overflow-wrap: revert !important; overflow: revert !important; page: revert !important; paint-order: revert !important; perspective: revert !important; perspective-origin: revert !important; pointer-events: revert !important; position: revert !important; print-color-adjust: revert !important; quotes: revert !important; r: revert !important; resize: revert !important; rotate: revert !important; row-gap: revert !important; rx: revert !important; ry: revert !important; scale: revert !important; scroll-behavior: revert !important; scroll-snap-align: revert !important; scroll-snap-stop: revert !important; scroll-snap-type: revert !important; scroll-timeline: revert !important; scrollbar-color: revert !important; scrollbar-gutter: revert !important; scrollbar-width: revert !important; shape-image-threshold: revert !important; shape-margin: revert !important; shape-outside: revert !important; shape-rendering: revert !important; size: revert !important; speak-as: revert !important; stop-color: revert !important; stop-opacity: revert !important; stroke: revert !important; stroke-color: revert !important; stroke-dasharray: revert !important; stroke-dashoffset: revert !important; stroke-linecap: revert !important; stroke-linejoin: revert !important; stroke-miterlimit: revert !important; stroke-opacity: revert !important; stroke-width: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: left !important; text-align-last: revert !important; text-anchor: revert !important; text-box-edge: revert !important; text-box-trim: revert !important; text-decoration-color: revert !important; text-decoration: revert !important; text-decoration-skip-ink: revert !important; text-decoration-style: revert !important; text-decoration-thickness: revert !important; text-emphasis-color: revert !important; text-emphasis-position: revert !important; text-emphasis-style: revert !important; text-group-align: revert !important; text-indent: revert !important; text-justify: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-underline-offset: revert !important; text-underline-position: revert !important; text-wrap: revert !important; touch-action: revert !important; transform: revert !important; transform-box: revert !important; transform-origin: revert !important; transform-style: revert !important; transition: revert !important; translate: revert !important; vector-effect: revert !important; vertical-align: revert !important; view-timeline: revert !important; view-transition-name: revert !important; white-space: revert !important; widows: revert !important; will-change: revert !important; word-break: revert !important; x: revert !important; y: revert !important; z-index: revert !important; -apple-color-filter: revert !important; -apple-pay-button-style: revert !important; -apple-pay-button-type: revert !important; -internal-text-autosizing-status: revert !important; border-spacing: revert !important; -webkit-box-align: revert !important; -webkit-box-decoration-break: revert !important; -webkit-box-direction: revert !important; -webkit-box-flex: revert !important; -webkit-box-flex-group: revert !important; -webkit-box-lines: revert !important; -webkit-box-ordinal-group: revert !important; -webkit-box-orient: revert !important; -webkit-box-pack: revert !important; -webkit-box-reflect: revert !important; -webkit-column-axis: revert !important; -webkit-column-progression: revert !important; -webkit-cursor-visibility: revert !important; -webkit-font-size-delta: revert !important; -webkit-hyphenate-limit-after: revert !important; -webkit-hyphenate-limit-before: revert !important; -webkit-hyphenate-limit-lines: revert !important; -webkit-initial-letter: revert !important; -webkit-line-align: revert !important; -webkit-line-box-contain: revert !important; -webkit-line-clamp: revert !important; -webkit-line-grid: revert !important; -webkit-line-snap: revert !important; -webkit-marquee-direction: revert !important; -webkit-marquee-increment: revert !important; -webkit-marquee-repetition: revert !important; -webkit-marquee-speed: revert !important; -webkit-marquee-style: revert !important; -webkit-nbsp-mode: revert !important; -webkit-rtl-ordering: revert !important; -webkit-text-decorations-in-effect: revert !important; -webkit-text-fill-color: revert !important; -webkit-text-security: revert !important; -webkit-text-stroke-color: revert !important; -webkit-text-stroke-width: revert !important; -webkit-user-drag: revert !important; -webkit-user-modify: revert !important; -webkit-user-select: revert !important; word-spacing: revert !important; backdrop-filter: revert !important; background-clip: revert !important; background-origin: revert !important; box-shadow: revert !important; mask-composite: revert !important; mask-mode: revert !important; text-combine-upright: revert !important; -webkit-backdrop-filter: revert !important; -webkit-background-clip: revert !important; -webkit-background-origin: revert !important; -webkit-box-shadow: revert !important; -webkit-mask: revert !important; -webkit-mask-composite: revert !important; -webkit-mask-source-type: revert !important; -webkit-text-combine: revert !important; display: block !important; visibility: revert !important; opacity: revert !important; background-color: rgb(208, 216, 220) !important; block-size: revert !important; border-block: revert !important; border-bottom-color: revert !important; border-bottom-style: revert !important; border-bottom-width: revert !important; border-end-end-radius: revert !important; border-end-start-radius: revert !important; border-inline: revert !important; border-left-color: revert !important; border-left-style: revert !important; border-left-width: revert !important; border-right-color: revert !important; border-right-style: revert !important; border-right-width: revert !important; border-start-end-radius: revert !important; border-start-start-radius: revert !important; inset: revert !important; contain-intrinsic-block-size: revert !important; contain-intrinsic-size: revert !important; contain-intrinsic-inline-size: revert !important; height: revert !important; inline-size: revert !important; inset-block: revert !important; inset-inline: revert !important; margin-block: revert !important; margin-inline: revert !important; max-block-size: revert !important; max-height: revert !important; max-inline-size: revert !important; max-width: revert !important; min-block-size: revert !important; min-height: revert !important; min-inline-size: revert !important; overscroll-behavior-block: revert !important; overscroll-behavior-inline: revert !important; overscroll-behavior: revert !important; padding-block: revert !important; padding-inline: revert !important; scroll-margin-block: revert !important; scroll-margin: revert !important; scroll-margin-inline: revert !important; scroll-padding-block: revert !important; scroll-padding: revert !important; scroll-padding-inline: revert !important; width: revert !important; margin: 16px 0px !important; padding: 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; border-top-width: 4px !important; border-top-style: solid !important; border-top-color: rgb(144, 164, 174) !important;"><div id="pfptBannermu5fqka" style="color-scheme: unset !important; writing-mode: unset !important; -webkit-ruby-position: unset !important; color: unset !important; font-family: unset !important; font-feature-settings: unset !important; font-kerning: unset !important; font-optical-sizing: unset !important; font-palette: unset !important; font-size: unset !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-synthesis-small-caps: unset !important; font-synthesis-style: unset !important; font-synthesis-weight: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-emoji: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-variant-position: unset !important; font-variation-settings: unset !important; font-weight: unset !important; text-orientation: unset !important; text-rendering: unset !important; zoom: unset !important; -webkit-font-smoothing: unset !important; -webkit-locale: unset !important; -webkit-text-size-adjust: unset !important; -webkit-text-zoom: unset !important; letter-spacing: unset !important; text-autospace: unset !important; text-spacing-trim: unset !important; accent-color: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; alignment-baseline: unset !important; alt: unset !important; animation-composition: unset !important; animation: unset !important; animation-timeline: unset !important; appearance: unset !important; aspect-ratio: unset !important; backface-visibility: unset !important; background: unset !important; background-blend-mode: unset !important; baseline-shift: unset !important; block-step-insert: unset !important; block-step-size: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; buffered-rendering: unset !important; caption-side: unset !important; caret-color: unset !important; clear: unset !important; clip: unset !important; clip-path: unset !important; clip-rule: unset !important; color-interpolation: unset !important; color-interpolation-filters: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; contain: unset !important; container-name: unset !important; container-type: unset !important; content: unset !important; content-visibility: unset !important; counter-increment: unset !important; counter-reset: unset !important; counter-set: unset !important; cursor: unset !important; cx: unset !important; cy: unset !important; dominant-baseline: unset !important; empty-cells: unset !important; fill: unset !important; fill-opacity: unset !important; fill-rule: unset !important; filter: unset !important; flex: unset !important; flex-flow: unset !important; float: left !important; flood-color: unset !important; flood-opacity: unset !important; glyph-orientation-horizontal: unset !important; glyph-orientation-vertical: unset !important; grid: unset !important; grid-column-end: unset !important; grid-column-start: unset !important; grid-row-end: unset !important; grid-row-start: unset !important; hanging-punctuation: unset !important; hyphenate-character: unset !important; hyphens: unset !important; image-orientation: unset !important; image-rendering: unset !important; input-security: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; justify-self: unset !important; kerning: unset !important; lighting-color: unset !important; line-break: unset !important; line-height: unset !important; list-style: unset !important; margin-trim: unset !important; marker: unset !important; mask: unset !important; mask-size: unset !important; mask-type: unset !important; masonry-auto-flow: unset !important; math-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; offset-anchor: unset !important; offset-distance: unset !important; offset-path: unset !important; offset-position: unset !important; offset-rotate: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; outline-offset: unset !important; overflow-anchor: unset !important; overflow-wrap: unset !important; overflow: unset !important; page: unset !important; paint-order: unset !important; perspective: unset !important; perspective-origin: unset !important; pointer-events: unset !important; position: unset !important; print-color-adjust: unset !important; quotes: unset !important; r: unset !important; resize: unset !important; rotate: unset !important; row-gap: unset !important; rx: unset !important; ry: unset !important; scale: unset !important; scroll-behavior: unset !important; scroll-snap-align: unset !important; scroll-snap-stop: unset !important; scroll-snap-type: unset !important; scroll-timeline: unset !important; scrollbar-color: unset !important; scrollbar-gutter: unset !important; scrollbar-width: unset !important; shape-image-threshold: unset !important; shape-margin: unset !important; shape-outside: unset !important; shape-rendering: unset !important; size: unset !important; speak-as: unset !important; stop-color: unset !important; stop-opacity: unset !important; stroke: unset !important; stroke-color: unset !important; stroke-dasharray: unset !important; stroke-dashoffset: unset !important; stroke-linecap: unset !important; stroke-linejoin: unset !important; stroke-miterlimit: unset !important; stroke-opacity: unset !important; stroke-width: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-align-last: unset !important; text-anchor: unset !important; text-box-edge: unset !important; text-box-trim: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-skip-ink: unset !important; text-decoration-style: unset !important; text-decoration-thickness: unset !important; text-emphasis-color: unset !important; text-emphasis-position: unset !important; text-emphasis-style: unset !important; text-group-align: unset !important; text-indent: unset !important; text-justify: unset !important; text-overflow: unset !important; text-shadow: unset !important; text-transform: unset !important; text-underline-offset: unset !important; text-underline-position: unset !important; text-wrap: unset !important; touch-action: unset !important; transform: unset !important; transform-box: unset !important; transform-origin: unset !important; transform-style: unset !important; transition: unset !important; translate: unset !important; vector-effect: unset !important; vertical-align: unset !important; view-timeline: unset !important; view-transition-name: unset !important; white-space: unset !important; widows: unset !important; will-change: unset !important; word-break: unset !important; x: unset !important; y: unset !important; z-index: unset !important; -apple-color-filter: unset !important; -apple-pay-button-style: unset !important; -apple-pay-button-type: unset !important; -internal-text-autosizing-status: unset !important; border-spacing: unset !important; -webkit-box-align: unset !important; -webkit-box-decoration-break: unset !important; -webkit-box-direction: unset !important; -webkit-box-flex: unset !important; -webkit-box-flex-group: unset !important; -webkit-box-lines: unset !important; -webkit-box-ordinal-group: unset !important; -webkit-box-orient: unset !important; -webkit-box-pack: unset !important; -webkit-box-reflect: unset !important; -webkit-column-axis: unset !important; -webkit-column-progression: unset !important; -webkit-cursor-visibility: unset !important; -webkit-font-size-delta: unset !important; -webkit-hyphenate-limit-after: unset !important; -webkit-hyphenate-limit-before: unset !important; -webkit-hyphenate-limit-lines: unset !important; -webkit-initial-letter: unset !important; -webkit-line-align: unset !important; -webkit-line-box-contain: unset !important; -webkit-line-clamp: unset !important; -webkit-line-grid: unset !important; -webkit-line-snap: unset !important; -webkit-marquee-direction: unset !important; -webkit-marquee-increment: unset !important; -webkit-marquee-repetition: unset !important; -webkit-marquee-speed: unset !important; -webkit-marquee-style: unset !important; -webkit-nbsp-mode: unset !important; -webkit-rtl-ordering: unset !important; -webkit-text-decorations-in-effect: unset !important; -webkit-text-fill-color: unset !important; -webkit-text-security: unset !important; -webkit-text-stroke-color: unset !important; -webkit-text-stroke-width: unset !important; -webkit-user-drag: unset !important; -webkit-user-modify: unset !important; -webkit-user-select: unset !important; word-spacing: unset !important; backdrop-filter: unset !important; box-shadow: unset !important; mask-composite: unset !important; mask-mode: unset !important; text-combine-upright: unset !important; -webkit-backdrop-filter: unset !important; -webkit-background-clip: unset !important; -webkit-background-origin: unset !important; -webkit-box-shadow: unset !important; -webkit-mask: unset !important; -webkit-mask-composite: unset !important; -webkit-mask-source-type: unset !important; -webkit-text-combine: unset !important; display: block !important; visibility: unset !important; opacity: unset !important; block-size: unset !important; border-block: unset !important; border-radius: unset !important; border-end-end-radius: unset !important; border-end-start-radius: unset !important; border-inline: unset !important; border-start-end-radius: unset !important; border-start-start-radius: unset !important; inset: unset !important; contain-intrinsic-block-size: unset !important; contain-intrinsic-size: unset !important; contain-intrinsic-inline-size: unset !important; height: unset !important; inline-size: unset !important; inset-block: unset !important; inset-inline: unset !important; margin-block: unset !important; margin-inline: unset !important; max-block-size: unset !important; max-height: unset !important; max-inline-size: unset !important; min-block-size: unset !important; min-height: unset !important; min-inline-size: unset !important; min-width: unset !important; overscroll-behavior-block: unset !important; overscroll-behavior-inline: unset !important; overscroll-behavior: unset !important; padding-block: unset !important; padding: unset !important; padding-inline: unset !important; scroll-margin-block: unset !important; scroll-margin: unset !important; scroll-margin-inline: unset !important; scroll-padding-block: unset !important; scroll-padding: unset !important; scroll-padding-inline: unset !important; width: unset !important; margin: 0px 0px 1px !important; max-width: 600px !important;"><div id="pfptBannermu5fqka" style="color-scheme: unset !important; writing-mode: unset !important; -webkit-ruby-position: unset !important; font-family: Arial, sans-serif !important; font-feature-settings: unset !important; font-kerning: unset !important; font-optical-sizing: unset !important; font-palette: unset !important; font-size: 14px !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-synthesis-small-caps: unset !important; font-synthesis-style: unset !important; font-synthesis-weight: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-variant-position: unset !important; font-variation-settings: unset !important; font-weight: bold !important; text-orientation: unset !important; text-rendering: unset !important; zoom: unset !important; -webkit-font-smoothing: unset !important; -webkit-locale: unset !important; -webkit-text-zoom: unset !important; letter-spacing: unset !important; accent-color: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; alignment-baseline: unset !important; alt: unset !important; animation-composition: unset !important; animation: unset !important; appearance: unset !important; aspect-ratio: unset !important; backface-visibility: unset !important; background-attachment: unset !important; background-blend-mode: unset !important; background-image: unset !important; background-position: unset !important; background-repeat: unset !important; background-size: unset !important; baseline-shift: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; buffered-rendering: unset !important; caption-side: unset !important; caret-color: unset !important; clear: unset !important; clip: unset !important; clip-path: unset !important; clip-rule: unset !important; color-interpolation: unset !important; color-interpolation-filters: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; contain: unset !important; container-name: unset !important; container-type: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; counter-set: unset !important; cursor: unset !important; cx: unset !important; cy: unset !important; dominant-baseline: unset !important; empty-cells: unset !important; fill: unset !important; fill-opacity: unset !important; fill-rule: unset !important; filter: unset !important; flex: unset !important; flex-flow: unset !important; float: unset !important; flood-color: unset !important; flood-opacity: unset !important; glyph-orientation-horizontal: unset !important; glyph-orientation-vertical: unset !important; grid: unset !important; grid-column-end: unset !important; grid-column-start: unset !important; grid-row-end: unset !important; grid-row-start: unset !important; hanging-punctuation: unset !important; hyphenate-character: unset !important; hyphens: unset !important; image-orientation: unset !important; image-rendering: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; justify-self: unset !important; kerning: unset !important; lighting-color: unset !important; line-break: unset !important; line-height: 18px !important; list-style: unset !important; margin-trim: unset !important; marker: unset !important; mask: unset !important; mask-type: unset !important; math-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; offset-anchor: unset !important; offset-distance: unset !important; offset-path: unset !important; offset-position: unset !important; offset-rotate: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; outline-offset: unset !important; overflow-wrap: unset !important; overflow: unset !important; page: unset !important; paint-order: unset !important; perspective: unset !important; perspective-origin: unset !important; pointer-events: unset !important; position: unset !important; print-color-adjust: unset !important; quotes: unset !important; r: unset !important; resize: unset !important; rotate: unset !important; row-gap: unset !important; rx: unset !important; ry: unset !important; scale: unset !important; scroll-behavior: unset !important; scroll-snap-align: unset !important; scroll-snap-stop: unset !important; scroll-snap-type: unset !important; shape-image-threshold: unset !important; shape-margin: unset !important; shape-outside: unset !important; shape-rendering: unset !important; size: unset !important; speak-as: unset !important; stop-color: unset !important; stop-opacity: unset !important; stroke: unset !important; stroke-color: unset !important; stroke-dasharray: unset !important; stroke-dashoffset: unset !important; stroke-linecap: unset !important; stroke-linejoin: unset !important; stroke-miterlimit: unset !important; stroke-opacity: unset !important; stroke-width: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-align-last: unset !important; text-anchor: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-skip-ink: unset !important; text-decoration-style: unset !important; text-decoration-thickness: unset !important; text-emphasis-color: unset !important; text-emphasis-position: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-shadow: unset !important; text-transform: unset !important; text-underline-offset: unset !important; text-underline-position: unset !important; text-wrap: unset !important; touch-action: unset !important; transform: unset !important; transform-box: unset !important; transform-origin: unset !important; transform-style: unset !important; transition: unset !important; translate: unset !important; vector-effect: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; will-change: unset !important; word-break: unset !important; x: unset !important; y: unset !important; z-index: unset !important; -apple-color-filter: unset !important; -apple-pay-button-style: unset !important; -apple-pay-button-type: unset !important; border-spacing: unset !important; -webkit-box-align: unset !important; -webkit-box-decoration-break: unset !important; -webkit-box-direction: unset !important; -webkit-box-flex: unset !important; -webkit-box-flex-group: unset !important; -webkit-box-lines: unset !important; -webkit-box-ordinal-group: unset !important; -webkit-box-orient: unset !important; -webkit-box-pack: unset !important; -webkit-box-reflect: unset !important; -webkit-column-axis: unset !important; -webkit-column-progression: unset !important; -webkit-cursor-visibility: unset !important; -webkit-hyphenate-limit-after: unset !important; -webkit-hyphenate-limit-before: unset !important; -webkit-hyphenate-limit-lines: unset !important; -webkit-initial-letter: unset !important; -webkit-line-align: unset !important; -webkit-line-box-contain: unset !important; -webkit-line-clamp: unset !important; -webkit-line-grid: unset !important; -webkit-line-snap: unset !important; -webkit-nbsp-mode: unset !important; -webkit-rtl-ordering: unset !important; -webkit-text-decorations-in-effect: unset !important; -webkit-text-fill-color: unset !important; -webkit-text-security: unset !important; -webkit-text-stroke-color: unset !important; -webkit-text-stroke-width: unset !important; -webkit-user-drag: unset !important; -webkit-user-modify: unset !important; -webkit-user-select: unset !important; word-spacing: unset !important; background-clip: unset !important; background-origin: unset !important; box-shadow: unset !important; mask-composite: unset !important; mask-mode: unset !important; text-combine-upright: unset !important; -webkit-backdrop-filter: unset !important; -webkit-background-clip: unset !important; -webkit-background-origin: unset !important; -webkit-box-shadow: unset !important; mask-size: unset !important; -webkit-mask: unset !important; -webkit-mask-composite: unset !important; -webkit-mask-source-type: unset !important; -webkit-text-combine: unset !important; display: block !important; visibility: visible !important; opacity: unset !important; background-color: rgb(208, 216, 220) !important; block-size: unset !important; border-block: unset !important; border-radius: unset !important; border-end-end-radius: unset !important; border-end-start-radius: unset !important; border-inline: unset !important; border-start-end-radius: unset !important; border-start-start-radius: unset !important; inset: unset !important; contain-intrinsic-block-size: unset !important; contain-intrinsic-size: unset !important; contain-intrinsic-inline-size: unset !important; height: unset !important; inline-size: unset !important; inset-block: unset !important; inset-inline: unset !important; margin-block: unset !important; margin: unset !important; margin-inline: unset !important; max-block-size: unset !important; max-height: unset !important; max-inline-size: unset !important; max-width: unset !important; min-block-size: unset !important; min-height: unset !important; min-inline-size: unset !important; min-width: unset !important; overscroll-behavior-block: unset !important; overscroll-behavior-inline: unset !important; overscroll-behavior: unset !important; padding-block: unset !important; padding: unset !important; padding-inline: unset !important; scroll-margin-block: unset !important; scroll-margin: unset !important; scroll-margin-inline: unset !important; scroll-padding-block: unset !important; scroll-padding: unset !important; scroll-padding-inline: unset !important; width: unset !important;">This Message Is From an External Sender<span class="Apple-converted-space"> </span></div><div id="pfptBannermu5fqka" style="color-scheme: unset !important; writing-mode: unset !important; -webkit-ruby-position: unset !important; font-family: Arial, sans-serif !important; font-feature-settings: unset !important; font-kerning: unset !important; font-optical-sizing: unset !important; font-palette: unset !important; font-size: 12px !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-synthesis-small-caps: unset !important; font-synthesis-style: unset !important; font-synthesis-weight: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-variant-position: unset !important; font-variation-settings: unset !important; font-weight: unset !important; text-orientation: unset !important; text-rendering: unset !important; zoom: unset !important; -webkit-font-smoothing: unset !important; -webkit-locale: unset !important; -webkit-text-zoom: unset !important; letter-spacing: unset !important; accent-color: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; alignment-baseline: unset !important; alt: unset !important; animation-composition: unset !important; animation: unset !important; appearance: unset !important; aspect-ratio: unset !important; backface-visibility: unset !important; background-attachment: unset !important; background-blend-mode: unset !important; background-image: unset !important; background-position: unset !important; background-repeat: unset !important; background-size: unset !important; baseline-shift: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; buffered-rendering: unset !important; caption-side: unset !important; caret-color: unset !important; clear: unset !important; clip: unset !important; clip-path: unset !important; clip-rule: unset !important; color-interpolation: unset !important; color-interpolation-filters: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; contain: unset !important; container-name: unset !important; container-type: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; counter-set: unset !important; cursor: unset !important; cx: unset !important; cy: unset !important; dominant-baseline: unset !important; empty-cells: unset !important; fill: unset !important; fill-opacity: unset !important; fill-rule: unset !important; filter: unset !important; flex: unset !important; flex-flow: unset !important; float: unset !important; flood-color: unset !important; flood-opacity: unset !important; glyph-orientation-horizontal: unset !important; glyph-orientation-vertical: unset !important; grid: unset !important; grid-column-end: unset !important; grid-column-start: unset !important; grid-row-end: unset !important; grid-row-start: unset !important; hanging-punctuation: unset !important; hyphenate-character: unset !important; hyphens: unset !important; image-orientation: unset !important; image-rendering: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; justify-self: unset !important; kerning: unset !important; lighting-color: unset !important; line-break: unset !important; line-height: 18px !important; list-style: unset !important; margin-trim: unset !important; marker: unset !important; mask: unset !important; mask-type: unset !important; math-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; offset-anchor: unset !important; offset-distance: unset !important; offset-path: unset !important; offset-position: unset !important; offset-rotate: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; outline-offset: unset !important; overflow-wrap: unset !important; overflow: unset !important; page: unset !important; paint-order: unset !important; perspective: unset !important; perspective-origin: unset !important; pointer-events: unset !important; position: unset !important; print-color-adjust: unset !important; quotes: unset !important; r: unset !important; resize: unset !important; rotate: unset !important; row-gap: unset !important; rx: unset !important; ry: unset !important; scale: unset !important; scroll-behavior: unset !important; scroll-snap-align: unset !important; scroll-snap-stop: unset !important; scroll-snap-type: unset !important; shape-image-threshold: unset !important; shape-margin: unset !important; shape-outside: unset !important; shape-rendering: unset !important; size: unset !important; speak-as: unset !important; stop-color: unset !important; stop-opacity: unset !important; stroke: unset !important; stroke-color: unset !important; stroke-dasharray: unset !important; stroke-dashoffset: unset !important; stroke-linecap: unset !important; stroke-linejoin: unset !important; stroke-miterlimit: unset !important; stroke-opacity: unset !important; stroke-width: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-align-last: unset !important; text-anchor: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-skip-ink: unset !important; text-decoration-style: unset !important; text-decoration-thickness: unset !important; text-emphasis-color: unset !important; text-emphasis-position: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-shadow: unset !important; text-transform: unset !important; text-underline-offset: unset !important; text-underline-position: unset !important; text-wrap: unset !important; touch-action: unset !important; transform: unset !important; transform-box: unset !important; transform-origin: unset !important; transform-style: unset !important; transition: unset !important; translate: unset !important; vector-effect: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; will-change: unset !important; word-break: unset !important; x: unset !important; y: unset !important; z-index: unset !important; -apple-color-filter: unset !important; -apple-pay-button-style: unset !important; -apple-pay-button-type: unset !important; border-spacing: unset !important; -webkit-box-align: unset !important; -webkit-box-decoration-break: unset !important; -webkit-box-direction: unset !important; -webkit-box-flex: unset !important; -webkit-box-flex-group: unset !important; -webkit-box-lines: unset !important; -webkit-box-ordinal-group: unset !important; -webkit-box-orient: unset !important; -webkit-box-pack: unset !important; -webkit-box-reflect: unset !important; -webkit-column-axis: unset !important; -webkit-column-progression: unset !important; -webkit-cursor-visibility: unset !important; -webkit-hyphenate-limit-after: unset !important; -webkit-hyphenate-limit-before: unset !important; -webkit-hyphenate-limit-lines: unset !important; -webkit-initial-letter: unset !important; -webkit-line-align: unset !important; -webkit-line-box-contain: unset !important; -webkit-line-clamp: unset !important; -webkit-line-grid: unset !important; -webkit-line-snap: unset !important; -webkit-nbsp-mode: unset !important; -webkit-rtl-ordering: unset !important; -webkit-text-decorations-in-effect: unset !important; -webkit-text-fill-color: unset !important; -webkit-text-security: unset !important; -webkit-text-stroke-color: unset !important; -webkit-text-stroke-width: unset !important; -webkit-user-drag: unset !important; -webkit-user-modify: unset !important; -webkit-user-select: unset !important; word-spacing: unset !important; background-clip: unset !important; background-origin: unset !important; box-shadow: unset !important; mask-composite: unset !important; mask-mode: unset !important; text-combine-upright: unset !important; -webkit-backdrop-filter: unset !important; -webkit-background-clip: unset !important; -webkit-background-origin: unset !important; -webkit-box-shadow: unset !important; mask-size: unset !important; -webkit-mask: unset !important; -webkit-mask-composite: unset !important; -webkit-mask-source-type: unset !important; -webkit-text-combine: unset !important; display: block !important; visibility: visible !important; opacity: unset !important; background-color: rgb(208, 216, 220) !important; block-size: unset !important; border-block: unset !important; border-radius: unset !important; border-end-end-radius: unset !important; border-end-start-radius: unset !important; border-inline: unset !important; border-start-end-radius: unset !important; border-start-start-radius: unset !important; inset: unset !important; contain-intrinsic-block-size: unset !important; contain-intrinsic-size: unset !important; contain-intrinsic-inline-size: unset !important; height: unset !important; inline-size: unset !important; inset-block: unset !important; inset-inline: unset !important; margin-block: unset !important; margin-bottom: unset !important; margin-inline: unset !important; margin-left: unset !important; margin-right: unset !important; max-block-size: unset !important; max-height: unset !important; max-inline-size: unset !important; max-width: unset !important; min-block-size: unset !important; min-height: unset !important; min-inline-size: unset !important; min-width: unset !important; overscroll-behavior-block: unset !important; overscroll-behavior-inline: unset !important; overscroll-behavior: unset !important; padding-block: unset !important; padding: unset !important; padding-inline: unset !important; scroll-margin-block: unset !important; scroll-margin: unset !important; scroll-margin-inline: unset !important; scroll-padding-block: unset !important; scroll-padding: unset !important; scroll-padding-inline: unset !important; width: unset !important; margin-top: 2px !important;">This message came from outside your organization.</div></div><div style="height: 0px; clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important;"></div></div><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Hello all,<br><br>I set MKL_VERBOSE = 1, but observed no print output specific to the use of MKL. Does PETSc enable this verbose output?<br><br>Best,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Yongzhong<br><br><br><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div id="mail-editor-reference-message-container"><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; border-image: none; padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="">From:<span class="Apple-converted-space"> </span></span></b><span style="">Pierre Jolivet <<a href="mailto:pierre@joliv.et" style="color: blue; text-decoration: underline;">pierre@joliv.et</a>><br><b>Date:<span class="Apple-converted-space"> </span></b>Friday, June 21, 2024 at 1:36</span><span style="font-family: Arial, sans-serif;"> </span><span style="">AM<br><b>To:<span class="Apple-converted-space"> </span></b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;">junchao.zhang@gmail.com</a>><br><b>Cc:<span class="Apple-converted-space"> </span></b>Yongzhong Li <<a href="mailto:yongzhong.li@mail.utoronto.ca" style="color: blue; text-decoration: underline;">yongzhong.li@mail.utoronto.ca</a>>,<span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<o:p></o:p></span></p></div><table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width: 861px; display: table; float: none;"><tbody><tr><td style="background: rgb(166, 166, 166); padding: 5.25pt 1.5pt;"></td><td width="100%" style="width: 827px; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt 5.25pt 11.25pt;"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">你通常不会收到来自</span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"><span class="Apple-converted-space"> </span><a href="mailto:pierre@joliv.et" style="color: blue; text-decoration: underline;">pierre@joliv.et</a><span class="Apple-converted-space"> </span></span><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">的电子邮件。</span><span style=""><a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!eXBeeIXo9Yqgp2nypqwKYimLnGBZXnF4dXxgLM1UoOIO6n8nt3XlfgjVWLPWJh4UOa5NNpx-nrJb_H828XRQKUREfR2m69oCbxI$" style="color: blue; text-decoration: underline;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian;">了解这一点为什么很重要</span></a></span><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"><o:p></o:p></span></div></div></td><td width="75" style="width: 56.25pt; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt;"></td></tr></tbody></table><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br><br></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">On 21 Jun 2024, at 6:42<span style="font-family: Arial, sans-serif;"> </span>AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;">junchao.zhang@gmail.com</a>> wrote:</div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div><div><div id="pfptBanners71iuax"><div id="pfptBanners71iuax"><div id="pfptBanners71iuax"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This Message Is From an External Sender<o:p></o:p></span></div></div><div id="pfptBanners71iuax"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-family: Arial, sans-serif;">This message came from outside your organization.<o:p></o:p></span></div></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">I remember there are some MKL env vars to print MKL routines called. <o:p></o:p></span></div></div></div></blockquote><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">The environment variable is MKL_VERBOSE</div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Thanks,</div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;">Pierre</div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><br><br></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">Maybe we can try it to see what MKL routines are really used and then we can understand why some petsc functions did not speed up <o:p></o:p></span></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"><br clear="all"><o:p></o:p></span></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">--Junchao Zhang<o:p></o:p></span></div></div></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"><o:p> </o:p></span></div></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">On Thu, Jun 20, 2024 at 10:39 PM Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">> wrote:<o:p></o:p></span></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); border-image: none; padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div id="m_-4861803116733450987pfptBannerqdehbik" style="border-width: 3pt medium medium; border-style: solid none none; border-color: rgb(144, 164, 174) currentcolor currentcolor; border-image: none; padding: 6pt 0cm 0cm; margin-top: 12pt; margin-bottom: 12pt; border-radius: 4px; min-width: 200px;"><div id="m_-4861803116733450987pfptBannerqdehbik" style="margin-bottom: 0.75pt; float: left;"><div id="m_-4861803116733450987pfptBannerqdehbik"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;"><o:p></o:p></span></b></div></div><div id="m_-4861803116733450987pfptBannerqdehbik" style="margin-top: 1.5pt;"><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"><o:p></o:p></span></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 0px; background: rgb(208, 216, 220);"><span style="font-size: 1pt; font-family: Helvetica;"> </span><span style="font-size: 1pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;">Hi Barry, sorry for my last results. I didn’t fully understand the stage profiling and logging in PETSc, now I only record KSPSolve() stage of my program. Some sample codes are as follow,<br><br> <span class="apple-converted-space"> </span>// Static variable to keep track of the stage counter</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>static int stageCounter = 1;</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>// Generate a unique stage name</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>std::ostringstream oss;</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>oss << "Stage " << stageCounter << " of Code";</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>std::string stageName = oss.str();</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>// Register the stage</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>PetscLogStage stagenum;</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>PetscLogStageRegister(stageName.c_str(), &stagenum);</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>PetscLogStagePush(stagenum);</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span><b>KSPSolve(*ksp_ptr, b, x);</b></span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>PetscLogStagePop();</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>stageCounter++;<br><br>I have attached my new logging results, there are 1 main stage and 4 other stages where each one is KSPSolve() call.<br><br>To provide some additional backgrounds, if you recall, I have been trying to get efficient iterative solution using multithreading. I found out by compiling PETSc with Intel MKL library instead of OpenBLAS, I am able to perform sparse matrix-vector multiplication faster, I am using MATSEQAIJMKL. This makes the shell matrix vector product in each iteration scale well with the #of threads. However, I found out the total GMERS solve time (~KSPSolve() time) is not scaling well the #of threads.<br><br>From the logging results I learned that when performing KSPSolve(), there are some CPU overheads in PCApply() and KSPGMERSOrthog(). I ran my programs using different number of threads and plotted the time consumption for PCApply() and KSPGMERSOrthog() against #of thread. I found out these two operations are not scaling with the threads at all! My results are attached as the pdf to give you a clear view.<br><br>My questions is,<br><br>From my understanding, in PCApply, MatSolve() is involved, KSPGMERSOrthog() will have many vector operations, so why these two parts can’t scale well with the # of threads when the intel MKL library is linked?<br><br>Thank you,<br>Yongzhong</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div><div id="m_-4861803116733450987mail-editor-reference-message-container"><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 9pt; font-family: Helvetica;">Barry Smith <</span><a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">bsmith@petsc.dev</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Date:<span class="apple-converted-space"> </span></b>Friday, June 14, 2024 at 11:36</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica;">AM<br><b>To:<span class="apple-converted-space"> </span></b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Cc:<span class="apple-converted-space"> </span></b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-users@mcs.anl.gov</span></a><span class="apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica;">>,<span class="apple-converted-space"> </span></span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-maint@mcs.anl.gov</span></a><span class="apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><</span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica;">>, Piero Triverio <</span><a href="mailto:piero.triverio@utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">piero.triverio@utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<o:p></o:p></span></p></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> I am a bit confused. Without the initial guess computation, there are still a bunch of events I don't understand <o:p></o:p></span></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatTranspose 79 1.0 4.0598e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatMatMultSym 110 1.0 1.7419e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatMatMultNum 90 1.0 1.2640e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatMatMatMultSym 20 1.0 1.3049e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatRARtSym 25 1.0 1.2492e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatMatTrnMultSym 25 1.0 8.8265e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatMatTrnMultNum 25 1.0 2.4820e+02 1.0 6.83e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 275<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatTrnMatMultSym 10 1.0 7.2984e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatTrnMatMultNum 10 1.0 9.3128e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">in addition there are many more VecMAXPY then VecMDot (in GMRES they are each done the same number of times)<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">VecMDot 5588 1.0 1.7183e+03 1.0 2.06e+13 1.0 0.0e+00 0.0e+00 0.0e+00 8 10 0 0 0 8 10 0 0 0 12016<o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">VecMAXPY 22412 1.0 8.4898e+03 1.0 4.17e+13 1.0 0.0e+00 0.0e+00 0.0e+00 39 20 0 0 0 39 20 0 0 0 4913<o:p></o:p></span></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">Finally there are a huge number of <o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">MatMultAdd 258048 1.0 1.4178e+03 1.0 6.10e+13 1.0 0.0e+00 0.0e+00 0.0e+00 7 29 0 0 0 7 29 0 0 0 43025<o:p></o:p></span></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">Are you making calls to all these routines? Are you doing this inside your MatMult() or before you call KSPSolve?<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">The reason I wanted you to make a simpler run without the initial guess code is that your events are far more complicated than would be produced by GMRES alone so it is not possible to understand the behavior you are seeing without fully understanding all the events happening in the code.<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>Barry<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"><o:p> </o:p></span></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">On Jun 14, 2024, at 1:19</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica;">AM, Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">> wrote:<o:p></o:p></span></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;">Thanks, I have attached the results without using any KSPGuess. At low frequency, the iteration steps are quite close to the one with KSPGuess, specifically<span class="m-4861803116733450987apple-converted-space"> </span><br><br> KSPGuess Object: 1 MPI process</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>type: fischer</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>Model 1, size 200<br><br>However, I found at higher frequency, the # of iteration steps are significant higher than the one with KSPGuess, I have attahced both of the results for your reference.<br><br>Moreover, could I ask why the one without the KSPGuess options can be used for a baseline comparsion? What are we comparing here? How does it relate to the performance issue/bottleneck I found? “</span><b><span style="font-size: 9pt; font-family: Helvetica;">I have noticed that the time taken by </span></b><b><span style="font-size: 9pt; font-family: "Courier New";">KSPSolve</span></b><b><span style="font-size: 9pt; font-family: Helvetica;"> is<span class="m-4861803116733450987apple-converted-space"> </span></span></b><b><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">almost two times<span class="m-4861803116733450987apple-converted-space"> </span></span></b><b><span style="font-size: 9pt; font-family: Helvetica;">greater than the CPU time for matrix-vector product multiplied by the number of iteration</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;">”<span class="m-4861803116733450987apple-converted-space"> </span><br><br>Thank you!<br>Yongzhong</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div id="m_-4861803116733450987mail-editor-reference-message-container"><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica;">From:<span class="m-4861803116733450987apple-converted-space"> </span></span></b><span style="font-size: 9pt; font-family: Helvetica;">Barry Smith <</span><a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">bsmith@petsc.dev</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Date:<span class="m-4861803116733450987apple-converted-space"> </span></b>Thursday, June 13, 2024 at 2:14</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica;">PM<br><b>To:<span class="m-4861803116733450987apple-converted-space"> </span></b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Cc:<span class="m-4861803116733450987apple-converted-space"> </span></b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-users@mcs.anl.gov</span></a><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica;">>,<span class="m-4861803116733450987apple-converted-space"> </span></span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-maint@mcs.anl.gov</span></a><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><</span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica;">>, Piero Triverio <</span><a href="mailto:piero.triverio@utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">piero.triverio@utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Subject:<span class="m-4861803116733450987apple-converted-space"> </span></b>Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<o:p></o:p></span></p></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>Can you please run the same thing without the KSPGuess option(s) for a baseline comparison?<o:p></o:p></span></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> Thanks<o:p></o:p></span></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> Barry<o:p></o:p></span></div></div></div><div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">On Jun 13, 2024, at 1:27</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica;">PM, Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">> wrote:<o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div id="m_-4861803116733450987pfptBannerxs2204y"><div id="m_-4861803116733450987pfptBannerxs2204y"><div id="m_-4861803116733450987pfptBannerxs2204y"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div id="m_-4861803116733450987pfptBannerxs2204y"><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Helvetica;">Hi Matt,<br><br>I have rerun the program with the keys you provided. The system output when performing ksp solve and the final petsc log output were stored in a .txt file attached for your reference.<br><br>Thanks!<br>Yongzhong</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 11pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div id="m_-4861803116733450987mail-editor-reference-message-container"><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica;">From:<span class="m-4861803116733450987apple-converted-space"> </span></span></b><span style="font-size: 9pt; font-family: Helvetica;">Matthew Knepley <</span><a href="mailto:knepley@gmail.com" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">knepley@gmail.com</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Date:<span class="m-4861803116733450987apple-converted-space"> </span></b>Wednesday, June 12, 2024 at 6:46</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica;">PM<br><b>To:<span class="m-4861803116733450987apple-converted-space"> </span></b>Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Cc:<span class="m-4861803116733450987apple-converted-space"> </span></b></span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-users@mcs.anl.gov</span></a><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica;">>,<span class="m-4861803116733450987apple-converted-space"> </span></span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-maint@mcs.anl.gov</span></a><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><</span><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">petsc-maint@mcs.anl.gov</span></a><span style="font-size: 9pt; font-family: Helvetica;">>, Piero Triverio <</span><a href="mailto:piero.triverio@utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">piero.triverio@utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">><br><b>Subject:<span class="m-4861803116733450987apple-converted-space"> </span></b>Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance Issue<o:p></o:p></span></p></div><table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width: 605.609375px; display: table; float: none;"><tbody><tr><td style="background: rgb(166, 166, 166); padding: 5.25pt 1.5pt;"></td><td width="100%" style="width: 571.609375px; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt 5.25pt 11.25pt;"><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">你通常不会收到来自</span><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span></span><span style=""><a href="mailto:knepley@gmail.com" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif;">knepley@gmail.com</span></a></span><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);"> </span></span><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian; color: rgb(33, 33, 33);">的电子邮件。</span><span style=""><a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!djGfJnEhNJROfsMsBJy5u_KoRKbug55xZ64oHKUFnH2cWku_Th1hwt4TDdoMd8pWYVDzJeqJslMNZwpO3y0Et94d31qk-oCEwo4$" target="_blank" style="color: blue; text-decoration: underline;"><span lang="ZH-CN" style="font-size: 9pt; font-family: DengXian;">了解这一点为什么很重要</span></a></span></div></div></div></div></td><td width="75" style="width: 56.25pt; background: rgb(234, 234, 234); padding: 5.25pt 3.75pt;"></td></tr></tbody></table><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">On Wed, Jun 12, 2024 at 6:36</span><span style="font-size: 9pt; font-family: Arial, sans-serif;"> </span><span style="font-size: 9pt; font-family: Helvetica;">PM Yongzhong Li <</span><a href="mailto:yongzhong.li@mail.utoronto.ca" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">yongzhong.li@mail.utoronto.ca</span></a><span style="font-size: 9pt; font-family: Helvetica;">> wrote:<o:p></o:p></span></div></div></div></div><div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-image: none; padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204);"><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; font-family: Helvetica; color: white;">Dear PETSc’s developers, I hope this email finds you well. I am currently working on a project using PETSc and have encountered a performance issue with the KSPSolve function. Specifically, I have noticed that the time taken by KSPSolve is<span class="m-4861803116733450987apple-converted-space"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; font-family: Helvetica; color: white;">ZjQcmQRYFpfptBannerStart</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div id="m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww" style="border-width: 3pt medium medium; border-style: solid none none; border-image: none; padding: 6pt 0cm 0cm; margin-top: 12pt; margin-bottom: 12pt; border-color: currentcolor; border-radius: 4px; min-width: 200px;"><div id="m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww" style="margin-bottom: 0.75pt; float: left;"><div id="m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div id="m_-4861803116733450987m_5223539340070699115pfptBannerpeukvww" style="margin-top: 1.5pt;"><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; line-height: 13.5pt; background: rgb(208, 216, 220);"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif; background: rgb(208, 216, 220);"><span style="font-size: 1pt; font-family: Helvetica;"> </span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 1pt; font-family: Helvetica; color: white;">ZjQcmQRYFpfptBannerEnd</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">Dear<span class="m-4861803116733450987apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">PETSc’s developers</span><span style="font-size: 9pt; font-family: Helvetica;">,<o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">I hope this email finds you well.<o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">I am currently working on a project using PETSc and have encountered a performance issue with the </span><span style="font-size: 9pt; font-family: "Courier New";">KSPSolve</span><span style="font-size: 9pt; font-family: Helvetica;"> function. Specifically,<span class="m-4861803116733450987apple-converted-space"> </span><b>I have noticed that the time taken by </b></span><b><span style="font-size: 9pt; font-family: "Courier New";">KSPSolve</span></b><b><span style="font-size: 9pt; font-family: Helvetica;"> is<span class="m-4861803116733450987apple-converted-space"> </span></span></b><b><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">almost two times<span class="m-4861803116733450987apple-converted-space"> </span></span></b><b><span style="font-size: 9pt; font-family: Helvetica;">greater than the CPU time for matrix-vector product multiplied by the number of iteration steps</span></b><span style="font-size: 9pt; font-family: Helvetica;">.<span class="m-4861803116733450987apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">I use C++ chrono to record CPU time.</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">For context, I am using a shell system matrix</span><span class="m-4861803116733450987apple-converted-space"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">A</span><span style="font-size: 9pt; font-family: Helvetica;">. Despite my efforts to<span class="m-4861803116733450987apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">parallelize</span><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;">the matrix-vector product</span><span class="m-4861803116733450987apple-converted-space"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">(Ax)</span><span style="font-size: 9pt; font-family: Helvetica;">, the overall solve time remains higher</span><span class="m-4861803116733450987apple-converted-space"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">than the matrix vector product per iteration indicates when multiple threads were used</span><span style="font-size: 9pt; font-family: Helvetica;">. Here are a few details of my setup:<o:p></o:p></span></div></div></div><ul type="disc" style="margin-bottom: 0cm;"><li class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica;">Matrix Type</span></b><span style="font-size: 9pt; font-family: Helvetica;">: Shell system matrix<o:p></o:p></span></li><li class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica;">Preconditioner</span></b><span style="font-size: 9pt; font-family: Helvetica;">:<span class="m-4861803116733450987apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">Shell PC</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></li><li class="MsoNormal" style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span style="font-size: 9pt; font-family: Helvetica;">Parallel Environment</span></b><span style="font-size: 9pt; font-family: Helvetica;">:<span class="m-4861803116733450987apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">Using Intel MKL as PETSc’s BLAS/LAPACK library, multithreading is enabled</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></li></ul><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">I have considered several potential reasons, such as preconditioner setup, additional solver operations, and the inherent overhead of using a shell system matrix.<span class="m-4861803116733450987apple-converted-space"> </span><b>However, since KSPSolve is a high-level API, I have been unable to pinpoint the exact cause of the increased solve time.</b></span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">Have you observed the same issue?<span class="m-4861803116733450987apple-converted-space"> </span></span><span style="font-size: 9pt; font-family: Helvetica;">Could you</span><span class="m-4861803116733450987apple-converted-space"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">please<span class="m-4861803116733450987apple-converted-space"> </span></span><span style="font-size: 9pt; font-family: Helvetica;">provide some<span class="m-4861803116733450987apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">experience<span class="m-4861803116733450987apple-converted-space"> </span></span><span style="font-size: 9pt; font-family: Helvetica;">on how to diagnose and address this performance discrepancy? Any insights or recommendations you could offer would be greatly appreciated.<o:p></o:p></span></div></div></div></div></div></div></blockquote><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">For any performance question like this, we need to see the output of your code run with<o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>-ksp_view -ksp_monitor_true_residual -ksp_converged_reason -log_view<o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <span class="apple-converted-space"> </span>Thanks,<o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> Matt<o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-image: none; padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204);"><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">Thank you for your time and assistance.<o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">Best</span><span class="m-4861803116733450987apple-converted-space"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">regards,</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 9pt; font-family: Helvetica;">Yongzhong</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">-----------------------------------------------------------</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Yongzhong Li</span></b><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">PhD student | Electromagnetics Group</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Department of Electrical & Computer Engineering</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">University of Toronto</span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.us/v3/__http://www.modelics.org__;!!G_uCfscf7eWS!cuLttMJEcegaqu461Bt4QLsO4fASfLM5vjRbtyNhWJQiInbjgNwkGNdkFE1ebSbFjOUatYB0-jd2yQWMWzqkDFFjwMvNl3ZKAr8$" target="_blank" style="color: blue; text-decoration: underline;"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(5, 99, 193);">http://www.modelics.org</span></a><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div></div></div></blockquote></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"><br clear="all"><o:p></o:p></span></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span class="m-4861803116733450987gmailsignatureprefix"><span style="font-size: 9pt; font-family: Helvetica;">--</span></span><span class="m-4861803116733450987apple-converted-space"><span style="font-size: 9pt; font-family: Helvetica;"> </span></span><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div><div><div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!djGfJnEhNJROfsMsBJy5u_KoRKbug55xZ64oHKUFnH2cWku_Th1hwt4TDdoMd8pWYVDzJeqJslMNZwpO3y0Et94d31qkNOuenGA$" target="_blank" style="color: blue; text-decoration: underline;"><span style="font-size: 9pt; font-family: Helvetica;">https://www.cse.buffalo.edu/~knepley/</span></a><span style="font-size: 9pt; font-family: Helvetica;"><o:p></o:p></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"><ksp_petsc_log.txt><o:p></o:p></span></div></div></div></blockquote></div><div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"> <o:p></o:p></span></div></div></div></div></div></div><div style="margin: 0cm; font-size: 12pt; font-family: Aptos, sans-serif;"><span style="font-size: 9pt; font-family: Helvetica;"><ksp_petsc_log.txt><ksp_petsc_log_noguess.txt></span></div></div></blockquote></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></div></blockquote></div><br></div></body></html>