<div dir="ltr">I've managed to read through your MR posts today.<div><br></div><div>As I said there getting parallel fieldsplit/additive solves is what I'm after. Maybe start with Jacobi. And I am MPI serial so my scatter gathers are local in PCApply_FieldSplit. </div><div><br></div><div>FYI, Christan Trott responded with this wrt the future Kokkos interface for streams.</div><div><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383431.196100" style="box-sizing:inherit;width:999px;outline-width:0px;outline-style:none;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><div class="gmail-c-message_kit__background gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit"><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383431.196100" style="box-sizing:inherit;width:999px"><div class="gmail-c-message_kit__background gmail-c-message_kit__background--hovered gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover gmail-c-message_kit__hover--hovered" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__labels__offscreen_message" style="box-sizing:inherit;overflow:hidden;height:1px;width:1px;padding:0px;border:0px"></div><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__left" style="box-sizing:inherit;margin-right:8px;display:flex"><button class="gmail-c-button-unstyled gmail-c-message_kit__avatar gmail-c-avatar gmail-c-avatar--interactive" tabindex="-1" type="button" style="box-sizing:inherit;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border-width:0px;border-style:initial;border-color:initial;font:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;outline:none;height:36px;width:36px"><img src="https://ca.slack-edge.com/T5ASL2F1N-U5C9R5T53-8b962016bb0f-48" class="gmail-c-base_icon gmail-c-base_icon--image" alt="" style="box-sizing: inherit; border-radius: 4px; flex-shrink: 0; background-size: 100%; background-repeat: no-repeat; display: inline-block; height: 36px; width: 36px;"></button></div><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><span class="gmail-c-message__sender gmail-c-message_kit__sender" style="box-sizing:inherit;font-weight:900;word-break:break-word"><a target="_blank" rel="noopener noreferrer" class="gmail-c-link gmail-c-message__sender_link" href="https://app.slack.com/team/U5C9R5T53" style="box-sizing:inherit;text-decoration-line:none;display:inline-block">crtrott<span class="gmail-c-opens_in_new_tab" style="box-sizing:inherit"></span></a><span class="gmail-c-custom_status" style="box-sizing:inherit;margin-left:4px"><span class="gmail-c-emoji gmail-c-emoji__small gmail-c-emoji--inline" style="box-sizing:inherit;overflow:visible;display:inline-flex;vertical-align:top;object-fit:contain;width:16px;height:auto"><img src="https://slack-imgs.com/?c=1&o1=gu&url=https%3A%2F%2Femoji.slack-edge.com%2FT5ASL2F1N%2Fslack_call%2Fb81fffd6dd.png" alt=":slack_call:" style="box-sizing: inherit; overflow: hidden; top: 11px; object-fit: contain; height: 16px; width: 16px;"></span></span></span> <a class="gmail-c-link gmail-c-timestamp" href="https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383431196100?thread_ts=1610309148.167100&cid=C5BGU5NDQ" style="box-sizing:inherit;text-decoration-line:none;font-size:12px"><span class="gmail-c-timestamp__label" style="box-sizing:inherit">3 hours ago</span></a><br style="box-sizing:inherit"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit">So basicall</div></div></div></div></div></div></div></div></div></div></div></div>So basically we are still fixing some kinks in the whole thing (you can imagine there are a million funky corner cases if you suddenly start throwing around even more asynchronous stuff, in particular if you potentially do that from multiple host threads, and if its not even clear if a stream object is permanently associated with one thread or passed back and forth)</div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383488.196300" style="box-sizing:inherit;width:999px;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><div class="gmail-c-message_kit__background gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__labels__offscreen_message" style="box-sizing:inherit;overflow:hidden;height:1px;width:1px;padding:0px;border:0px"></div><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__left" style="box-sizing:inherit;margin-right:8px;display:flex"><button class="gmail-c-button-unstyled gmail-c-message_kit__avatar gmail-c-avatar gmail-c-avatar--interactive" tabindex="-1" type="button" style="box-sizing:inherit;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border-width:0px;border-style:initial;border-color:initial;font:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;outline:none;height:36px;width:36px"><img src="https://ca.slack-edge.com/T5ASL2F1N-U5C9R5T53-8b962016bb0f-48" class="gmail-c-base_icon gmail-c-base_icon--image" alt="" style="box-sizing: inherit; border-radius: 4px; flex-shrink: 0; background-size: 100%; background-repeat: no-repeat; display: inline-block; height: 36px; width: 36px;"></button></div><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><span class="gmail-c-message__sender gmail-c-message_kit__sender" style="box-sizing:inherit;font-weight:900;word-break:break-word"><a target="_blank" rel="noopener noreferrer" class="gmail-c-link gmail-c-message__sender_link" href="https://app.slack.com/team/U5C9R5T53" style="box-sizing:inherit;text-decoration-line:none;display:inline-block">crtrott<span class="gmail-c-opens_in_new_tab" style="box-sizing:inherit"></span></a><span class="gmail-c-custom_status" style="box-sizing:inherit;margin-left:4px"><span class="gmail-c-emoji gmail-c-emoji__small gmail-c-emoji--inline" style="box-sizing:inherit;overflow:visible;display:inline-flex;vertical-align:top;object-fit:contain;width:16px;height:auto"><img src="https://slack-imgs.com/?c=1&o1=gu&url=https%3A%2F%2Femoji.slack-edge.com%2FT5ASL2F1N%2Fslack_call%2Fb81fffd6dd.png" alt=":slack_call:" style="box-sizing: inherit; overflow: hidden; top: 11px; object-fit: contain; height: 16px; width: 16px;"></span></span></span> <a class="gmail-c-link gmail-c-timestamp" href="https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383488196300?thread_ts=1610309148.167100&cid=C5BGU5NDQ" style="box-sizing:inherit;text-decoration-line:none;font-size:12px"><span class="gmail-c-timestamp__label" style="box-sizing:inherit">3 hours ago</span></a><br style="box-sizing:inherit"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit">but in the future you will be able to do something like: auto instances = Kokkos::partition_exec_space(Kokkos::DefaultExecutionSpace(),4);</div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383504.196500" style="box-sizing:inherit;width:999px;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><div class="gmail-c-message_kit__background gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__labels__offscreen_message" style="box-sizing:inherit;overflow:hidden;height:1px;width:1px;padding:0px;border:0px"></div><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__left" style="box-sizing:inherit;margin-right:8px;display:flex"><button class="gmail-c-button-unstyled gmail-c-message_kit__avatar gmail-c-avatar gmail-c-avatar--interactive" tabindex="-1" type="button" style="box-sizing:inherit;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border-width:0px;border-style:initial;border-color:initial;font:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;outline:none;height:36px;width:36px"><img src="https://ca.slack-edge.com/T5ASL2F1N-U5C9R5T53-8b962016bb0f-48" class="gmail-c-base_icon gmail-c-base_icon--image" alt="" style="box-sizing: inherit; border-radius: 4px; flex-shrink: 0; background-size: 100%; background-repeat: no-repeat; display: inline-block; height: 36px; width: 36px;"></button></div><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><span class="gmail-c-message__sender gmail-c-message_kit__sender" style="box-sizing:inherit;font-weight:900;word-break:break-word"><a target="_blank" rel="noopener noreferrer" class="gmail-c-link gmail-c-message__sender_link" href="https://app.slack.com/team/U5C9R5T53" style="box-sizing:inherit;text-decoration-line:none;display:inline-block">crtrott<span class="gmail-c-opens_in_new_tab" style="box-sizing:inherit"></span></a><span class="gmail-c-custom_status" style="box-sizing:inherit;margin-left:4px"><span class="gmail-c-emoji gmail-c-emoji__small gmail-c-emoji--inline" style="box-sizing:inherit;overflow:visible;display:inline-flex;vertical-align:top;object-fit:contain;width:16px;height:auto"><img src="https://slack-imgs.com/?c=1&o1=gu&url=https%3A%2F%2Femoji.slack-edge.com%2FT5ASL2F1N%2Fslack_call%2Fb81fffd6dd.png" alt=":slack_call:" style="box-sizing: inherit; overflow: hidden; top: 11px; object-fit: contain; height: 16px; width: 16px;"></span></span></span> <a class="gmail-c-link gmail-c-timestamp" href="https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383504196500?thread_ts=1610309148.167100&cid=C5BGU5NDQ" style="box-sizing:inherit;text-decoration-line:none;font-size:12px"><span class="gmail-c-timestamp__label" style="box-sizing:inherit">3 hours ago</span></a><br style="box-sizing:inherit"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit">and get four semantically independent execution spaces back</div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383541.196700" style="box-sizing:inherit;width:999px;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><div class="gmail-c-message_kit__background gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__labels__offscreen_message" style="box-sizing:inherit;overflow:hidden;height:1px;width:1px;padding:0px;border:0px"></div><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__left" style="box-sizing:inherit;margin-right:8px;display:flex"><button class="gmail-c-button-unstyled gmail-c-message_kit__avatar gmail-c-avatar gmail-c-avatar--interactive" tabindex="-1" type="button" style="box-sizing:inherit;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border-width:0px;border-style:initial;border-color:initial;font:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;outline:none;height:36px;width:36px"><img src="https://ca.slack-edge.com/T5ASL2F1N-U5C9R5T53-8b962016bb0f-48" class="gmail-c-base_icon gmail-c-base_icon--image" alt="" style="box-sizing: inherit; border-radius: 4px; flex-shrink: 0; background-size: 100%; background-repeat: no-repeat; display: inline-block; height: 36px; width: 36px;"></button></div><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><span class="gmail-c-message__sender gmail-c-message_kit__sender" style="box-sizing:inherit;font-weight:900;word-break:break-word"><a target="_blank" rel="noopener noreferrer" class="gmail-c-link gmail-c-message__sender_link" href="https://app.slack.com/team/U5C9R5T53" style="box-sizing:inherit;text-decoration-line:none;display:inline-block">crtrott<span class="gmail-c-opens_in_new_tab" style="box-sizing:inherit"></span></a><span class="gmail-c-custom_status" style="box-sizing:inherit;margin-left:4px"><span class="gmail-c-emoji gmail-c-emoji__small gmail-c-emoji--inline" style="box-sizing:inherit;overflow:visible;display:inline-flex;vertical-align:top;object-fit:contain;width:16px;height:auto"><img src="https://slack-imgs.com/?c=1&o1=gu&url=https%3A%2F%2Femoji.slack-edge.com%2FT5ASL2F1N%2Fslack_call%2Fb81fffd6dd.png" alt=":slack_call:" style="box-sizing: inherit; overflow: hidden; top: 11px; object-fit: contain; height: 16px; width: 16px;"></span></span></span> <a class="gmail-c-link gmail-c-timestamp" href="https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383541196700?thread_ts=1610309148.167100&cid=C5BGU5NDQ" style="box-sizing:inherit;text-decoration-line:none;font-size:12px"><span class="gmail-c-timestamp__label" style="box-sizing:inherit">3 hours ago</span></a><br style="box-sizing:inherit"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit">with some backends they will still serialize (like CPU OpenMP) since you just get back the default instance 4 times</div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383549.196900" style="box-sizing:inherit;width:999px;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><div class="gmail-c-message_kit__background gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__labels__offscreen_message" style="box-sizing:inherit;overflow:hidden;height:1px;width:1px;padding:0px;border:0px"></div><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__left" style="box-sizing:inherit;margin-right:8px;display:flex"><button class="gmail-c-button-unstyled gmail-c-message_kit__avatar gmail-c-avatar gmail-c-avatar--interactive" tabindex="-1" type="button" style="box-sizing:inherit;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border-width:0px;border-style:initial;border-color:initial;font:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;outline:none;height:36px;width:36px"><img src="https://ca.slack-edge.com/T5ASL2F1N-U5C9R5T53-8b962016bb0f-48" class="gmail-c-base_icon gmail-c-base_icon--image" alt="" style="box-sizing: inherit; border-radius: 4px; flex-shrink: 0; background-size: 100%; background-repeat: no-repeat; display: inline-block; height: 36px; width: 36px;"></button></div><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><span class="gmail-c-message__sender gmail-c-message_kit__sender" style="box-sizing:inherit;font-weight:900;word-break:break-word"><a target="_blank" rel="noopener noreferrer" class="gmail-c-link gmail-c-message__sender_link" href="https://app.slack.com/team/U5C9R5T53" style="box-sizing:inherit;text-decoration-line:none;display:inline-block">crtrott<span class="gmail-c-opens_in_new_tab" style="box-sizing:inherit"></span></a><span class="gmail-c-custom_status" style="box-sizing:inherit;margin-left:4px"><span class="gmail-c-emoji gmail-c-emoji__small gmail-c-emoji--inline" style="box-sizing:inherit;overflow:visible;display:inline-flex;vertical-align:top;object-fit:contain;width:16px;height:auto"><img src="https://slack-imgs.com/?c=1&o1=gu&url=https%3A%2F%2Femoji.slack-edge.com%2FT5ASL2F1N%2Fslack_call%2Fb81fffd6dd.png" alt=":slack_call:" style="box-sizing: inherit; overflow: hidden; top: 11px; object-fit: contain; height: 16px; width: 16px;"></span></span></span> <a class="gmail-c-link gmail-c-timestamp" href="https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383549196900?thread_ts=1610309148.167100&cid=C5BGU5NDQ" style="box-sizing:inherit;text-decoration-line:none;font-size:12px"><span class="gmail-c-timestamp__label" style="box-sizing:inherit">3 hours ago</span></a><br style="box-sizing:inherit"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit">on stuff like CUDA you get independent streams</div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail-c-virtual_list__item" tabindex="-1" id="gmail-C5BGU5NDQ-1610309148.167100-thread-list_1610383571.197100" style="box-sizing:inherit;width:999px;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><div class="gmail-c-message_kit__background gmail-c-message_kit__message gmail-c-message_kit__thread_message" style="box-sizing:inherit;line-height:1.46668"><div class="gmail-c-message_kit__hover" style="box-sizing:inherit"><div class="gmail-c-message_kit__actions gmail-c-message_kit__actions--default" style="box-sizing:inherit"><div class="gmail-c-message_kit__labels__offscreen_message" style="box-sizing:inherit;overflow:hidden;height:1px;width:1px;padding:0px;border:0px"></div><div class="gmail-c-message_kit__gutter" style="box-sizing:inherit;display:flex;padding:8px 24px 8px 16px"><div class="gmail-c-message_kit__gutter__left" style="box-sizing:inherit;margin-right:8px;display:flex"><button class="gmail-c-button-unstyled gmail-c-message_kit__avatar gmail-c-avatar gmail-c-avatar--interactive" tabindex="-1" type="button" style="box-sizing:inherit;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border-width:0px;border-style:initial;border-color:initial;font:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;outline:none;height:36px;width:36px"><img src="https://ca.slack-edge.com/T5ASL2F1N-U5C9R5T53-8b962016bb0f-48" class="gmail-c-base_icon gmail-c-base_icon--image" alt="" style="box-sizing: inherit; border-radius: 4px; flex-shrink: 0; background-size: 100%; background-repeat: no-repeat; display: inline-block; height: 36px; width: 36px;"></button></div><div class="gmail-c-message_kit__gutter__right" style="box-sizing:inherit;min-width:0px;padding:8px 0px 8px 16px"><span class="gmail-c-message__sender gmail-c-message_kit__sender" style="box-sizing:inherit;font-weight:900;word-break:break-word"><a target="_blank" rel="noopener noreferrer" class="gmail-c-link gmail-c-message__sender_link" href="https://app.slack.com/team/U5C9R5T53" style="box-sizing:inherit;text-decoration-line:none;display:inline-block">crtrott<span class="gmail-c-opens_in_new_tab" style="box-sizing:inherit"></span></a><span class="gmail-c-custom_status" style="box-sizing:inherit;margin-left:4px"><span class="gmail-c-emoji gmail-c-emoji__small gmail-c-emoji--inline" style="box-sizing:inherit;overflow:visible;display:inline-flex;vertical-align:top;object-fit:contain;width:16px;height:auto"><img src="https://slack-imgs.com/?c=1&o1=gu&url=https%3A%2F%2Femoji.slack-edge.com%2FT5ASL2F1N%2Fslack_call%2Fb81fffd6dd.png" alt=":slack_call:" style="box-sizing: inherit; overflow: hidden; top: 11px; object-fit: contain; height: 16px; width: 16px;"></span></span></span> <a class="gmail-c-link gmail-c-timestamp" href="https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383571197100?thread_ts=1610309148.167100&cid=C5BGU5NDQ" style="box-sizing:inherit;text-decoration-line:none;font-size:12px"><span class="gmail-c-timestamp__label" style="box-sizing:inherit">3 hours ago</span></a><br style="box-sizing:inherit"><div class="gmail-c-message_kit__blocks gmail-c-message_kit__blocks--rich_text" style="box-sizing:inherit;max-width:none;margin-bottom:4px"><div class="gmail-c-message__message_blocks gmail-c-message__message_blocks--rich_text" style="box-sizing:inherit;max-width:none"><div class="gmail-p-block_kit_renderer" style="box-sizing:inherit;width:923px"><div class="gmail-p-block_kit_renderer__block_wrapper gmail-p-block_kit_renderer__block_wrapper--first" style="box-sizing:inherit;display:flex"><div class="gmail-p-rich_text_block" dir="auto" style="box-sizing:inherit;width:923px;line-height:1.46668"><div class="gmail-p-rich_text_section" style="box-sizing:inherit">and on other thread based backends we may partition the threadpool for real</div></div></div></div></div></div></div></div></div></div></div></div></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 11, 2021 at 11:52 AM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Hmm I suppose this means Kokkos should accept a stream like we expect it to? According to this somewhat recent merged PR: <a href="https://github.com/kokkos/kokkos/pull/1919" target="_blank">https://github.com/kokkos/kokkos/pull/1919</a> you can now make a "Kokkos::Cuda” object, and pass it as arg1 to range policies as an execution space. Here’s what I found on it (the cuda specific one is useless):<div><br></div><div><a href="https://github.com/kokkos/kokkos/wiki/ExecutionSpaceConcept" target="_blank">https://github.com/kokkos/kokkos/wiki/ExecutionSpaceConcept</a></div><div><a href="https://github.com/kokkos/kokkos/wiki/Kokkos::ExecutionSpaceConcept" target="_blank">https://github.com/kokkos/kokkos/wiki/Kokkos%3A%3AExecutionSpaceConcept</a></div><div><a href="https://github.com/kokkos/kokkos/wiki/Kokkos::Cuda" target="_blank">https://github.com/kokkos/kokkos/wiki/Kokkos%3A%3ACuda</a> <—— cuda specific<br><div><br></div><div><div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br>Cell: (312) 694-3391</div></div>
</div>
<div><br><blockquote type="cite"><div>On Jan 11, 2021, at 10:35, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr"><div>Jacob, I'm not sure I understand this response. I could not find you on the Kokkos slack channel.</div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><br></span></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures">Me: And My colleague in PETSc, Jacob Faibussowitsch, has talked to you about Kokkos taking a Cuda, Hip, etc., stream. This is something that would make it easier to deal with asynchronous GPU solvers in PETSc. We just wanted to check on this.</span><br></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures"><br></span></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">Trott: Kokkos itself can do it for practically every operation</span><br></div><div><br></div><div>Maybe you want to talk with him at some point, but we can worry about getting Cuda to work for now.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 10, 2021 at 2:28 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>I would like as much as possible to pass the cuda and hip streams to Kokkos, since I can directly handle much of the annoyance with wrangling multiple streams and stream objects externally. Last I checked on this Kokkos was moving towards allowing association of streams to functions, but admittedly this was a while back.</div><div><br><div>
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br>Cell: (312) 694-3391</div></div>
</div>
<div><br><blockquote type="cite"><div>On Jan 10, 2021, at 13:10, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 9, 2021 at 7:37 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">It is a single object that holds a pointer to every stream implementation and toggleable type so it can be universally passed around. Currently has a cudaStream and a hipStream but this is easily extendable to any other stream implementation. <br></div></blockquote><div><br></div><div>Do you have any thoughts on how this would work with Kokkos?</div><div><br></div><div>Would you want to feed Kokkos your Cuda/Hip, etc, stream or add a Kokkos backend to your object? </div><div><br></div><div>Junchao might be the person to ask. I would guess Kokkos View (vector) objects carry a stream because they block on a "deep_copy", that moves data to/from the GPU, and it is blocking.</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><br><div dir="ltr"><span style="background-color:rgba(255,255,255,0)">Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br>Cell: +1 (312) 694-3391</span></div><div dir="ltr"><br><blockquote type="cite">On Jan 9, 2021, at 18:19, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr">Is this stream object going to have Cuda, Kokkos, etc., implementations?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 9, 2021 at 4:09 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I’m currently working on an implementation of a general PetscStream object. Currently it only supports Vector ops and has a proof of concept KSPCG, but should be extensible to other objects when finished. Junchao is also indirectly working on pipeline support in his NVSHMEM MR. Take a look at either MR, it would be very useful to get your input, as tailoring either of these approaches for pipelined algorithms is key.<div><br><div>
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br>Cell: (312) 694-3391</div></div>
</div>
<div><br><blockquote type="cite"><div>On Jan 9, 2021, at 15:01, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">I would like to put a non-overlapping ASM solve on the GPU. It's not clear that we have a model for this. <div><br></div><div>PCApply_ASM currently pipelines the scater with the subdomain solves. I think we would want to change this and do a 1) scatter begin loop, 2) scatter end and non-blocking solve loop, 3) solve-wait and scatter begging loop and 4) scatter end loop.<br></div><div><br></div><div>I'm not sure how to go about doing this.</div><div> * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining algorithm and rewrite PCApply_ASM?</div><div> * Add a solver-wait method to KSP?</div><div><br></div><div>Thoughts?</div><div><br></div><div>Mark</div></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div></blockquote></div></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</div></blockquote></div><br></div></div></div></blockquote></div>