<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>True that it's not a huge efficiency concern, as this only affects the setup stage. This is more wondering if there's a simpler way to do the setup (sounds like there isn't at the moment).</div><div><br class=""></div><div><blockquote type="cite" class=""><div class="">Am 19.02.2021 um 11:41 schrieb Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" class="">I dont understand the issue. I assume your vecscatter is not a throw away object but you will reuse it many times. Once the setup is done, the indices used internally by the implementation should be on gpu , or not?</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Ven 19 Feb 2021, 13:33 Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com" class="">patrick.sanan@gmail.com</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class="">Thanks! That helps a lot. <div class=""><br class=""></div><div class=""><div class="">I assume "no," but is ISCUDA simple to add?</div><div class=""><br class=""></div><div class="">More on what I'm trying to do, in case I'm missing an obvious approach:</div><div class=""><br class=""></div><div class="">I'm working on a demo code that uses an external library, based on Kokkos, as a solver - I create a Vec of type KOKKOS and populate it with the solution data from the library, by getting access to the raw Kokkos view with VecKokkosGetDeviceView() * .</div><div class=""><br class=""></div><div class="">I then want to reorder that solution data into PETSc-native ordering (for a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to do that.</div><div class=""><br class=""></div><div class="">The issue is that to create this scatter, I need to use information (essentially, an element-to-index map) from the external library's mesh-management object, which lives on the device. This doesn't work (when host != device), because of course the ISs live on the host and to create them I need to provide host arrays of indices.</div><div class=""><br class=""></div><div class="">Am I stuck, for now, with sending the index information information from the device to the host, using it to create the IS, and then having essentially the same information go back to the device when I use the scatter?</div><div class=""><br class=""></div><div class="">* As an aside, it looks like some of these Kokkos-related functions and types are missing man pages - if you have time to add them, even as stubs, that'd be great (if not let me know and I'll just try to formally do it, so that at least the existence of the functions in the API is reflected on the website).</div><div class=""><br class=""></div><div class=""><div class=""><blockquote type="cite" class=""><div class="">Am 18.02.2021 um 23:17 schrieb Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer" class="">junchao.zhang@gmail.com</a>>:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank" rel="noreferrer" class="">fdkong.jd@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data. </div></div></blockquote><div class=""><br class=""></div><div class="">Just out of curiosity, is1 and is2 can not be created on a GPU device in the first place? That being said, it is technically impossible? Or we just did not implement them yet?<br class=""></div></div></div></blockquote><div class="">Simply because we do not have an ISCUDA class.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class=""></div><div class=""><br class=""></div><div class="">Fande,<br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">When the SF is used to communicate device data, indices are copied to the device..</div><div class=""><br class=""></div><div class="">--Junchao Zhang<br class=""></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com" target="_blank" rel="noreferrer" class="">patrick.sanan@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs. <div class=""><br class=""></div><div class="">Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c, </div><div class=""><br class=""></div><div class=""><code class=""><span class=""><span style="white-space:pre-wrap" class=""> </span>ierr</span> <span class="">=</span> <span class="">VecScatterCreate</span><span class="">(</span><span class="">x</span><span class="">,</span><span class="">is1</span><span class="">,</span><span class="">y</span><span class="">,</span><span class="">is2</span><span class="">,</span><span class="">&</span><span class="">ctx</span><span class="">);</span><span class="">CHKERRQ</span><span class="">(</span><span class="">ierr</span><span class="">);</span></code></div><div class=""><br class=""></div><div class="">(from <a href="https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44" target="_blank" rel="noreferrer" class="">https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44</a>)</div><div class=""><br class=""></div><div class="">Here, x and y can be set to type KOKKOS using -vec_type kokkos at the command line. But is1 and is2 are (I think), always</div><div class="">CPU/host data. Assuming that the scatter itself can happen on the GPU, the indices must make it to the device somehow - are they copied there when the scatter is created? Is there a way to create the scatter using indices already on the GPU (Maybe using SF more directly)?</div><div class=""><br class=""></div></div></blockquote></div>
</blockquote></div></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div>
</div></blockquote></div><br class=""></body></html>