<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hello,<div class=""><br class=""/></div><div class="">I would like to report that after recompiling openmpi with cuda support and recompiling petsc with the option</div><div class=""><br class=""/></div><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><div class=""><blockquote type="cite" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class="">--download-hypre-configure-arguments=--enable-unified-memory </div></div></div></blockquote></div></div></div></blockquote></div></blockquote></div></div></div></blockquote><div class=""><br class=""/></div><div class="">I still got the error </div><div class=""><br class=""/></div><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class="">CUDA ERROR (code = 101, invalid device ordinal) at memory.c:139</div></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote><br class=""/></div><div class="">when using hypre.</div><div class=""><br class=""/></div><div class="">So it is not an issue of mpi cuda support. I ran ex19 without the "-use_gpu_aware_mpi 0” option. Maybe it has something to do with the Tesla K40m GPUs. However, on their website:</div><div class=""><br class=""/></div><div class="">“In CUDA 6, Unified Memory is supported starting with the Kepler GPU architecture (Compute Capability 3.0 or higher), on 64-bit Windows 7, 8, and Linux operating systems (Kernel 2.6.18+).”</div><div class=""><br class=""/></div><div class="">In any case, thanks again for helping out with this.</div><div class=""><br class=""/></div><div class="">Juan S.</div><div class=""><br class=""/></div><div><br class=""/></div></div></body></html>