<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div>PetscMallocSetCUDAHost() switches from using the regular malloc on the CPU to using cudaMallocHost() it also switches the free.These means between the PetscMallocSetCUDAHost() and the PetscMallocResetCUDAHost() all mallocs are done with cudaHost version and so are all frees.</div><div><br class=""></div><div>If any memory that was allocated before the call to PetscMallocSetCUDAHost() so it was allocated with a regular malloc is freed inside the block it will be freed with the incorrect cudaHostFree and will crash. This makes these routines very fragile </div><div><br class=""></div><div>I don't understand the purpose of PetscMallocSetCUDAHost(), possibly it is intended to be used with Nvidia unified memory so the same addresses can be used on the GPU. PETSc does not use or need unified memory in its programming model for GPUs. </div><div><br class=""></div><div>As far as I am aware you don't have any reason to use these routines.</div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 25, 2020, at 6:46 PM, Sajid Ali <<a href="mailto:sajidsyed2021@u.northwestern.edu" class="">sajidsyed2021@u.northwestern.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="markdown-here-wrapper" style=""><p style="margin:0px 0px 1.2em!important" class="">Hi PETSc-developers, </p><p style="margin:0px 0px 1.2em!important" class="">Is it valid to allocate matrix values on host for use on a GPU later by embedding all allocation logic (i.e the code block that calls PetscMalloc1 for values and indices and sets them using MatSetValues) within a section marked by PetscMalloc(Re)SetCUDAHost ? </p><p style="margin:0px 0px 1.2em!important" class="">My understanding was that PetscMallocSetCUDAHost would set mallocs to be on the host but I’m getting an error as shown below (for some strange reason it happens to be the 5th column on the 0th row (if that helps) both when setting one value at a time and when setting the whole 0th row together):</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px" class=""><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline;white-space:pre;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important" class="">[sajid@xrmlite cuda]$ mpirun -np 1 ~/packages/pirt/src/pirt -inputfile shepplogan.h5
PIRT -- Parallel Iterative Reconstruction Tomography
Reading in real data from shepplogan.h5
After loading data, nTau:100, nTheta:50
After detector geometry context initialization
Initialized PIRT
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: cuda error 1 (cudaErrorInvalidValue) : invalid argument
[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.13.2-947-gc2372adeb2 GIT Date: 2020-08-25 21:07:25 +0000
[0]PETSC ERROR: /home/sajid/packages/pirt/src/pirt on a arch-linux-c-debug named xrmlite by sajid Tue Aug 25 18:30:55 2020
[0]PETSC ERROR: Configure options --with-hdf5=1 --with-cuda=1
[0]PETSC ERROR: #1 PetscCUDAHostFree() line 14 in /home/sajid/packages/petsc/src/sys/memory/cuda/<a href="http://mcudahost.cu/" class="">mcudahost.cu</a>
[0]PETSC ERROR: #2 PetscFreeA() line 475 in /home/sajid/packages/petsc/src/sys/memory/mal.c
[0]PETSC ERROR: #3 MatSeqXAIJFreeAIJ() line 135 in /home/sajid/packages/petsc/include/../src/mat/impls/aij/seq/aij.h
[0]PETSC ERROR: #4 MatSetValues_SeqAIJ() line 498 in /home/sajid/packages/petsc/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #5 MatSetValues() line 1392 in /home/sajid/packages/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #6 setMatrixElements() line 248 in /home/sajid/packages/pirt/src/geom.cxx
[0]PETSC ERROR: #7 construct_matrix() line 91 in /home/sajid/packages/pirt/src/<a href="http://matrix.cu/" class="">matrix.cu</a>
[0]PETSC ERROR: #8 main() line 20 in /home/sajid/packages/pirt/src/pirt.cxx
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -inputfile shepplogan.h5
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to <a href="mailto:petsc-maint@mcs.anl.gov" class="">petsc-maint@mcs.anl.gov</a>----------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF
with errorcode 20076.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[sajid@xrmlite cuda]$
</code></pre><p style="margin:0px 0px 1.2em!important" class="">PetscCUDAHostFree is called within the PetscMalloc(Re)SetCUDAHost block as described earlier which should’ve created valid memory on the host. </p><p style="margin:0px 0px 1.2em!important" class="">Could someone explain if this is the correct approach to take and what the above error means ?</p><p style="margin:0px 0px 1.2em!important" class="">(PS : I’ve run ksp tutorial-ex2 with -vec_type cuda -mat_type aijcusparse to test the installation and everything works as expected.)</p><p style="margin:0px 0px 1.2em!important" class="">Thank You,<br class="">Sajid Ali | PhD Candidate<br class="">Applied Physics<br class="">Northwestern University<br class=""><a href="http://s-sajid-ali.github.io/" class="">s-sajid-ali.github.io</a></p>
<div title="MDH:PGRpdj48ZGl2PjxkaXY+PGRpdj5IaSBQRVRTYy1kZXZlbG9wZXJzLCA8YnI+PGJyPjwvZGl2Pklz
IGl0IHZhbGlkIHRvIGFsbG9jYXRlIG1hdHJpeCB2YWx1ZXMgb24gaG9zdCBmb3IgdXNlIG9uIGEg
R1BVIGxhdGVyIGJ5IGVtYmVkZGluZyBhbGwgYWxsb2NhdGlvbiBsb2dpYyAoaS5lIHRoZSBjb2Rl
IGJsb2NrIHRoYXQgY2FsbHMgUGV0c2NNYWxsb2MxIGZvciB2YWx1ZXMgYW5kIGluZGljZXMgYW5k
IHNldHMgdGhlbSB1c2luZyBNYXRTZXRWYWx1ZXMpIHdpdGhpbiBhIHNlY3Rpb24gbWFya2VkIGJ5
IFBldHNjTWFsbG9jKFJlKVNldENVREFIb3N0ID8gPGJyPjxicj48L2Rpdj5NeSB1bmRlcnN0YW5k
aW5nIHdhcyB0aGF0IFBldHNjTWFsbG9jU2V0Q1VEQUhvc3Qgd291bGQgc2V0IG1hbGxvY3MgdG8g
YmUgb24gdGhlIGhvc3QgYnV0IEknbSBnZXR0aW5nIGFuIGVycm9yIGFzIHNob3duIGJlbG93IChm
b3Igc29tZSBzdHJhbmdlIHJlYXNvbiBpdCBoYXBwZW5zIHRvIGJlIHRoZSA1dGggY29sdW1uIG9u
IHRoZSAwdGggcm93IChpZiB0aGF0IGhlbHBzKSBib3RoIHdoZW4gc2V0dGluZyBvbmUgdmFsdWUg
YXQgYSB0aW1lIGFuZCB3aGVuIHNldHRpbmcgdGhlIHdob2xlIDB0aCByb3cgdG9nZXRoZXIpOjxi
cj5gYGA8YnI+W3NhamlkQHhybWxpdGUgY3VkYV0kIG1waXJ1biAtbnAgMSB+L3BhY2thZ2VzLzxz
cGFuIHpldW00YzU9IlBSXzNfMCIgZGF0YS1kZG53YWI9IlBSXzNfMCIgYXJpYS1pbnZhbGlkPSJz
cGVsbGluZyIgY2xhc3M9IkxJIG5nIj5waXJ0PC9zcGFuPi9zcmMvPHNwYW4gemV1bTRjNT0iUFJf
NF8wIiBkYXRhLWRkbndhYj0iUFJfNF8wIiBhcmlhLWludmFsaWQ9InNwZWxsaW5nIiBjbGFzcz0i
TEkgbmciPnBpcnQ8L3NwYW4+IC1pbnB1dGZpbGUgPHNwYW4gemV1bTRjNT0iUFJfNV8wIiBkYXRh
LWRkbndhYj0iUFJfNV8wIiBhcmlhLWludmFsaWQ9InNwZWxsaW5nIiBjbGFzcz0iTEkgbmciPnNo
ZXBwbG9nYW4uaDU8L3NwYW4+PGJyPlBJUlQgLS0gUGFyYWxsZWwgSXRlcmF0aXZlIFJlY29uc3Ry
dWN0aW9uIFRvbW9ncmFwaHk8YnI+UmVhZGluZyBpbiByZWFsIGRhdGEgZnJvbSA8c3BhbiB6ZXVt
NGM1PSJQUl82XzAiIGRhdGEtZGRud2FiPSJQUl82XzAiIGFyaWEtaW52YWxpZD0ic3BlbGxpbmci
IGNsYXNzPSJMSSBuZyI+c2hlcHBsb2dhbi5oNTwvc3Bhbj48YnI+QWZ0ZXIgbG9hZGluZyBkYXRh
LCBuVGF1OjEwMCwgblRoZXRhOjUwPGJyPkFmdGVyIGRldGVjdG9yIGdlb21ldHJ5IGNvbnRleHQg
aW5pdGlhbGl6YXRpb248YnI+SW5pdGlhbGl6ZWQgUElSVDxicj5bMF1QRVRTQyBFUlJPUjogLS0t
LS0tLS0tLS0tLS0tLS0tLS0tIEVycm9yIE1lc3NhZ2UgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS08YnI+WzBdUEVUU0MgRVJST1I6
IEVycm9yIGluIGV4dGVybmFsIGxpYnJhcnk8YnI+WzBdUEVUU0MgRVJST1I6IGN1ZGEgZXJyb3Ig
MSAoY3VkYUVycm9ySW52YWxpZFZhbHVlKSA6IGludmFsaWQgYXJndW1lbnQ8YnI+WzBdUEVUU0Mg
RVJST1I6IFNlZSBodHRwczovL3d3dy5tY3MuYW5sLmdvdi9wZXRzYy9kb2N1bWVudGF0aW9uL2Zh
cS5odG1sIGZvciB0cm91YmxlIHNob290aW5nLjxicj5bMF1QRVRTQyBFUlJPUjogUGV0c2MgRGV2
ZWxvcG1lbnQgR0lUIHJldmlzaW9uOiB2My4xMy4yLTk0Ny1nYzIzNzJhZGViMiAmbmJzcDtHSVQg
RGF0ZTogMjAyMC0wOC0yNSAyMTowNzoyNSArMDAwMDxicj5bMF1QRVRTQyBFUlJPUjogL2hvbWUv
c2FqaWQvcGFja2FnZXMvPHNwYW4gemV1bTRjNT0iUFJfN18wIiBkYXRhLWRkbndhYj0iUFJfN18w
IiBhcmlhLWludmFsaWQ9InNwZWxsaW5nIiBjbGFzcz0iTEkgbmciPnBpcnQ8L3NwYW4+L3NyYy88
c3BhbiB6ZXVtNGM1PSJQUl84XzAiIGRhdGEtZGRud2FiPSJQUl84XzAiIGFyaWEtaW52YWxpZD0i
c3BlbGxpbmciIGNsYXNzPSJMSSBuZyI+cGlydDwvc3Bhbj4gb24gYSBhcmNoLWxpbnV4LWMtZGVi
dWcgbmFtZWQgeHJtbGl0ZSBieSBzYWppZCBUdWUgQXVnIDI1IDE4OjMwOjU1IDIwMjA8YnI+WzBd
UEVUU0MgRVJST1I6IENvbmZpZ3VyZSBvcHRpb25zIC0td2l0aC1oZGY1PTEgLS13aXRoLWN1ZGE9
MTxicj5bMF1QRVRTQyBFUlJPUjogIzEgUGV0c2NDVURBSG9zdEZyZWUoKSBsaW5lIDE0IGluIC9o
b21lL3NhamlkL3BhY2thZ2VzL3BldHNjL3NyYy9zeXMvbWVtb3J5L2N1ZGEvbWN1ZGFob3N0LmN1
PGJyPlswXVBFVFNDIEVSUk9SOiAjMiA8c3BhbiB6ZXVtNGM1PSJQUl85XzAiIGRhdGEtZGRud2Fi
PSJQUl85XzAiIGFyaWEtaW52YWxpZD0ic3BlbGxpbmciIGNsYXNzPSJMSSBuZyI+UGV0c2NGcmVl
QTwvc3Bhbj4oKSBsaW5lIDQ3NSBpbiAvaG9tZS9zYWppZC9wYWNrYWdlcy9wZXRzYy9zcmMvc3lz
L21lbW9yeS9tYWwuYzxicj5bMF1QRVRTQyBFUlJPUjogIzMgTWF0U2VxWEFJSkZyZWVBSUooKSBs
aW5lIDEzNSBpbiAvaG9tZS9zYWppZC9wYWNrYWdlcy9wZXRzYy9pbmNsdWRlLy4uL3NyYy9tYXQv
aW1wbHMvYWlqL3NlcS9haWouaDxicj5bMF1QRVRTQyBFUlJPUjogIzQgTWF0U2V0VmFsdWVzX1Nl
cUFJSigpIGxpbmUgNDk4IGluIC9ob21lL3NhamlkL3BhY2thZ2VzL3BldHNjL3NyYy9tYXQvaW1w
bHMvYWlqL3NlcS9haWouYzxicj5bMF1QRVRTQyBFUlJPUjogIzUgTWF0U2V0VmFsdWVzKCkgbGlu
ZSAxMzkyIGluIC9ob21lL3NhamlkL3BhY2thZ2VzL3BldHNjL3NyYy9tYXQvaW50ZXJmYWNlL21h
dHJpeC5jPGJyPlswXVBFVFNDIEVSUk9SOiAjNiBzZXRNYXRyaXhFbGVtZW50cygpIGxpbmUgMjQ4
IGluIC9ob21lL3NhamlkL3BhY2thZ2VzL3BpcnQvc3JjL2dlb20uY3h4PGJyPlswXVBFVFNDIEVS
Uk9SOiAjNyBjb25zdHJ1Y3RfbWF0cml4KCkgbGluZSA5MSBpbiAvaG9tZS9zYWppZC9wYWNrYWdl
cy9waXJ0L3NyYy9tYXRyaXguY3U8YnI+WzBdUEVUU0MgRVJST1I6ICM4IG1haW4oKSBsaW5lIDIw
IGluIC9ob21lL3NhamlkL3BhY2thZ2VzLzxzcGFuIHpldW00YzU9IlBSXzEwXzAiIGRhdGEtZGRu
d2FiPSJQUl8xMF8wIiBhcmlhLWludmFsaWQ9InNwZWxsaW5nIiBjbGFzcz0iTEkgbmciPnBpcnQ8
L3NwYW4+L3NyYy88c3BhbiB6ZXVtNGM1PSJQUl8xMV8wIiBkYXRhLWRkbndhYj0iUFJfMTFfMCIg
YXJpYS1pbnZhbGlkPSJzcGVsbGluZyIgY2xhc3M9IkxJIG5nIj5waXJ0LmN4eDwvc3Bhbj48YnI+
WzBdUEVUU0MgRVJST1I6IFBFVFNjIE9wdGlvbiBUYWJsZSBlbnRyaWVzOjxicj5bMF1QRVRTQyBF
UlJPUjogLWlucHV0ZmlsZSA8c3BhbiB6ZXVtNGM1PSJQUl8xMl8wIiBkYXRhLWRkbndhYj0iUFJf
MTJfMCIgYXJpYS1pbnZhbGlkPSJzcGVsbGluZyIgY2xhc3M9IkxJIG5nIj5zaGVwcGxvZ2FuLmg1
PC9zcGFuPjxicj5bMF1QRVRTQyBFUlJPUjogLS0tLS0tLS0tLS0tLS0tLUVuZCBvZiBFcnJvciBN
ZXNzYWdlIC0tLS0tLS1zZW5kIGVudGlyZSBlcnJvciBtZXNzYWdlIHRvIHBldHNjLW1haW50QG1j
cy5hbmwuZ292LS0tLS0tLS0tLTxicj4tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLTxicj5NUElfQUJPUlQgd2Fz
IGludm9rZWQgb24gcmFuayAwIGluIGNvbW11bmljYXRvciBNUElfQ09NTV9TRUxGPGJyPndpdGgg
ZXJyb3Jjb2RlIDIwMDc2Ljxicj48YnI+Tk9URTogaW52b2tpbmcgTVBJX0FCT1JUIGNhdXNlcyBP
cGVuIE1QSSB0byBraWxsIGFsbCBNUEkgcHJvY2Vzc2VzLjxicj5Zb3UgbWF5IG9yIG1heSBub3Qg
c2VlIG91dHB1dCBmcm9tIG90aGVyIHByb2Nlc3NlcywgZGVwZW5kaW5nIG9uPGJyPmV4YWN0bHkg
d2hlbiBPcGVuIE1QSSBraWxscyB0aGVtLjxicj4tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLTxicj5bc2FqaWRA
eHJtbGl0ZSBjdWRhXSQ8YnI+YGBgPGJyPiZuYnNwOyA8YnI+PC9kaXY+UGV0c2NDVURBSG9zdEZy
ZWUgaXMgY2FsbGVkIHdpdGhpbiB0aGUgClBldHNjTWFsbG9jKFJlKVNldENVREFIb3N0IGJsb2Nr
IGFzIGRlc2NyaWJlZCBlYXJsaWVyIHdoaWNoIHNob3VsZCd2ZSBjcmVhdGVkIHZhbGlkIG1lbW9y
eSBvbiB0aGUgaG9zdC4gPGJyPjxicj5Db3VsZCBzb21lb25lIGV4cGxhaW4gaWYgdGhpcyBpcyB0
aGUgY29ycmVjdCBhcHByb2FjaCB0byB0YWtlIGFuZCB3aGF0IHRoZSBhYm92ZSBlcnJvciBtZWFu
cyA/PGJyPjxicj48L2Rpdj4oUFMgOiBJJ3ZlIHJ1biBrc3AgdHV0b3JpYWwtZXgyIHdpdGggLXZl
Y190eXBlIGN1ZGEgLW1hdF90eXBlIDxzcGFuIHpldW00YzU9IlBSXzE1XzAiIGRhdGEtZGRud2Fi
PSJQUl8xNV8wIiBhcmlhLWludmFsaWQ9InNwZWxsaW5nIiBjbGFzcz0iTEkgbmciPmFpamN1c3Bh
cnNlPC9zcGFuPiB0byB0ZXN0IHRoZSBpbnN0YWxsYXRpb24gYW5kIGV2ZXJ5dGhpbmcgd29ya3Mg
YXMgZXhwZWN0ZWQuKTxicj48ZGl2PjxkaXY+PGRpdj48ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+
VGhhbmsgWW91LDxicj48L2Rpdj48ZGl2PjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9zaWdu
YXR1cmUiIGRhdGEtc21hcnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPjxkaXYgZGlyPSJsdHIiPjxk
aXY+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2IGRpcj0ibHRyIj48ZGl2IHN0eWxlPSJmb250LXNp
emU6MTIuOHB4Ij5TYWppZCBBbGkgfCBQaEQgQ2FuZGlkYXRlPGJyPjwvZGl2PjxkaXYgc3R5bGU9
ImZvbnQtc2l6ZToxMi44cHgiPkFwcGxpZWQgUGh5c2ljczxicj48L2Rpdj48ZGl2IHN0eWxlPSJm
b250LXNpemU6MTIuOHB4Ij5Ob3J0aHdlc3Rlcm4gVW5pdmVyc2l0eTwvZGl2PjxkaXYgc3R5bGU9
ImZvbnQtc2l6ZToxMi44cHgiPjxhIGhyZWY9Imh0dHA6Ly9zLXNhamlkLWFsaS5naXRodWIuaW8i
IHRhcmdldD0iX2JsYW5rIj5zLXNhamlkLWFsaS5naXRodWIuaW88L2E+PC9kaXY+PC9kaXY+PC9k
aXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0" class=""></div></div></div>
</div></blockquote></div><br class=""></body></html>