<html><head><title>Re: [petsc-users] petsc4py & GPU</title>
</head>
<body>
<span style=" font-family:'courier new'; font-size: 9pt;">Hi,<br>
<br>
The memory limit (2GB) is set to memory size of a single GPU.<br>
For, let's say 10 nodes with 1 PPU each we have 10 cards with 20GB memory storage.<br>
Thus, when creating & loading a matrix with size > 2GB it should be (evenly) distributed across all nodes/GPU cards.<br>
The initial question was rather. Will:<br>
<br>
viewer = petsc4py.PETSc.Viewer().createBinary(aFilename, 'r')<br>
A = PETSc.Mat().create(comm=MPI.COMM_WORLD)<br>
A.setType(PETSc.Mat.Type.MPIAIJCUSPARSE)<br>
A.load(viewer)<br>
<br>
<span style=" font-family:'Tahoma'; font-size: 10pt;">do this job for me?<br>
<br>
Best,<br>
Damian<br>
<br>
<span style=" font-family:'courier new'; font-size: 9pt;">W liście datowanym 21 kwietnia 2017 (17:08:04) napisano:<br>
<br>
</span></span></span><table>
<tr>
<td width=2 bgcolor= #0000ff><br>
</td>
<td><span style=" font-family:'tahoma'; font-size: 10pt;">On Fri, Apr 21, 2017 at 9:27 AM, Damian Kaliszan <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;">> wrote:<br>
Hi Francesco, Matthew, Lisandro,<br>
<br>
Thank you a lot, it works:)<br>
<br>
However the next problem I'm facing is:<br>
- I'm trying to run run a problem (Ax=b) with the size of A (equals to<br>
3375x3375) using PBS on 10 nodes, each node has 2<br>
GPU cards with 2GB each.<br>
- 'qstat' shows these nodes are allocated, job<br>
is running<br>
- I use ksp/gmres<br>
- the error I get is as follows:<br>
=>> PBS: job killed: mem job total 5994876 kb exceeded limit 2048000 kb<br>
<br>
You exceeded a memory limit. This should be mailed to your cluster admin who can tell<br>
you how to schedule it appropriately.<br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[0]PETSC ERROR: or see </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">[0]PETSC ERROR: or try </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://valgrind.org">http://valgrind.org</a><span style=" font-family:'tahoma'; font-size: 10pt;"> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0] VecNorm_MPICUDA line 34 /home/users/damiank/petsc_bitbucket/src/vec/vec/impls/mpi/mpicuda/</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://mpicuda.cu">mpicuda.cu</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">[0]PETSC ERROR: [0] VecNorm line 217 /home/users/damiank/petsc_bitbucket/src/vec/vec/interface/rvector.c<br>
[0]PETSC ERROR: [0] VecNormalize line 317 /home/users/damiank/petsc_bitbucket/src/vec/vec/interface/rvector.c<br>
[0]PETSC ERROR: [0] KSPInitialResidual line 42 /home/users/damiank/petsc_bitbucket/src/ksp/ksp/interface/itres.c<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSPGMRESCycle line 122 /home/users/damiank/petsc_bitbucket/src/ksp/ksp/impls/gmres/gmres.c<br>
[0]PETSC ERROR: [0] KSPInitialResidual line 42 /home/users/damiank/petsc_bitbucket/src/ksp/ksp/interface/itres.c<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
[0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 277 /home/users/damiank/petsc_bitbucket/include/petsc/private/kspimpl.h<br>
<br>
- 'A' matrix is loaded in the following way (so as 'b' vector):<br>
viewer = petsc4py.PETSc.Viewer().createBinary(A, 'r')<br>
A = PETSc.Mat().create(comm=MPI.COMM_WORLD)<br>
A.setType(PETSc.Mat.Type.MPIAIJCUSPARSE)<br>
A.load(viewer)<br>
- Do you know what is going wrong?<br>
<br>
Best,<br>
Damian<br>
<br>
W liście datowanym 20 kwietnia 2017 (19:15:33) napisano:<br>
<br>
> Hi Damian,<br>
<br>
> You can use the "parse_known_args" method of the ArgumentParser<br>
> class; it will create a Namespace with the args you defined and<br>
> return the uknown ones as a list of strings you can pass to petsc4py.init.<br>
> See:<br>
> </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="https://docs.python.org/3.6/library/argparse.html">https://docs.python.org/3.6/library/argparse.html</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> section 16.4.5.7.<br>
<br>
> I used it successfully in the past. Hope this helps.<br>
<br>
> Best,<br>
> --<br>
> Francesco Caimmi<br>
<br>
> Laboratorio di Ingegneria dei Polimeri<br>
> </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://www.chem.polimi.it/polyenglab/">http://www.chem.polimi.it/polyenglab/</a><br>
<br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> Politecnico di Milano - Dipartimento di Chimica,<br>
> Materiali e Ingegneria Chimica “Giulio Natta”<br>
<br>
> P.zza Leonardo da Vinci, 32<br>
> I-20133 Milano<br>
> Tel. </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="tel:%2B39.02.2399.4711">+39.02.2399.4711</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> Fax </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="tel:%2B39.02.7063.8173">+39.02.7063.8173</a><br>
<br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:francesco.caimmi@polimi.it">francesco.caimmi@polimi.it</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> Skype: fmglcaimmi<br>
<br>
> ________________________________________<br>
> From: </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a><span style=" font-family:'tahoma'; font-size: 10pt;">> on behalf of Damian Kaliszan <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;">><br>
> Sent: Thursday, April 20, 2017 6:26 PM<br>
> To: Lisandro Dalcin<br>
> Cc: PETSc<br>
> Subject: Re: [petsc-users] petsc4py & GPU<br>
<br>
> Hi,<br>
> There might be the problem because I'm using ArgumentParser class<br>
> to catch complex command line arguments. In this case is there any<br>
> chance to make both to cooperate or the only solution is to pass<br>
> everything through argument to init method of petsc4py?<br>
> Best,<br>
> Damian<br>
> W dniu 20 kwi 2017, o 17:37, użytkownik Lisandro Dalcin<br>
> <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:dalcinl@gmail.com">dalcinl@gmail.com</a><span style=" font-family:'tahoma'; font-size: 10pt;"><mailto:</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:dalcinl@gmail.com">dalcinl@gmail.com</a><span style=" font-family:'tahoma'; font-size: 10pt;">>> napisał:<br>
<br>
> On 20 April 2017 at 17:09, Damian Kaliszan<br>
> <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;"><mailto:</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;">>> wrote:<br>
> Thank you for reply:) Sorry for maybe stupid question in the scope of setting petsc(4py) options.<br>
> Should the following calls (somewhere before creating matrix & vectors):<br>
<br>
> PETSc.Options().setValue("ksp_view", "")<br>
> PETSc.Options().setValue("log_view", "")<br>
<br>
> Unfortunately, no. There are a few options (-log_view ?) that you<br>
> should set before calling PetscInitialize() (which happens<br>
> automatically at import time), otherwise things do not work as<br>
> expected. To pass things from the command line and set them before<br>
> PetscInitialize() the usual idiom is:<br>
<br>
> import sys, petsc4py<br>
> petsc4py.init(sys.argv)<br>
> from petsc4py import PETSc<br>
<br>
<br>
<br>
> --<br>
> Lisandro Dalcin<br>
> ============<br>
> Research Scientist<br>
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)<br>
> Extreme Computing Research Center (ECRC)<br>
> King Abdullah University of Science and Technology (KAUST)<br>
> </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://ecrc.kaust.edu.sa/">http://ecrc.kaust.edu.sa/</a><br>
<br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> 4700 King Abdullah University of Science and Technology<br>
> al-Khawarizmi Bldg (Bldg 1), Office # 0109<br>
> Thuwal 23955-6900, Kingdom of Saudi Arabia<br>
> </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://www.kaust.edu.sa">http://www.kaust.edu.sa</a><br>
<br>
<span style=" font-family:'tahoma'; font-size: 10pt;">> Office Phone: </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="tel:%2B966%2012%20808-0459">+966 12 808-0459</a><br>
<br>
<br>
<br>
<span style=" font-family:'tahoma'; font-size: 10pt;">-------------------------------------------------------<br>
Damian Kaliszan<br>
<br>
Poznan Supercomputing and Networking Center<br>
HPC and Data Centres Technologies<br>
ul. Jana Pawła II 10<br>
61-139 Poznan<br>
POLAND<br>
<br>
phone </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="tel:%28%2B48%2061%29%20858%205109">(+48 61) 858 5109</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;">e-mail </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;"> www - </span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://www.man.poznan.pl/">http://www.man.poznan.pl/</a><br>
<span style=" font-family:'tahoma'; font-size: 10pt;"> -------------------------------------------------------<br>
<br>
<br>
<br>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</td>
</tr>
</table>
<br><br>
<br>
<br>
<span style=" font-family:'Tahoma'; font-size: 10pt;">-------------------------------------------------------<br>
Damian Kaliszan<br>
<br>
Poznan Supercomputing and Networking Center<br>
HPC and Data Centres Technologies<br>
ul. Jana Pawła II 10<br>
61-139 Poznan<br>
POLAND<br>
<br>
phone (+48 61) 858 5109<br>
e-mail </span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><br>
<span style=" font-family:'courier new'; font-size: 9pt;">www - </span><a style=" font-family:'courier new'; font-size: 9pt;" href="http://www.man.poznan.pl/">http://www.man.poznan.pl/</a><br>
<span style=" font-family:'courier new'; font-size: 9pt;">------------------------------------------------------- </body></html>