<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div><br></div>It would be nice if we could automatically detect this issue more often.<div><br></div><div> Barry</div><div><br><div style=""><br><div>Begin forwarded message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; color:rgba(0, 0, 0, 1.0);"><b>From: </b></span><span style="font-family:'Helvetica';">Mathis Friesdorf <<a href="mailto:mathisfriesdorf@gmail.com">mathisfriesdorf@gmail.com</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; color:rgba(0, 0, 0, 1.0);"><b>Subject: </b></span><span style="font-family:'Helvetica';"><b>Re: [petsc-users] Unexpected "Out of memory error" with SLEPC</b><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; color:rgba(0, 0, 0, 1.0);"><b>Date: </b></span><span style="font-family:'Helvetica';">June 26, 2014 at 5:08:23 AM CDT<br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; color:rgba(0, 0, 0, 1.0);"><b>To: </b></span><span style="font-family:'Helvetica';">Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at">rupp@iue.tuwien.ac.at</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; color:rgba(0, 0, 0, 1.0);"><b>Cc: </b></span><span style="font-family:'Helvetica';"><<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>, <<a href="mailto:slepc-maint@grycap.upv.es">slepc-maint@grycap.upv.es</a>><br></span></div><br><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Dear Karl,<br><br>thanks a lot! This indeed worked. I recompiled PETSC with<br><br>./configure --with-64-bit-indices=1<br>PETSC_ARCH = arch-linux2-c-debug-int64<br><br>and the error is gone. Again thanks for your help! <br>
<br>All the best, Mathis<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 25, 2014 at 12:42 PM, Karl Rupp <span dir="ltr"><<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mathis,<br>
<br>
this looks very much like an integer overflow:<br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#with-64-bit-indices" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/faq.html#with-<u></u>64-bit-indices</a><br>
<br>
Best regards,<br>
Karli<div class=""><br>
<br>
On 06/25/2014 12:31 PM, Mathis Friesdorf wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
Dear all,<br>
<br>
after a very useful email exchange with Jed Brown quite a while ago, I<br>
was able to find the lowest eigenvalue of a large matrix which is<br>
constructed as a tensor product. Admittedly the solution is a bit<br>
hacked, but is based on a Matrix shell and Armadillo and therefore<br>
reasonably fast. The problem seems to work well for smaller systems, but<br>
once the vectors reach a certain size, I get "out of memory" errors. I<br>
have tested the initialization of a vector of that size and<br>
multiplication by the matrix. This works fine and takes roughly 20GB of<br>
memory. There are 256 GB available, so I see no reason why the esp<br>
solvers should complain. Does anyone have an idea what goes wrong here?<br>
The error message is not very helpful and claims that a memory is<br></div>
requested that is way beyond any reasonable number: /Memory requested<br>
18446744056529684480./<div class=""><br>
<br>
Thanks and all the best, Mathis Friesdorf<br>
<br>
<br></div>
*Output of the Program:*<br>
/mathis@n180:~/localisation$ ./local_plus 27/<br>
/System Size: 27<div><div class="h5"><br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
[[30558,1],0]: A high-performance Open MPI point-to-point messaging module<br>
was unable to find any relevant network interfaces:<br>
<br>
Module: OpenFabrics (openib)<br>
Host: n180<br>
<br>
Another transport will be used instead, although this may result in<br>
lower performance.<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
[0]PETSC ERROR: --------------------- Error Message<br>
------------------------------<u></u>------<br>
[0]PETSC ERROR: Out of memory. This could be due to allocating<br>
[0]PETSC ERROR: too large an object or bleeding by not properly<br>
[0]PETSC ERROR: destroying unneeded objects.<br>
[0]PETSC ERROR: Memory allocated 3221286704 Memory used by process<br>
3229827072<br>
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<br>
[0]PETSC ERROR: Memory requested 18446744056529684480!<br>
[0]PETSC ERROR:<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: Petsc Release Version 3.4.4, Mar, 13, 2014<br>
[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html for manual pages.<br>
[0]PETSC ERROR:<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: ./local_plus on a arch-linux2-cxx-debug named n180 by<br>
mathis Wed Jun 25 12:23:01 2014<br>
[0]PETSC ERROR: Libraries linked from /home/mathis/bin_nodebug/lib<br>
[0]PETSC ERROR: Configure run at Wed Jun 25 00:03:34 2014<br>
[0]PETSC ERROR: Configure options PETSC_DIR=/home/mathis/petsc-<u></u>3.4.4<br>
--with-debugging=1 COPTFLAGS="-O3 -march=p4 -mtune=p4" --with-fortran=0<br>
-with-mpi=1 --with-mpi-dir=/usr/lib/<u></u>openmpi --with-clanguage=cxx<br>
--prefix=/home/mathis/bin_<u></u>nodebug<br>
[0]PETSC ERROR:<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
[0]PETSC ERROR: PetscMallocAlign() line 46 in<br>
/home/mathis/petsc-3.4.4/src/<u></u>sys/memory/mal.c<br>
[0]PETSC ERROR: PetscTrMallocDefault() line 189 in<br>
/home/mathis/petsc-3.4.4/src/<u></u>sys/memory/mtr.c<br>
[0]PETSC ERROR: VecDuplicateVecs_Contiguous() line 62 in<br>
src/vec/contiguous.c<br>
[0]PETSC ERROR: VecDuplicateVecs() line 589 in<br>
/home/mathis/petsc-3.4.4/src/<u></u>vec/vec/interface/vector.c<br>
[0]PETSC ERROR: EPSAllocateSolution() line 51 in src/eps/interface/mem.c<br>
[0]PETSC ERROR: EPSSetUp_KrylovSchur() line 141 in<br>
src/eps/impls/krylov/<u></u>krylovschur/krylovschur.c<br>
[0]PETSC ERROR: EPSSetUp() line 147 in src/eps/interface/setup.c<br>
[0]PETSC ERROR: EPSSolve() line 90 in src/eps/interface/solve.c<br>
[0]PETSC ERROR: main() line 48 in local_plus.cpp<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>
with errorcode 55.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
<br></div></div>
/<br>
*Output of make:<br>
*/mathis@n180:~/localisation$ make local_plus<div class=""><br>
mpicxx -o local_plus.o -c -Wall -Wwrite-strings -Wno-strict-aliasing<br>
-Wno-unknown-pragmas -g -fPIC<br>
-I/home/mathis/armadillo-4.<u></u>300.8/include -lblas -llapack<br>
-L/home/mathis/armadillo-4.<u></u>300.8 -O3 -larmadillo -fomit-frame-pointer<br>
-I/home/mathis/bin_nodebug/<u></u>include -I/home/mathis/bin_nodebug/<u></u>include<br>
-I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/<u></u>openmpi<br>
-D__INSDIR__= -I/home/mathis/bin_nodebug<br>
-I/home/mathis/bin_nodebug//<u></u>include -I/home/mathis/bin_nodebug/<u></u>include<br>
local_plus.cpp<br>
local_plus.cpp:22:0: warning: "__FUNCT__" redefined [enabled by default]<br>
In file included from /home/mathis/bin_nodebug/<u></u>include/petscvec.h:10:0,<br>
from local_plus.cpp:10:<br>
/home/mathis/bin_nodebug/<u></u>include/petscviewer.h:386:0: note: this is the<br>
location of the previous definition<br>
g++ -o local_plus local_plus.o -Wl,-rpath,/home/mathis/bin_<u></u>nodebug//lib<br>
-L/home/mathis/bin_nodebug//<u></u>lib -lslepc<br>
-Wl,-rpath,/home/mathis/bin_<u></u>nodebug/lib -L/home/mathis/bin_nodebug/lib<br>
-lpetsc -llapack -lblas -lpthread -Wl,-rpath,/usr/lib/openmpi/<u></u>lib<br>
-L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_<u></u>64-linux-gnu/4.7<br>
-L/usr/lib/gcc/x86_64-linux-<u></u>gnu/4.7 -Wl,-rpath,/usr/lib/x86_64-<u></u>linux-gnu<br>
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-<u></u>gnu<br>
-L/lib/x86_64-linux-gnu -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte<br>
-lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl<br>
/bin/rm -f local_plus.o<br></div>
/<br>
*Code:<br>
*///Author: Mathis Friesdorf<br>
//<a href="mailto:MathisFriesdorf@gmail.com" target="_blank">MathisFriesdorf@gmail.com</a> <mailto:<a href="mailto:MathisFriesdorf@gmail.com" target="_blank">MathisFriesdorf@gmail.<u></u>com</a>><div><div class="h5">
<br>
<br>
static char help[] = "1D chain\n";<br>
<br>
#include <iostream><br>
#include <typeinfo><br>
#include <armadillo><br>
#include <petscsys.h><br>
#include <petscvec.h><br>
#include <petscmat.h><br>
#include <slepcsys.h><br>
#include <slepceps.h><br>
#include <math.h><br>
#include <assert.h><br>
<br>
PetscErrorCode BathMult(Mat H, Vec x,Vec y);<br>
PetscInt L=30,d=2,dsys;<br>
PetscErrorCode ierr;<br>
arma::mat hint = "1.0 0 0 0.0; 0 -1.0 2.0 0; 0 2.0 -1.0 0; 0 0 0 1.0;";<br>
<br>
#define __FUNCT__ "main"<br>
int main(int argc, char **argv)<br>
{<br>
Mat H;<br>
EPS eps;<br>
Vec xr,xi;<br>
PetscScalar kr,ki;<br>
PetscInt j, nconv;<br>
<br>
L = strtol(argv[1],NULL,10);<br>
dsys = pow(d,L);<br>
printf("%s","System Size: ");<br>
printf("%i",L);<br>
printf("%s","\n");<br>
SlepcInitialize(&argc,&argv,(<u></u>char*)0,help);<br>
<br>
MatCreateShell(PETSC_COMM_<u></u>WORLD,dsys,dsys,dsys,dsys,<u></u>NULL,&H);<br>
MatShellSetOperation(H,MATOP_<u></u>MULT,(void(*)())BathMult);<br>
ierr = MatGetVecs(H,NULL,&xr); CHKERRQ(ierr);<br>
ierr = MatGetVecs(H,NULL,&xi); CHKERRQ(ierr);<br>
<br>
ierr = EPSCreate(PETSC_COMM_WORLD, &eps); CHKERRQ(ierr);<br>
ierr = EPSSetOperators(eps, H, NULL); CHKERRQ(ierr);<br>
ierr = EPSSetProblemType(eps, EPS_HEP); CHKERRQ(ierr);<br>
ierr = EPSSetWhichEigenpairs(eps,EPS_<u></u>SMALLEST_REAL); CHKERRQ(ierr);<br>
ierr = EPSSetFromOptions( eps ); CHKERRQ(ierr);<br>
ierr = EPSSolve(eps); CHKERRQ(ierr);<br>
ierr = EPSGetConverged(eps, &nconv); CHKERRQ(ierr);<br>
for (j=0; j<1; j++) {<br>
EPSGetEigenpair(eps, j, &kr, &ki, xr, xi);<br>
printf("%s","Lowest Eigenvalue: ");<br>
PetscPrintf(PETSC_COMM_WORLD,"<u></u>%9F",kr);<br>
PetscPrintf(PETSC_COMM_WORLD,"<u></u>\n");<br>
}<br>
EPSDestroy(&eps);<br>
<br>
ierr = SlepcFinalize();<br>
return 0;<br>
}<br>
#undef __FUNCT__<br>
<br>
#define __FUNCT__ "BathMult"<br>
PetscErrorCode BathMult(Mat H, Vec x, Vec y)<br>
{<br>
PetscInt l;<br>
uint slice;<br>
PetscScalar *arrayin,*arrayout;<br>
<br>
VecGetArray(x,&arrayin);<br>
VecGetArray(y,&arrayout);<br>
arma::cube A = arma::cube(arrayin,1,1,pow(d,<u></u>L),<br>
/*copy_aux_mem*/false,/*<u></u>strict*/true);<br>
arma::mat result = arma::mat(arrayout,pow(d,L),1,<br>
/*copy_aux_mem*/false,/*<u></u>strict*/true);<br>
for (l=0;l<L-1;l++){<br>
A.reshape(pow(d,L-2-l),pow(d,<u></u>2),pow(d,l));<br>
result.reshape(pow(d,L-l),pow(<u></u>d,l));<br>
for (slice=0;slice<A.n_slices;<u></u>slice++){<br>
result.col(slice) += vectorise(A.slice(slice)*hint)<u></u>;<br>
}<br>
}<br>
arrayin = A.memptr();<br>
ierr = VecRestoreArray(x,&arrayin); CHKERRQ(ierr);<br>
arrayout = result.memptr();<br>
ierr = VecRestoreArray(y,&arrayout); CHKERRQ(ierr);<br>
PetscFunctionReturn(0);<br>
}<br></div></div>
#undef __FUNCT__/*<br>
*<br>
</blockquote>
<br>
</blockquote></div><br></div>
</div></blockquote></div><br></div></body></html>