Thank you very much for your reply.<br><br>Unfortunately, I do not have access to the cluster at this moment because there is some problem with it.<br><br>But I made a seperate program that reads a matrix of size 2500x2500 from a file and calculates eigenvalues.The matrix itself is sparse, symmetric and has non-zero values randomly distributed. I read more about the krylovschur and other solvers, and parallel computing and mpi, and I probably understand what you mean. The eigensolvers are efficient for calculating eigenvalues at the peripheries, and especially quickly (less number of iterations) on the larger side than the smaller side, right ? Although I still dont understand what Maximum Projected Dimension exactly means. I played with mpd values nonetheless, and it doesnt seem to have much effect on memory usage. I am running it on my laptop.<br>
<br>But the main problem seems to be that in the program, different threads dont seem to 'cooperate'. Every thread is calculating its own solution and storing its own matrix, as if running its own version of EPSSolve. This is because in the output to top command, each process (I used 2) uses nearly equal amount of memory, which is equal to the memory usage if I run the program with only 1 process. (so I suppose memory distribution is not happening ?) I am using MPIAIJ now. I am putting the code below:<br>
(I understand why the program prints things multiple times, but I know I can fix that by something like "if(processID=0) printSomething"....what I am not able to understand is how to distribute memory while solving for eigenvalues, which is the call to EPSSolve. There doesnt seem to be any parameter to tell it to solve while cooperating with other threads)<br>
<br><i><span style="color: rgb(102, 102, 102);">#include "slepceps.h"</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">#include <stdio.h></span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">#include <time.h></span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">#include <math.h></span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">#define TOL 0.000001</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">#define MAXIT 1000</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);">int main(int argc,char**argv)</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">{</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> FILE *fp;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> double entry;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> int N;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> int nzval=0,Istart,Iend;</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> Mat A; </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> PetscErrorCode ierr;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> double *value,kr,ki;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> int i,j,nconv,nev,maxit,its;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> int col_nz;/*number of non-zero columns in a row*/</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> Vec xr, xi;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> double tol,error,re,im;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> EPS eps; /*Eigen solver context*/</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> const EPSType type;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> int ncv,mpd;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> /*PetscInt rows;*/</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> int* columns;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> SlepcInitialize(&argc,&argv,(char*)0,PETSC_NULL);</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> /*Read matrix from file*/</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> fp=fopen("sparseMatrix.txt","r");</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> fscanf(fp,"%d",&N);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> fscanf(fp,"%lf",&entry);/*not required, but only to move file pointer forward*/</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> /*Allocate memory to arrays*/</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> value=(double*)malloc(N*sizeof(double));</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> columns=(int*)malloc(N*sizeof(int)); </span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr=MatSetType(A,MATMPIAIJ);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> //ierr=MatSeqAIJSetPreallocation(A,N,PETSC_NULL);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr=MatMPIAIJSetPreallocation(A,N,PETSC_NULL,N,PETSC_NULL);CHKERRQ(ierr);//?????!!!!!!!!!!!!!!!</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr); </span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> for(i=0;i<N;i++)</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> {</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> col_nz=0;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> if(i<Iend && i>=Istart) //important, see where interval closed and open</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> {</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> for(j=0;j<N;j++)</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> {</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> fscanf(fp,"%lf",&entry);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> if(entry!=0)</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> {</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> value[col_nz]=entry;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> columns[col_nz]=j;</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> col_nz++;</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> }</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> } </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = MatSetValues(A,1,&i,col_nz,columns,value,INSERT_VALUES);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> }</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> }</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> fclose(fp);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> printf("MATRIX ASSMEBLY DONE !!!!!!!! \n\n");</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> /*Solve the Eigen Problem*/</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> /***********START TIME************/</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> clock_t start=clock();</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> /********************************/</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = EPSSetOperators(eps,A,PETSC_NULL);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr=EPSSetWhichEigenpairs(eps,EPS_LARGEST_REAL);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr=EPSSetDimensions(eps,0.5*N,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);/*N specifies that find ALL eigenvalues*/</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = EPSSetTolerances(eps,TOL,500);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> //ierr = MatDestroy(A);CHKERRQ(ierr); //dont need this anymore</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSolve(eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> printf("Solved for half the largest values\n\n\n\n");</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %d\n\n",nconv);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = EPSDestroy(eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSetOperators(eps,A,PETSC_NULL);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr=EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr=EPSSetDimensions(eps,0.5*N,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);/*N specifies that find ALL eigenvalues*/</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = EPSSetTolerances(eps,TOL,500);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSSolve(eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> ierr = PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %d\n\n",nconv);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = EPSDestroy(eps);CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> /**********STOP TIME**************/</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> clock_t end=clock();</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> /********************************/</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">ierr = PetscPrintf(PETSC_COMM_WORLD,"Time taken to compute all eigenvalues of a %d x %d matrix is : %lf seconds\n\n",N,N,(double)(end-start)/CLOCKS_PER_SEC );CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<span style="color: rgb(102, 102, 102);"> </span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> ierr = SlepcFinalize();CHKERRQ(ierr);</span><br style="color: rgb(102, 102, 102);">
<br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);"> return 0;</span><br style="color: rgb(102, 102, 102);"><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">}<br>
<br></span></i><span style="color: rgb(102, 102, 102);"><font color="#000000">I</font></span><i><span style="color: rgb(102, 102, 102);"><font color="#000000"> </font></span></i><span style="color: rgb(102, 102, 102);"><font color="#000000">ran it with:<br>
<br>mpirun -np 2 ./slepcEigenMPI -eps_monitor<br><br>I didnt do exactly what you said, because the matrix generation part in the actual program is quite time consuming itself. But I assume what I am doing is equivalent to what you meant to do? Also, I put MPD as PETSC_DECIDE, because I didnt know what to put it for this matrix dimension.<br>
<br>This is the output I get: (part of the output)<br><i style="color: rgb(102, 102, 102);">MATRIX ASSMEBLY DONE !!!!!!!! <br><br>MATRIX ASSMEBLY DONE !!!!!!!! <br><br> 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05)<br>
1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05)<br> 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 (2.49532175e-04)<br> 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 (2.49532175e-04)<br>
3 EPS nconv=386 first unconverged value (error) 2.81217e-25 (6.07438011e-04)<br> 3 EPS nconv=386 first unconverged value (error) 2.81217e-25 (6.07438011e-04)<br> 4 EPS nconv=420 first unconverged value (error) 2.75201e-27 (1.65345126e-03)<br>
4 EPS nconv=420 first unconverged value (error) 2.75201e-27 (1.65345126e-03)<br> 6 EPS nconv=1001 first unconverged value (error) 1.93852e-27 (8.40346506e-02)<br> 6 EPS nconv=1001 first unconverged value (error) 1.93852e-27 (8.40346506e-02)<br>
7 EPS nconv=1027 first unconverged value (error) 2.00333e-27 (7.40100945e-03)<br> 7 EPS nconv=1027 first unconverged value (error) 2.00333e-27 (7.40100945e-03)</i><br style="color: rgb(102, 102, 102);"><br>How do I make "different threads store the matrix and more importantly, run EPSSolve together" ? Thats what I want right ?<br>
<br><br><br>Thank you very much !<br><br>Shitij<br><br><br><br style="color: rgb(102, 102, 102);"></font></span><i><span style="color: rgb(102, 102, 102);"></span></i><br><br><br>On 4 August 2011 16:57, Jose E. Roman <span dir="ltr"><<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>></span> wrote:<br>
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
El 04/08/2011, a las 08:48, Shitij Bhargava escribió:<br>
<div><br>
> Thank you very much for your reply.<br>
><br>
> I am a summer trainee, and I talked to my guide about this, and now what I want might be a little different.<br>
><br>
> I really do want all the eigenvalues. There cannot be any compromise on that. But please note that time is not much of a concern here, the much bigger concern is of memory. Even if I can distribute the memory somehow for the computation of all eigenvalues, that is fine. (doesnt matter if it takes a huge amount of time, he is willing to wait). Can any slepc Eigensolver distribute memory this way while solving for all eigenvalues ?<br>
><br>
> I am sorry if what I am asking is obvious, I dont know much about distributed memory, etc. also. I am stupid at this moment, but I am learning.<br>
><br>
> Also, when I ran the slepc program on a small cluster, and then ran the top command, I saw that while solving for eigenvalues, the program was already using upto 500% CPU. Does this mean that the eigensolver was "distributing memory" automatically among different nodes ? I understand that it must be distributing computational effort, but how do I know if it was distributing memory also ? It simply showed me the RES memory usage to be 1.3 gb, and the %MEM to be about 17%. Also, I did not use any MPIXXX data structures. I simply used SeqAIJ. Will using MPIAIJ "distribute memory" while solving for all eigenvalues ? (I understand that it will distribute memory to store the matrix, but the memory bottleneck is eigenvalue solver, so its more important for memory to be distributed then ) My guide said that one node will not have enough memory, but all nodes combined will, and hence the requirement of memory to be distributed.<br>
><br>
> I read about ScaLAPACK, and I have already asked on their forums to confirm if it is suitable for me. I am waiting for an answer.<br>
><br>
> Thank you once again.<br>
><br>
> Shitij<br>
<br>
</div>Try something like this:<br>
<br>
$ mpirun -np 8 ./program -eps_nev 4800 -eps_mpd 400 -eps_largest_real<br>
$ mpirun -np 8 ./program -eps_nev 4800 -eps_mpd 400 -eps_smallest_real<br>
<br>
But this may not work for the smallest ones.<br>
<font color="#888888"><br>
Jose<br>
<br>
</font></blockquote></div><br>