[petsc-users] About MatGetOwnershipRange()

[Joon Hee Choi]

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "Joon Hee Choi" <choi240 at purdue.edu>
Cc: petsc-users at mcs.anl.gov
Sent: Thursday, September 12, 2013 7:51:21 PM
Subject: Re: [petsc-users] About MatGetOwnershipRange()

Joon Hee Choi <choi240 at purdue.edu> writes:

>> Hi Jed,
>>
>> Thank you for your reply. I tried to follow your method, but I think
>> my code still has something wrong because "localsize" is the same as
>> global row size("I"). 

>Did you run in parallel?

Yes. I tested my code with 2 nodes and 4 ppn(processors per node).

>> Could you let me know what I am missing? My new code is as follows:
>>
>> PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J)
>> {
>>    PetscInt i, j;
>>    PetscScalar val;
>>
>>    MatCreate(PETSC_COMM_WORLD, X);
>>    MatSetType(*X, MATMPIAIJ);
>>    MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J);
>>
>>    PetscInt begin, end, localsize=PETSC_DECIDE;
>>    PetscSplitOwnership(PETSC_COMM_WORLD, &localsize, &I);
>>    MPI_Scan(&localsize, &end, 1, MPI_INT, MPI_SUM, PETSC_COMM_WORLD);
>>    begin = end - localsize;
>>    PetscPrintf(PETSC_COMM_WORLD, "Local Size: %d, begin: %d, end: %d \n", localsize, begin, end);
>>
>>    ...
>> }
>>
>> Also, tuples are the total pairs of (row dimension, column dimension,
>> element), values of sparse matrix, which are read from a file. 

>Using files this way is a massive bottleneck that you'll have to
>eliminate if you want your code to be scalable.

What do you mean a massive bottleneck? Actually, I succeeded in setting up the large matrix using seqaij type and calculating products between huge matrices. However, I need to use multiple machines because I have to run loops of the product many times. And I am a novice of mpi.

>> The tuples are sorted and distributed. 

>When you distribute, are you sure that each process really gets the
>entire row, or would it be possible to cut in the middle of a row?

I think each process gets the entire row. The following is the code for getting tuples from text file:

   ierr = PetscFOpen(PETSC_COMM_WORLD, inputFile, "r", &file); CHKERRQ(ierr);
   while (fscanf(file, "%d %d %d", &ii, &jj, &vv) == 3)
   {
      tups.push_back(std::tr1::make_tuple (ii-1, jj-1, vv));
      if (ii > I) I = ii;
      if (jj > J) J = jj;
   }
   ierr = PetscFClose(PETSC_COMM_WORLD, file); CHKERRQ(ierr);