[petsc-users] Vec, Mat and binaryfiles.

[Jed Brown]

Sal Am <tempohoper at gmail.com> writes:

> Yes I was just hoping there'd be more than that.
>
> I have tried using one of them as basis:
>
> import PetscBinaryIO
> import numpy as np
> import scipy.sparse
>
> b_vector = np.array(np.fromfile('Vector_b.bin'
> ,dtype=np.dtype((np.float64,2))));
> A_matrix =
> np.array(np.fromfile('Matrix_A_cmplx.bin',dtype=np.dtype((np.float64,2))));
> A_indexs = np.array(np.fromfile('Matrix_A_int.bin'
> ,dtype=np.dtype((np.int32       ,2))));
>
> b_vector = b_vector[:,0] + 1j*b_vector[:,1]
> A_matrix = A_matrix[:,0] + 1j*A_matrix[:,1]
>
> A_matrix = scipy.sparse.csr_matrix((A_matrix, (A_indexs[:,0]-1,
> A_indexs[:,1]-1)))
>
> vec = b_vector.view(PetscBinaryIO.Vec)
>
> io = PetscBinaryIO.PetscBinaryIO()
> io.writeBinaryFile('Vector_b.dat', [vec,])
> io.writeMatSciPy('Matrix_A.dat', A_matrix)

The "fh" argument to writeMatSciPy is a file handle.  You can use
writeBinaryFile with matrices, in one file or separate files.

> I was able to do it convert and read the Vector_b in fine.
> So what I am having trouble with is actually converting the
> scipy.sparse.csr_matrix into the PETSc format. I thought the writeMatSciPy
> would do that by calling writeMatSparse?
> Maybe I have misunderstood something here?
>
> All the best,
> S
>
> On Thu, Nov 8, 2018 at 10:10 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Nov 8, 2018 at 4:56 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Thanks, I missed that,
>>>
>>> I do care about scalability, the long term plan is to make it fully
>>> compatible on a cluster so I guess I would need to convert it to PETSc
>>> binary format before reading it in.
>>>
>>> Is this the file you were referring to
>>> https://github.com/erdc/petsc-dev/blob/master/bin/pythonscripts/PetscBinaryIO.py
>>>
>>> Cannot seem to find docs on how to use it, do you happen to have any
>>> examples?
>>>
>>
>> The documentation is at the top of the file. You can see it by clicking
>> the link.
>>
>>    Matt
>>
>>
>>> All the best,
>>> S
>>>
>>>
>>>
>>>
>>> On Wed, Nov 7, 2018 at 4:07 PM Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Please always use "reply-all" so that your messages go to the list.
>>>> This is standard mailing list etiquette.  It is important to preserve
>>>> threading for people who find this discussion later and so that we do
>>>> not waste our time re-answering the same questions that have already
>>>> been answered in private side-conversations.  You'll likely get an
>>>> answer faster that way too.
>>>>
>>>> Sal Am <tempohoper at gmail.com> writes:
>>>>
>>>> > Thank you Jed for the quick response!
>>>> >
>>>> >>
>>>> >> Yes, of course the formats would have to match.  I would recommend
>>>> >> writing the files in an existing format such as PETSc's binary format.
>>>> >>
>>>> >
>>>> > Unfortunately I do not think I can change the source code to output
>>>> those
>>>> > two files in PETSc format and it would probably take a very long time
>>>> > converting everything into PETSc (it is not even my code).
>>>>
>>>> File-based workflows are very often bottlenecks.  You can also use any
>>>> convenient software (e.g., Python or Matlab/Octave) to convert your
>>>> custom binary formats to PETSc binary format (see PetscBinaryIO provided
>>>> with PETSc), at which point you'll be able to read in parallel.  If you
>>>> don't care about scalability or only need to read once, then you can
>>>> write code of the type you propose.
>>>>
>>>> > Do you have any other suggestions on how to read in those two complex
>>>> > binary files? Also why would it be more difficult to parallelise as
>>>> > I thought getting the two files in PETSc vector format would allow me
>>>> to
>>>> > use the rest of PETSc library the usual way?
>>>> >
>>>> > Kind regards,
>>>> > S
>>>> >
>>>> >
>>>> > On Mon, Nov 5, 2018 at 4:49 PM Jed Brown <jed at jedbrown.org> wrote:
>>>> >
>>>> >> Sal Am via petsc-users <petsc-users at mcs.anl.gov> writes:
>>>> >>
>>>> >> > Hi,
>>>> >> >
>>>> >> > I am trying to solve a Ax=b complex system. the vector b and
>>>> "matrix" A
>>>> >> are
>>>> >> > both binary and NOT created by PETSc. So I keep getting error
>>>> messages
>>>> >> that
>>>> >> > they are not correct format when I read the files with
>>>> >> PetscViewBinaryOpen,
>>>> >> > after some digging it seems that one cannot just read a binary file
>>>> that
>>>> >> > was created by another software.
>>>> >>
>>>> >> Yes, of course the formats would have to match.  I would recommend
>>>> >> writing the files in an existing format such as PETSc's binary format.
>>>> >> While the method you describe can be made to work, it will be more
>>>> work
>>>> >> to make it parallel.
>>>> >>
>>>> >> > How would I go on to solve this problem?
>>>> >> >
>>>> >> > More info and trials:
>>>> >> >
>>>> >> > "matrix" A consists of two files, one that contains row column index
>>>> >> > numbers and one that contains the non-zero values. So what I would
>>>> have
>>>> >> to
>>>> >> > do is multiply the last term in a+b with PETSC_i to get a real +
>>>> >> imaginary
>>>> >> > vector A.
>>>> >> >
>>>> >> > vector b is in binary, so what I have done so far (not sure if it
>>>> works)
>>>> >> is:
>>>> >> >
>>>> >> > std::ifstream input("Vector_b.bin", std::ios::binary );
>>>> >> > while (input.read(reinterpret_cast<char*>(&v), sizeof(float)))
>>>> >> >      ierr    =
>>>> VecSetValues(u,1,&iglobal,&v,INSERT_VALUES);CHKERRQ(ierr);
>>>> >> >
>>>> >> > where v is a PetscScalar.
>>>> >> >
>>>> >> > Once I am able to read both matrices I think I can figure out the
>>>> solvers
>>>> >> > to solve the system.
>>>> >> >
>>>> >> > All the best,
>>>> >> > S
>>>> >>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>