[petsc-users] Vec, Mat and binaryfiles.

[Sal Am]

For anyone in the future, to read in binary files written using raw C++
write and to convert to PETSc format using the PetscBinaryIO.py script:
"""
Ax = b
Where mat A is divided into A index and A non-zero elements.
"""
import PetscBinaryIO
import numpy as np
import scipy.sparse

b_vector = np.array(np.fromfile('Vector_b.bin'
,dtype=np.dtype((np.float64,2))));
A_matrix =
np.array(np.fromfile('Matrix_A_cmplx.bin',dtype=np.dtype((np.float64,2))));
# non-zero elements.
A_indexs = np.array(np.fromfile('Matrix_A_int.bin'
,dtype=np.dtype((np.int32  ,2)))); # row column indexes for the non zero
elements

b_vector = b_vector[:,0] + 1j*b_vector[:,1]
A_matrix = A_matrix[:,0] + 1j*A_matrix[:,1]

A_matrix = scipy.sparse.csr_matrix((A_matrix, (A_indexs[:,0]-1,
A_indexs[:,1]-1)))

vec = b_vector.view(PetscBinaryIO.Vec)

io = PetscBinaryIO.PetscBinaryIO()
io.writeBinaryFile('Vector_b.dat', [vec,])
io.writeBinaryFile('Matrix_A.dat',[A_matrix,])

This will output two PETSc ready files Vector_b.dat and Matrix_A.dat that
you can exploit further with Vec/MatLoad or similar.

Cheers,
S



On Thu, Nov 8, 2018 at 7:53 PM Jed Brown <jed at jedbrown.org> wrote:

> Sal Am <tempohoper at gmail.com> writes:
>
> > Yes I was just hoping there'd be more than that.
> >
> > I have tried using one of them as basis:
> >
> > import PetscBinaryIO
> > import numpy as np
> > import scipy.sparse
> >
> > b_vector = np.array(np.fromfile('Vector_b.bin'
> > ,dtype=np.dtype((np.float64,2))));
> > A_matrix =
> >
> np.array(np.fromfile('Matrix_A_cmplx.bin',dtype=np.dtype((np.float64,2))));
> > A_indexs = np.array(np.fromfile('Matrix_A_int.bin'
> > ,dtype=np.dtype((np.int32       ,2))));
> >
> > b_vector = b_vector[:,0] + 1j*b_vector[:,1]
> > A_matrix = A_matrix[:,0] + 1j*A_matrix[:,1]
> >
> > A_matrix = scipy.sparse.csr_matrix((A_matrix, (A_indexs[:,0]-1,
> > A_indexs[:,1]-1)))
> >
> > vec = b_vector.view(PetscBinaryIO.Vec)
> >
> > io = PetscBinaryIO.PetscBinaryIO()
> > io.writeBinaryFile('Vector_b.dat', [vec,])
> > io.writeMatSciPy('Matrix_A.dat', A_matrix)
>
> The "fh" argument to writeMatSciPy is a file handle.  You can use
> writeBinaryFile with matrices, in one file or separate files.
>
> > I was able to do it convert and read the Vector_b in fine.
> > So what I am having trouble with is actually converting the
> > scipy.sparse.csr_matrix into the PETSc format. I thought the
> writeMatSciPy
> > would do that by calling writeMatSparse?
> > Maybe I have misunderstood something here?
> >
> > All the best,
> > S
> >
> > On Thu, Nov 8, 2018 at 10:10 AM Matthew Knepley <knepley at gmail.com>
> wrote:
> >
> >> On Thu, Nov 8, 2018 at 4:56 AM Sal Am via petsc-users <
> >> petsc-users at mcs.anl.gov> wrote:
> >>
> >>> Thanks, I missed that,
> >>>
> >>> I do care about scalability, the long term plan is to make it fully
> >>> compatible on a cluster so I guess I would need to convert it to PETSc
> >>> binary format before reading it in.
> >>>
> >>> Is this the file you were referring to
> >>>
> https://github.com/erdc/petsc-dev/blob/master/bin/pythonscripts/PetscBinaryIO.py
> >>>
> >>> Cannot seem to find docs on how to use it, do you happen to have any
> >>> examples?
> >>>
> >>
> >> The documentation is at the top of the file. You can see it by clicking
> >> the link.
> >>
> >>    Matt
> >>
> >>
> >>> All the best,
> >>> S
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Nov 7, 2018 at 4:07 PM Jed Brown <jed at jedbrown.org> wrote:
> >>>
> >>>> Please always use "reply-all" so that your messages go to the list.
> >>>> This is standard mailing list etiquette.  It is important to preserve
> >>>> threading for people who find this discussion later and so that we do
> >>>> not waste our time re-answering the same questions that have already
> >>>> been answered in private side-conversations.  You'll likely get an
> >>>> answer faster that way too.
> >>>>
> >>>> Sal Am <tempohoper at gmail.com> writes:
> >>>>
> >>>> > Thank you Jed for the quick response!
> >>>> >
> >>>> >>
> >>>> >> Yes, of course the formats would have to match.  I would recommend
> >>>> >> writing the files in an existing format such as PETSc's binary
> format.
> >>>> >>
> >>>> >
> >>>> > Unfortunately I do not think I can change the source code to output
> >>>> those
> >>>> > two files in PETSc format and it would probably take a very long
> time
> >>>> > converting everything into PETSc (it is not even my code).
> >>>>
> >>>> File-based workflows are very often bottlenecks.  You can also use any
> >>>> convenient software (e.g., Python or Matlab/Octave) to convert your
> >>>> custom binary formats to PETSc binary format (see PetscBinaryIO
> provided
> >>>> with PETSc), at which point you'll be able to read in parallel.  If
> you
> >>>> don't care about scalability or only need to read once, then you can
> >>>> write code of the type you propose.
> >>>>
> >>>> > Do you have any other suggestions on how to read in those two
> complex
> >>>> > binary files? Also why would it be more difficult to parallelise as
> >>>> > I thought getting the two files in PETSc vector format would allow
> me
> >>>> to
> >>>> > use the rest of PETSc library the usual way?
> >>>> >
> >>>> > Kind regards,
> >>>> > S
> >>>> >
> >>>> >
> >>>> > On Mon, Nov 5, 2018 at 4:49 PM Jed Brown <jed at jedbrown.org> wrote:
> >>>> >
> >>>> >> Sal Am via petsc-users <petsc-users at mcs.anl.gov> writes:
> >>>> >>
> >>>> >> > Hi,
> >>>> >> >
> >>>> >> > I am trying to solve a Ax=b complex system. the vector b and
> >>>> "matrix" A
> >>>> >> are
> >>>> >> > both binary and NOT created by PETSc. So I keep getting error
> >>>> messages
> >>>> >> that
> >>>> >> > they are not correct format when I read the files with
> >>>> >> PetscViewBinaryOpen,
> >>>> >> > after some digging it seems that one cannot just read a binary
> file
> >>>> that
> >>>> >> > was created by another software.
> >>>> >>
> >>>> >> Yes, of course the formats would have to match.  I would recommend
> >>>> >> writing the files in an existing format such as PETSc's binary
> format.
> >>>> >> While the method you describe can be made to work, it will be more
> >>>> work
> >>>> >> to make it parallel.
> >>>> >>
> >>>> >> > How would I go on to solve this problem?
> >>>> >> >
> >>>> >> > More info and trials:
> >>>> >> >
> >>>> >> > "matrix" A consists of two files, one that contains row column
> index
> >>>> >> > numbers and one that contains the non-zero values. So what I
> would
> >>>> have
> >>>> >> to
> >>>> >> > do is multiply the last term in a+b with PETSC_i to get a real +
> >>>> >> imaginary
> >>>> >> > vector A.
> >>>> >> >
> >>>> >> > vector b is in binary, so what I have done so far (not sure if it
> >>>> works)
> >>>> >> is:
> >>>> >> >
> >>>> >> > std::ifstream input("Vector_b.bin", std::ios::binary );
> >>>> >> > while (input.read(reinterpret_cast<char*>(&v), sizeof(float)))
> >>>> >> >      ierr    =
> >>>> VecSetValues(u,1,&iglobal,&v,INSERT_VALUES);CHKERRQ(ierr);
> >>>> >> >
> >>>> >> > where v is a PetscScalar.
> >>>> >> >
> >>>> >> > Once I am able to read both matrices I think I can figure out the
> >>>> solvers
> >>>> >> > to solve the system.
> >>>> >> >
> >>>> >> > All the best,
> >>>> >> > S
> >>>> >>
> >>>>
> >>>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> >> experiments is infinitely more interesting than any results to which
> their
> >> experiments lead.
> >> -- Norbert Wiener
> >>
> >> https://www.cse.buffalo.edu/~knepley/
> >> <http://www.cse.buffalo.edu/~knepley/>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181112/f64bb678/attachment.html>