passing data between different communicators using PETSc

[Khan, Irfan]

Hi
I have divided my processes into fluid compute ranks (n) and solid compute ranks (m), where n > m. For most of the part, the fluid compute ranks communicate among themselves and so do the solid compute nodes. However twice during each timestep the some data is transfered between the fluid and solid compute nodes. 

For instance, the fluid compute nodes generate fluid force on the solid body that needs to be transmitted every time step and the solid compute node calculate the resulting displacement from the forces and transmit back the displacement information to the fluid nodes. The force and displacement vectors for total nodes are distributed randomly on the fluid and solid nodes. I have attached a pdf that basically shows that the distribution of vectors is random

I am currently trying to use MPI_Allgatherv to accumulate all the data from send ranks to receive ranks, but I am sure this will be very costly for large data distributed over many ranks. Is there an efficient way to do this using PETSc?

Please let me know if you need more information or better explanation.

Thanks
Irfan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LBM-FEM-communicationNodes.pdf
Type: application/pdf
Size: 186076 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090309/448ef659/attachment-0001.pdf>