sieve-dev loading mesh in parallel

[Shi Jin]

Hi,

I have been thinking about the parallelization of the mesh loading recently.
I guess the easiest and dirtiest way to implement is to do the simulation in
two steps:
1. Run the code on a large shared memory system (with more than 64GB RAM),
only to distribute the mesh to the desired number of processes and then save
the sieve information of each process into files and then simply stop. This
is possible since it avoids the large amount of memory usage in a real
calculation such as matrices and vectors.
2. With the sieve information now stored in  files, run the same sized full
calculation on a production system (could be a distributed memory cluster),
where instead of reading the whole mesh on the root process and then
distributing it, we directly read the sieve information from the files and
construct the local mesh out of it. If this is done correctly, it should end
up with exactly the same distributed mesh as in step 1. Then matrices and
vectors can be created and production simulations will be carried out.

This method sounds very dirty but it avoids dealing with the detailed mesh
distribution and thus may be easier for me to implement. I could call the
view() function on each sieve  to dump its information to a file. What I am
using is

>     const Obj<ALE::Mesh::sieve_type>&sieve= m->getSieve();
>     //sieve->view("mesh sieve",comm);
>
>     ostringstream  filename;
>     std::ofstream fs;
>     filename<<basename<<"_sieveNP"<<size<<"Rank"<<rank<<".txt";
>     fs.open(filename.str().c_str());
>
>     double rawdata=true;
>     sieve->view(fs,filename.str().c_str(),rawdata);
>
>     fs.close();
>     MPI_Barrier(comm);
>

At this stage, I am not very sure whether the generated file will provide
sufficient data to reconstruct the same sieve. Hopefully they do. I am
trying to work out a way to construct sieves from these dumped files and
then construct the local mesh. I will report back on the progress.

At this stage, I would love to hear some insider comment on this approach.
It is definitely not the best way but I hope it is feasible since it seems
the easiest for me.
Thank you very much.

Shi
On Sun, Mar 9, 2008 at 12:18 AM, Shi Jin <jinzishuai at gmail.com> wrote:

> Hi,
>
> Thanks to your help, my CFD code using sieve is close to complete. One
> last part to do is to enable each process to load its own grid in parallel.
> Currently, I first ask the master process to load the whole mesh and call
> MeshDistribute() to distribute the smaller local mesh into each process.
> This method works fine if the mesh is small to medium in size. As the mesh
> gets very large, the memory require to store the whole mesh on a single
> process becomes impossible (this way, the memory of the master process is
> much larger than the others). Since each process only needs its own portion
> of the mesh anyway, it should just load its own. I am wondering if some help
> on this topic can be provided from the sieve-dev team. Thanks a lot.
>
> I have thought about a few possibilities:
> 1. Like the example codes, I can use tetgen to generate the mesh on the
> fly and then refine it. However, this way the domain decomposition is quite
> random which does not work with our particle simulation code where a clearly
> defined domain boundary (such as some z values) is necessary to track the
> particles.
> 2. I can load a coarse mesh that I am using right now. After calling
> MeshDistribute() to obtain the correct domain decomposition, I can then
> refine each mesh using tetgen to the required resolution. However, when I
> tried this method, the refined mesh is actually repeating the coarse
> elements many times (thus although the number of nodes and vertexes are
> larger, the mesh is the same as before).
> 3. Still based on the current code, each process will try to load the same
> input global mesh file. Instead of loading everything, each process will
> only take the data that is within the pre-defined domain and throw away the
> outside data. This way, I can completely separate the mesh generator and the
> CFD code (the mesh can be generated with some sophisticated software
> separately). But it is important to keep the data structure across process
> consistent, i.e., the numbering  and the ghost points must make sense. I
> think this is exactly what the MeshDistribute() code does and it is the
> perfect solution. However, to do this, one may need a deeper understanding
> of the sieve library.
>
> I think the third possibility is the best and if this is not already
> implemented, it is probably good to have some kind of function like it in
> the library. I am more than happy to contribute my code to this development
> and of course I need some guidance. Thank you very much.
>
> Shi
>
> --
> Sincerely,
> Shi Jin, Ph.D.
> http://www.ualberta.ca/~sjin1/ <http://www.ualberta.ca/%7Esjin1/>




-- 
Sincerely,
Shi Jin, Ph.D.
http://www.ualberta.ca/~sjin1/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mcs.anl.gov/mailman/private/sieve-dev/attachments/20080312/cbe3c405/attachment.htm>