[MPICH] implementing a private work queue in each process with job-stealing and a pthread listener to listen job-stealing requests from other queues

[Rajeev Thakur]

If you want to use multithreading with MPI, you need to call MPI_Init_thread
instead of MPI_Init. You need to specify in MPI_Init_thread the right level
of threading required (in this case it is MPI_THREAD_MULITPLE) and check the
"provided" value to see if the implementation supports that level. You might
want to read the threads specification in the MPI-2 standard because there
are some other restrictions on what users can do. A concise description of
the threads spec can also be found in Sec 2 of this paper:
www.mcs.anl.gov/~thakur/papers/mpi-threads.pdf .

Rajeev


> -----Original Message-----
> From: owner-mpich-discuss at mcs.anl.gov 
> [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Dominik Margraf
> Sent: Friday, March 30, 2007 7:04 PM
> To: mpich-discuss at mcs.anl.gov
> Subject: [MPICH] implementing a private work queue in each 
> process with job-stealing and a pthread listener to listen 
> job-stealing requests from other queues
> 
> I am trying to implemenrt a task queue system where each process has
> its own task queue.  In main(), each process empties its own task
> queue first, then it tries to ask other processes for jobs.
> 
> After MPI is initialized and just before emptying its own task queue,
> a listener pthread is started to listen job requests from others.
> 
> The job-stealing requests are in main() but the request-receiving call
> and job sending call are in the pthread.  I tried to program but it
> hangs as soons as the program starts asking for jobs.  Then I suspect
> there mayt be some problems and I wrote a simpler toy example, which
> the master sends a message to all processes (including itself) and the
> listener thread of all processes invokes MPI_Recv() to receive
> requests and report it.
> 
> It works when the master process try to send it itself.  However it
> hangs when it tries to send to another process.  What's wrong with my
> algorithm?  I have attached the code of the toy example.
> 
> Thanks!
> 
> Dominik
> 
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <mpi.h>
> #include <pthread.h>
> 
> #define MASTER 0
> 
> int nprocs;
> int proc_id;
> int shard_data;
> 
> void *listener(void *arg) {
>  int dummy;
>  MPI_Status status;
> 
>  MPI_Recv(&dummy, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
> MPI_COMM_WORLD,
>           &status);
> 
>  printf("received message from %d\n", status.MPI_SOURCE);
>  return NULL;
> }
> 
> int main(int argc, char **argv) {
>  int i;
>  int dummy = 0;
>  void *exit_status;
>  pthread_t thread_ID;
> 
>  MPI_Init(&argc, &argv);
>  MPI_Comm_rank(MPI_COMM_WORLD, &proc_id);
>  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
> 
>  pthread_create(&thread_ID, NULL, listener, NULL);
> 
>  if (MASTER == proc_id) {
>    for (i = 0; i < nprocs; i++) {
>      MPI_Send(&dummy, 1, MPI_INT, i, 1, MPI_COMM_WORLD);
>    }
>  }
> 
>  pthread_join(thread_ID, exit_status);
>  MPI_Finalize();
>  return EXIT_SUCCESS;
> }
> 
>