[MPICH] implementing a private work queue in each process with job-stealing and a pthread listener to listen job-stealing requests from other queues

[Dominik Margraf]

I am trying to implemenrt a task queue system where each process has
its own task queue.  In main(), each process empties its own task
queue first, then it tries to ask other processes for jobs.

After MPI is initialized and just before emptying its own task queue,
a listener pthread is started to listen job requests from others.

The job-stealing requests are in main() but the request-receiving call
and job sending call are in the pthread.  I tried to program but it
hangs as soons as the program starts asking for jobs.  Then I suspect
there mayt be some problems and I wrote a simpler toy example, which
the master sends a message to all processes (including itself) and the
listener thread of all processes invokes MPI_Recv() to receive
requests and report it.

It works when the master process try to send it itself.  However it
hangs when it tries to send to another process.  What's wrong with my
algorithm?  I have attached the code of the toy example.

Thanks!

Dominik


#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <pthread.h>

#define MASTER 0

int nprocs;
int proc_id;
int shard_data;

void *listener(void *arg) {
 int dummy;
 MPI_Status status;

 MPI_Recv(&dummy, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
MPI_COMM_WORLD,
          &status);

 printf("received message from %d\n", status.MPI_SOURCE);
 return NULL;
}

int main(int argc, char **argv) {
 int i;
 int dummy = 0;
 void *exit_status;
 pthread_t thread_ID;

 MPI_Init(&argc, &argv);
 MPI_Comm_rank(MPI_COMM_WORLD, &proc_id);
 MPI_Comm_size(MPI_COMM_WORLD, &nprocs);

 pthread_create(&thread_ID, NULL, listener, NULL);

 if (MASTER == proc_id) {
   for (i = 0; i < nprocs; i++) {
     MPI_Send(&dummy, 1, MPI_INT, i, 1, MPI_COMM_WORLD);
   }
 }

 pthread_join(thread_ID, exit_status);
 MPI_Finalize();
 return EXIT_SUCCESS;
}