Newsgroups: comp.parallel.mpi
From: Klaus-Dieter Oertel <oertel@zib-berlin.de>
Subject: Re: Problem with Isend/Irecv
Organization: Konrad-Zuse-Zentrum fr Informationstechnik Berlin (ZIB)
Date: 9 Feb 1996 14:22:26 GMT
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <4fflb2$36e@newsserv.zib-berlin.de>

markusd@cs.tu-berlin.de (Markus Dahm) wrote:

>The question simply is, how can I make the attached example run
>correctly ?
>
>Test() sends two number to its right neigbour, i.e. to proc 0 if
>it's running on the last processor. So each proc sends and receives
>two numbers each. After the loop everyone calls MPI_Waitany to wait for
>the data and unpacks it. There is a very similar situation whithin 
>my application so I ran into the following problem.
>
>If you call test() just once, everything works fine, but within
>the second call it stalls in MPI_Waitany, i.e. that call doesn't
>return. I figured that it probably has something to do with
>the MPI_Requests and experimented a lot with MPI_Wait, MPI_Request_free, etc..
>When I used uniq message tags at each invokation of test() it worked, 
>but that's not what I desire. I want to be able to "reuse" message tags.

>#include <mpi.h>
>#include <stdio.h>
>
>#define MSG 4711
>
>int  pid, nprocs;
>
>void test()
>{
>  int from, to, i, tag, index;
>  int buf[] = {3,1,4,1,5,9,2,7};
>  int recv_buf[2];
>
>  MPI_Request request, requests[2];
>  MPI_Status status;
>
>  from =  ((pid) + nprocs - 1) % nprocs;
>  to   =  ((pid) + 1) % nprocs;
>
>  /*  printf("[%d] from = %d, to = %d\n", pid, from, to); fflush(stdout);*/
>
>  for(i=0; i < 2; i++) {
>
>    /* Receive message from the left neighbour */
>    tag = MSG + (from + 1) * (i + 1);
>    printf("[%d] recvtag = (%d, %d)\n", pid, tag, from);fflush(stdout);
>    MPI_Irecv(&recv_buf[i]     /* buf      */,
>	      1              /* count    */, 
>	      MPI_INT       /* datatype */,
>	      from             /* source   */,
>	      tag              /* tag      */,
>	      MPI_COMM_WORLD      /* comm     */,
>	      &requests[i] /* request  */);
>
>    /* Send message to the right neighbour */
>    tag = MSG + (pid + 1) * (i + 1);
>    printf("[%d] sendtag = (%d, %d)\n", pid, tag, to);fflush(stdout);
>
>    MPI_Isend(&buf[(pid * 2) + i] /* buf      */,
>	      1           /* count    */, 
>	      MPI_INT     /* datatype */,
>	      to          /* dest     */,
>	      tag         /* tag      */,
>	      MPI_COMM_WORLD /* comm     */,
>	      &request /* request  */);
>
>    /*    MPI_Wait(&request, &status);*/
>  }
>
>  for(i=0; i < 2; i++) {
>    printf("[%d] Before MPI_Waitany\n", pid);fflush(stdout);
>    MPI_Waitany(2, requests, &index, &status);
>    printf("[%d] After MPI_Waitany\n", pid);fflush(stdout);
>    tag  =  MSG + (from + 1) * (index + 1);
>
>    MPI_Irecv(&recv_buf[index]     /* buf      */,
>	      1              /* count    */, 
>	      MPI_INT       /* datatype */,
>	      from             /* source   */,
>	      tag              /* tag      */,
>	      MPI_COMM_WORLD      /* comm     */,
>	      &requests[index] /* request  */);
>    /*    MPI_Waitany(2, &send_requests, &index, &status);*/
>  }
>
>  printf("[%d] {%d, %d}\n", pid, recv_buf[0], recv_buf[1]);fflush(stdout);
>}
>
>void main(int argc, char *argv[])
>{
>  MPI_Init(&argc, &argv);
>  MPI_Comm_rank(MPI_COMM_WORLD, &pid);
>  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
>
>  test();
>  puts("----------");fflush(stdout);
>  test();
>  MPI_Finalize();
>}

The reason for the problem can be seen in the following picture
which shows the situation on 2 processes:

	PROC 0                          PROC 1

First call of test()
    first loop
        MPI_Irecv(proc1,tag1)           MPI_Irecv(proc0,tag2)
        MPI_Isend(proc1,tag2)           MPI_Isend(proc0,tag1)

        MPI_Irecv(proc1,tag3)           MPI_Irecv(proc0,tag4)
        MPI_Isend(proc1,tag4)           MPI_Isend(proc0,tag3)

    second loop (order of tags depends on messages received)
        MPI_Waitany                     MPI_Waitany
        MPI_Irecv(proc1,tag1)<-       ->MPI_Irecv(proc0,tag2)
                               \     /
        MPI_Waitany             \   /   MPI_Waitany
        MPI_Irecv(proc1,tag3)    \ /    MPI_Irecv(proc0,tag4)
                                  X
Second call of test()            / \
    first loop                  /   \
        MPI_Irecv(proc1,tag1)  /     \  MPI_Irecv(proc0,tag2)
        MPI_Isend(proc1,tag2)--       --MPI_Isend(proc0,tag1)

        MPI_Irecv(proc1,tag3)           MPI_Irecv(proc0,tag4)
        MPI_Isend(proc1,tag4)           MPI_Isend(proc0,tag3)

    second loop (order of tags depends on messages received)
        MPI_Waitany                     MPI_Waitany
        MPI_Irecv(proc1,tag1)           MPI_Irecv(proc0,tag2)

        MPI_Waitany                     MPI_Waitany
        MPI_Irecv(proc1,tag3)           MPI_Irecv(proc0,tag4)

The recv-tags from the first loop are reused in the second loop of test().
These Irecvs match the messages send in the second call of test().

Regards
Klaus-Dieter Oertel


