Newsgroups: comp.parallel.mpi
From: Klaus-Dieter Oertel <oertel@zib-berlin.de>
Subject: Re: Problem with Isend/Irecv
Organization: Konrad-Zuse-Zentrum fr Informationstechnik Berlin (ZIB)
Date: 9 Feb 1996 14:31:33 GMT
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <4ffls5$36e@newsserv.zib-berlin.de>

markusd@cs.tu-berlin.de (Markus Dahm) wrote:

>The question simply is, how can I make the attached example run
>correctly ?
>
>Test() sends two number to its right neigbour, i.e. to proc 0 if
>it's running on the last processor. So each proc sends and receives
>two numbers each. After the loop everyone calls MPI_Waitany to wait for
>the data and unpacks it. There is a very similar situation whithin 
>my application so I ran into the following problem.
>
>If you call test() just once, everything works fine, but within
>the second call it stalls in MPI_Waitany, i.e. that call doesn't
>return. I figured that it probably has something to do with
>the MPI_Requests and experimented a lot with MPI_Wait, MPI_Request_free, etc..
>When I used uniq message tags at each invokation of test() it worked, 
>but that's not what I desire. I want to be able to "reuse" message tags.

>#include <mpi.h>
>#include <stdio.h>
>
>#define MSG 4711
>
>int  pid, nprocs;
>
>void test()
>{
>  int from, to, i, tag, index;
>  int buf[] = {3,1,4,1,5,9,2,7};
>  int recv_buf[2];
>
>  MPI_Request request, requests[2];
>  MPI_Status status;
>
>  from =  ((pid) + nprocs - 1) % nprocs;
>  to   =  ((pid) + 1) % nprocs;
>
>  /*  printf("[%d] from = %d, to = %d\n", pid, from, to); fflush(stdout);*/
>
>  for(i=0; i < 2; i++) {
>
>    /* Receive message from the left neighbour */
>    tag = MSG + (from + 1) * (i + 1);
>    printf("[%d] recvtag = (%d, %d)\n", pid, tag, from);fflush(stdout);
>    MPI_Irecv(&recv_buf[i]     /* buf      */,
>             1              /* count    */, 
>             MPI_INT       /* datatype */,
>             from             /* source   */,
>             tag              /* tag      */,
>             MPI_COMM_WORLD      /* comm     */,
>             &requests[i] /* request  */);
>
>    /* Send message to the right neighbour */
>    tag = MSG + (pid + 1) * (i + 1);
>    printf("[%d] sendtag = (%d, %d)\n", pid, tag, to);fflush(stdout);
>
>    MPI_Isend(&buf[(pid * 2) + i] /* buf      */,
>             1           /* count    */, 
>             MPI_INT     /* datatype */,
>             to          /* dest     */,
>             tag         /* tag      */,
>             MPI_COMM_WORLD /* comm     */,
>             &request /* request  */);
>
>    /*    MPI_Wait(&request, &status);*/
>  }
>
>  for(i=0; i < 2; i++) {
>    printf("[%d] Before MPI_Waitany\n", pid);fflush(stdout);
>    MPI_Waitany(2, requests, &index, &status);
>    printf("[%d] After MPI_Waitany\n", pid);fflush(stdout);
>    tag  =  MSG + (from + 1) * (index + 1);
>
>    MPI_Irecv(&recv_buf[index]     /* buf      */,
>             1              /* count    */, 
>             MPI_INT       /* datatype */,
>             from             /* source   */,
>             tag              /* tag      */,
>             MPI_COMM_WORLD      /* comm     */,
>             &requests[index] /* request  */);
>    /*    MPI_Waitany(2, &send_requests, &index, &status);*/
>  }
>
>  printf("[%d] {%d, %d}\n", pid, recv_buf[0], recv_buf[1]);fflush(stdout);
>}
>
>void main(int argc, char *argv[])
>{
>  MPI_Init(&argc, &argv);
>  MPI_Comm_rank(MPI_COMM_WORLD, &pid);
>  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
>
>  test();
>  puts("----------");fflush(stdout);
>  test();
>  MPI_Finalize();
>}

The reason for the problem can be seen in the following picture
which shows the situation on 2 processes:

        PROC 0                          PROC 1

First call of test()
    first loop
        MPI_Irecv(proc1,tag1)           MPI_Irecv(proc0,tag2)
        MPI_Isend(proc1,tag2)           MPI_Isend(proc0,tag1)

        MPI_Irecv(proc1,tag3)           MPI_Irecv(proc0,tag4)
        MPI_Isend(proc1,tag4)           MPI_Isend(proc0,tag3)

    second loop (order of tags depends on messages received)
        MPI_Waitany                     MPI_Waitany
        MPI_Irecv(proc1,tag1)<-       ->MPI_Irecv(proc0,tag2)
                               \     /
        MPI_Waitany             \   /   MPI_Waitany
        MPI_Irecv(proc1,tag3)    \ /    MPI_Irecv(proc0,tag4)
                                  X
Second call of test()            / \
    first loop                  /   \
        MPI_Irecv(proc1,tag1)  /     \  MPI_Irecv(proc0,tag2)
        MPI_Isend(proc1,tag2)--       --MPI_Isend(proc0,tag1)

        MPI_Irecv(proc1,tag3)           MPI_Irecv(proc0,tag4)
        MPI_Isend(proc1,tag4)           MPI_Isend(proc0,tag3)

    second loop (order of tags depends on messages received)
        MPI_Waitany                     MPI_Waitany
        MPI_Irecv(proc1,tag1)           MPI_Irecv(proc0,tag2)

        MPI_Waitany                     MPI_Waitany
        MPI_Irecv(proc1,tag3)           MPI_Irecv(proc0,tag4)

The Irecv-tags from the first loop are reused in the second loop of test().
These Irecvs match the messages send in the second call of test().
Therefore nothing can be received by none of the Irecvs in the second
call of test();

Regards
Klaus-Dieter Oertel


-------------------------------------------------------------------------
 Klaus-Dieter Oertel			Konrad-Zuse-Zentrum fuer
 Phone: +49-30-89604-282		Informationstechnik Berlin (ZIB)
 Fax  : +49-30-89604-125		Heilbronner Str. 10
 Email: oertel@zib-berlin.de		D-10711 Berlin
                                	Germany
-------------------------------------------------------------------------


