However, Revoke notification messages are not posted under the control of the user, and therefore they are not completed before MPI_COMM_FREE. Thus, it is still possible that
some continue to be delivered after the loosely synchronizing MPI_COMM_FREE has completed. In order to discriminate between different communicators using the same index, the Revoke message compounds the index with the epoch number, representing how many times this index has been allocated. This compound key is then used to perform the communicator lookup (in the case of Revoke messages only, normal MPI messages still employ the normal MPI matching with
context identifiers only). If a communicator does not exist anymore (the message epoch is lower than the index epoch), the Revoke message is dropped; this is safe, as when the communicator doesn’t exist anymore, the loosely synchronized MPI_COMM_FREE guarantees that it has been freed at every other process too. When the communicator with the correct epoch exists, there are two cases; 1) the communicator had already been revoked, then the callback drops the message and returns; 2) the communicator is not yet revoked, then it is revoked immediately and the Revoke message is broadcast to all neighbors. When a communicator is revoked for the first time, the
list of pending MPI requests is traversed to mark all requests on that communicator as completed in error. Their status is set to the special error code MPIX_ERR_REVOKED, pending RDMA operations are cancelled, and the memory registrations are withdrawn. In addition, the unexpected and matching queues of the communicator are also traversed to discard incoming message fragments.