The callback first seeks the communicator associated with the context identifier. A technical difficulty arises here: 1) in Open MPI, context identifiers are actually indices in an
array, and for performance reasons it is desirable to reuse lower indices as soon as the related communicator is freed, rather than increase the size of the communicator associative array; 2) MPI_COMM_FREE is not synchronizing. As a consequence, in some cases, a message may arrive after the associated communicator has been freed locally, and it is important to verify that the operation has no side effects on an unrelated communicator reusing that context identifier. In
more details, imagine that, in plan A, Pr posts a reception from Ps; meanwhile, process Pk revokes the communicator, which interrupts the reception at Pr. Now, imagine that the
delivery of the revoke notification is unusually slow at Ps, which then proceeds to send its message to Pr, unaware (yet) that it is following a revoked communication plan. As long
as Pr does not free the communicator, the normal matching logic of MPI will correctly dispatch this stall message to the revoked communicator, where it will thereafter be appropriately discarded. However, when Pr frees the communicator, the context identifier becomes available for reuse, and if Pr creates a new communicator (as an example, a duplicate of MPI_COMM_SELF, which requires no communication), then that stall message could incorrectly be delivered in the newly
created communicator. To avoid this caveat without impacting the MPI matching logic, yet still allow for the reuse of the communicator identifiers, the MPI_COMM_FREE function needs to become loosely synchronizing.