Let us now take a deeper dive into the send and receive operation and study some important properties. And these are message ordering and deadlock. So for message ordering, let me consider a simple example with four ranks R0, R1, R2, R3. And we'll think of time going in the downwards direction. And let's say that R0, wants to receive a message, A from R1, so R1 will need to send it. So, send A to R0 and R2 wants to send B to R3, which means that R3 wants to receive B from R2. So you have these two messages going on. The question is what is the ordering? Suppose we know for a fact that R2 sent its message to R3 after R1 sent its message to R0. Is there a guarantee of fairness that R0 must receive its message before R3? And the answer is no. We are talking about a big distributed parallell computer. And to allow things to operate in parallel, we cannot have any of these global fairness or ordering guarantees. So it's quite possible that the message over here, B from R2, comes first. And the message A from R1 is received later, because they are independent senders and receivers. The message ordering is only guaranteed if the sender is the same for the two messages. The receiver is the same for the two messages. The data type is the same for the two messages. And the tag is the same for two messages. So same order for two messages let's say. So in this example we had different senders and receivers, so there's no order guaranteed for those messages. Whereas, if you do have these properties satisfied, the same sender, the same receiver, the same type, and the same tag, and there are multiple messages flowing between the same sender and the same receiver with these properties, then they will occur in order. So that's very helpful to know when you are writing distributed memory parallel programs with message passing. Now, send and receive operations can lead to a very interesting challenge called deadlock. And this is how it might arise. Suppose we have two ranks, R0 and R1. And R0 wants to send X to R1. And R1 wants to send Y to R0 and then, R0 wants to receive Y from R1. And R1 wants to receive X from R0. Now this seems like a very reasonable message passing protocol, where a pair of ranks just wants to exchange data with each other. But using the MPI primitives, the send and receive operations we've learned so far, we're going to have a major problem. Send and receive are what are known as blocking operations. Meaning that when you perform a send, that rank process just waits and waits until a matching receive has been detected. So over here what happens is R0 is blocked waiting on the send and so is R1. And neither can proceed because the matching receives only come after the sends. This is what's called a deadlock. There's a cycle of waiting, and we've seen examples of deadlocks before, in parallel and concurrent programming. Now we are seeing it with message passing. So the two ways to fix it are, one approach is to pay, is to pay very careful attention to the ordering of the send and receives. And in fact in R1, if we exchange these two, if R0 did the send to R1. And R1 did a receive from R0. And then R0 did a receive from R1 and R1 did a send to R0. And we could achieve this, as I mentioned, by just interchanging these two statements in R1. Now we do not have a deadlock, because what will happen is the send to R1 from rank R0 will match the receive from R0 and rank R1. And the communication will occur and each will move on to the next operation, which in the case of R0, is a receive and in the case of R1 is a send. So these blocking send and receive operations can lead to deadlock very easily. Because if you ordered your messages in different ranks in a way that they do not match, it immediately implies a deadlock, if you are using blocking send receive operations. However, if you ensure they do match, then you can avoid the deadlock. Another approach that works in this case, is that MPI has an API that combines send and receive. So you could do a send, receive essentially, by concatenating all of the parameters of the send API call with all the parameters of the receive in the two cases. And the MPI runtime will ensure that deadlock is avoided because now the send-receive pair is matched rather than matching a single send with a single receive. So this allows a two-way exchange in one operation. So we have seen that send and receive are very simple operations at first blush. But they have interesting properties. There's the message ordering property where two messages are only ordered under certain circumstances when they have the same sender, receiver, type, and tag. And there's the blocking nature where a call to send or receive just blocks and waits till the matching call is encountered. And if that does not occur, it can wait indefinitely which is a deadlock. And we've also learned some approaches to addressing deadlock and we'll learn more approaches to avoiding deadlocks in some of the future lessons.