The Mann-Whitney test doesn't really compare medians

You'll sometimes read that the Mann-Whitney test compares the medians of two groups. But this is not exactly true, as this example demonstrates.

The graph shows each value obtained from control and treated subjects. The two-tail P value from the Mann-Whitney test is 0.0288, so you conclude that there is a statistically significant difference between the groups. But the two medians, shown by the horizontal lines, are identical. The Mann-Whitney test compared the distributions of ranks, which is quite different in the two groups.

It is not correct, however, to say that the Mann-Whitney test asks whether the two groups come from populations with different distributions. The two groups in the graph below clearly come from different distributions, but the P value from the Mann-Whitney test is high (0.46).

The Mann-Whitney test compares sums of ranks -- it does not compare medians and does not compare distributions. To interpret the test as being a comparison of medians, you have to make an additional assumption -- that the distributions of the two populations have the same shape, even if they are shifted (have different medians). With this assumption, if you reject the Mann-Whitney test reports a small P value, you can conclude that the medians are different.