He raises his hand as if holding a card. When I saw this on the TV news, the presenter queried the colour of the imaginary card, but of course it must have been yellow because the player remained in the pitch.
Suppose the player had gone off, would their have been a doubt whether it was a second yellow or a straight red? No, because in the case of a second yellow the ref would have raised his hand twice: one for the second yellow and once for the consequent red.
So it seems the actual card is not necessary? There are a number of functions it serves.
1 First, it increases the signal-to-noise ratio. It is much easier to see the brightly coloured card than notice whether the ref has deliberately raised his hand miming the holding of card - though for the purpose the card colour doesn't matter, fluorescent pink might to the job better.
2 The actual colour of the card provides redundancy (in the technical, information-theoretic sense). This is perhaps important in the communication with the player, especially since the ref and player might not speak the same language (though 'you're off!' wouldn't be difficult to understand), and for the spectator might be most important with the 'two yellows or straight red' distinction.
3 In the language of semiotics, the emotive function of the card also needs to be considered. Actually seeing the card colour would stir up feelings. The appearance of the coloured card would immediately link in to all the associations from past experiences.
One final thought, the action of the referee in miming the florishing of a card worked because we have seen refs do that with real cards in the past. Isn't this the way signs develop? Over time a 'real action' becomes replaced by a stylised sign?