Muhammad al-Maghrabi, 41, was killed as he lay on the floor following his conviction over the death of the little girl, Rana Yahya Al-Matari.
He was shot dead in front of a crowd of thousands of people, including many who recorded what happened on their mobile phones, in the capital city of Sanaa.
His hands were tied behind his back as he was led from a prison van before being laid down on a rug in front of the crowds of people.
Murder, according to Sharia law, is punishable by the death sentence, but families of the victim can request that the punishment is less severe.
al-Maghrabi admitted to raping the girl in his hut to the south of the capital city on the first day of Eid al-Fitr.