You are assuming that the Caliph Umar was the one who began the process of putting discretion into such legal rulings.
There are many Hadith that describe The Prophet’s own discretion with regard to this subject and these are further elaborated by Aisha.
The Caliph Ali also enumerated that he would not continue to punish the person who steals (without cause) as he said:
"I feel guilty in front of God if I leave him without hands (i.e. amputating both) by which he eats, and feet, by which he walks"
In order to understand the Hadd and the application of Hadd you have to understand this very central concept:
For every duty (taklif) imposed upon a Muslim there is always a corresponding right (haqq).
If one’s rights are not fulfilled, than a duty cannot be imposed.
- Atheist: Can your Lord fit the entire universe into an egg without making the egg any bigger and the universe any smaller?
- Jafar as Sadiq: I would like you to look at the sky, and at that bird in the sky, and at that tree, and at all those people who have gathered around us and now look at me.
- Atheist: I see them all.
- Jafar as Sadiq: If Allah can fit all of those things inside the tiny pupil of your eye, do you think that he cannot fit the universe into an egg?