What is the most difficult aspect of managing?

Managing those you supervise, managing those who supervise you, managing people across teams, functions and throughout the organization -- which aspect of managing do you find most difficult? Why?