Why do they keep Simon Cowell on the show if nobody likes him?

Last night I noticed that the Ryan Secrest thanked the judges, Randy Jackson and Paula Abdul but not Simon. And the camera showed Simon and he wasn't smiling!