I'm a few days late commenting on this post from the Frontal Cortex: http://scienceblogs.com/mt/pings/35979
Go there to read many more learned responses than mine. These are just my thoughts.
Another consideration about the trolley example: in the case of the choice between killing five workers and killing one, the driver is choosing between the only available courses of action, either of which results in killing someone. In his split second considering the situation, he makes the decision he knows he can better live with. You, innocent bystander on the bridge with your fat buddy, are not inevitably involved in the accident.
You're throwing a whole new variable into the mix by making yourself an actor in the situation. Who are you to decide that taking your buddy's life is more morally acceptable than a tragic accident? What if the trolley derails after hitting the fat man, and several passengers die? What if the trolley driver dies after hitting the fat man? What if your buddy, in panic when you shove him, manages to grab you by the belt and drag you over the bridge rail with him? And you don't die, but are left quadriplegic?
My point is that the questions of morality involved go farther than a simple arithmetic of how many are killed. When the big picture is out of your control, you fiddle with the details at your peril. I think a parallel is the argument for the Iraq war that Saddam was a murderer, and that by not removing him, we were somehow complicit in those murders. This argument failed by both tests of morality - defeating Saddam is taking many more lives than would otherwise have been lost, and, by making ourselves actors in the situation, we have become culpable for the murders we commit. It was never true that we were complicit with Saddam's murders - that guilt was his. We are now complicit with many murders.
You, on the bridge with your buddy, are innocent of the accident, horrible as it may be to witness. If you involve yourself by killing your friend, you have become a murderer.