Social responsibility, and what is "good"?

What would you do if you had enough money to not have to work? I recently said I'd still pursue software development - it's exciting and fulfilling and interesting and I'd still want to find a way to contribute intellectually to the tech world.

The person I responded to (let's assume he/she is male for pronoun simplicity) was less than enthused with my answer - once given a means to support himself financially, he would wrap things up and go find a way to help people in a third-world / struggling country.

So here's my question: is it selfish to want to pursue your own interests over some greater good / social responsibility? His goal in life is to work in order to prepare for retirement, after which he plans to find a way to help others. But the world needs people to also continue their own careers, advance their fields. So who decides who does what? Are you only allowed to be selfish and focus on your own career if you're simply blind to all the suffering going on in the world?

I understand that everyone has some sort of obligation to better humanity - but who's to say to what extent? Should everyone who is able and aware stop pursuing their dreams (additional point - can dreams be selfish? I suppose they can - but then why don't we push social responsibility to kids as much as "dream big, you can achieve anything"?) to serve others? How do you decide who has to give up their dreams (to be the best X, to achieve Y) for the betterment of some other society? Or - is helping other people automatically going to trump the fulfillment of achieving any of the previously identified dreams?

I'm conflicted. Frankly, startups, new media, and technology rank much higher on my list of interests than poverty and hunger. Does that mean that, hands down, I'm a selfish person for not caring about others? Does that mean all interests are ranked - some interests are inherently better or less selfish than others? Does being concerned about poverty and racial issues mean that you're a better person than those who care about the environment? Or gender issues? Or socioeconomic issues within the US, versus those outside?

At what point do we start drawing the line and saying, "you don't care enough about X. You're being selfish"? Isn't this a slippery slope - creating these strict definitions for "caring about others" and judging people based on it? By these definitions, I can care extensively about and for others in my life to the extreme and still be considered selfish. Can you set bounds within which caring is irrelevant? If you always deliver soup to sick friends - must you also donate and be involved in Red Cross / Salvation Army work to be a good person?

I don't know where to draw the line.

Let me know what you think on Twitter.