You're not selfish. We all need our me time and I feel the same, when I spend most of the days with others and I am left with no time for myself. Not to mention when you have things to do. But then again, these times mean memories as well and there comes a time when all you'll be left with is these memories. I think you need to find a balance and you'll be fine.