I feel like we're difting apart, how do we stop?

Sounds like you are in a normal relationship that has run it's course. Maybe take a break for a few weeks and if you miss each other terribly then get back together. There is an Ash song about this called Sometimes http://www.youtube.com/watch?v=CdisCk31Z_s It's just life.