If people who have a close relationship grow apart , they gradually start to have different interests and opinions from each other, and their relationship starts to fail.
He and his wife grew apart...
It sounds as if you have grown apart from Tom.
PHRASAL VERB : pl-n V P , V P from n