1.
used to add a comment on sth that you have just said, usually adding sth that you think the other person will be interested in :
It's a nice place. We've stayed there ourselves, as a matter of fact.
2.
used to disagree with sth that sb has just said
SYN actually :
'I suppose you'll be leaving soon, then?' 'No, as a matter of fact I'll be staying for another two years.'