Yea, if the person has all the Info to it, they call it in an Microsoft will take it back an then if you were the last one to recover the account, you will be console banned
if you tag swapped it no, unless the account you tag swapped it to was someone elses yeah... a tag can always be called in and taken back if they had it on their xbox. but if it's swapped... you're fucked there's no way to get it back