I believe the only way to remove one, is to add another one.
Microsoft is slick, they wanna make money so they're not going to let you remove shit without you giving them another payment method
I had to ring when I could not remember the account details to one of my accounts with a card on, and they just removed it for me in a flash, that's your best bet if you can't find another way