Rep kinda matters, but not really. It's how people look at you. Rep can be earned just for saying something funny, or something true, or even when someone feels sorry for you, they send it to you as a rep. I've had a friend who earned around 20 rep on HF just for posting a funny video. In my personal opinion, it kinda hurts to see a neg rep on me, but mainly it's just a troll which you can't do anything about it.
Vouches on the other hand matter, but also don't matter. Vouches just means how many deals you've done with others, people with low vouches mainly means they don't go into the marketplace that often, legit or not, while people with higher vouches are often in the marketplace.