Because if downvotes are used to express disagreement then all it does is sort by “things agreed on by the first people in this community that saw it.” In fact many sites (like Reddit) explicitly say that’s not what it’s there for.
Is that how most forums and communities should operate to you? Do you believe that’s how we should sort things? If so then I simply disagree with that notion and there’s not much more to say. If you don’t, then there’s your answer.
It’s totally worth debating it because we could simply not have downvotes or at least hide vote scores. That changes people’s behavior demonstrably.