All governments suffer a recurring problem: Power attracts pathological personalities. It is not that power corrupts but that it is magnetic to the corruptible. Such people have a tendency to become drunk on violence, a condition to which they are quickly addicted.
Missionaria Protectiva, Text QIV (decto) Heretics of Dune (1984)
While the traditional distinction between left and right-wing politics does have its criticisms, it still is a commonly understood concept. Personally, I like the political compass extension with authoritarianism vs. libertarianism as added vertical dimension.
And when it comes to being a libertarian, well, it sucks. You are being attacked by the left or the right, depending on who is in power at the given time.
The reason is quite simple: Those in power did seek out that power, usually for its own sake. And once they are in power, they usually want to use that power. If only with the «you either do it ‹voluntarily› or I make you do it by force» method that was «nicely» applied during the pandemic.
And when they do use that power, they do it to further their political agenda. No matter whether they believe in it or not, it put them in power and maintains them in power. And this happens on the right or the left. The goals pursued in these agendas usually conflict with freedom of choice, as they interfere with the personal freedom of the citizens.
Thus, when you have freedom as the highest value and just want to be left alone, those political decisions undermine your freedom (or remove it outright, also as seen during the pandemic). And when you act up against it, well, then you are always against the political party in power.
Thus, if you are a libertarian, you are always on the losing political side. And I only mean the losing side, not the wrong side(*). Although given the moralization of politics (you are not wrong anymore, you are morally wrong, i.e. evil) it is often seen this way.
But still, while it might be a high price to pay to be always seen as morally reprehensible, the costs of not pursuing freedom would be much, much higher.
(*) Thanks, «Firefly»: «May have been the losing side. Still not convinced it was the wrong one.» Malcolm Reynolds in «Firefly».