This actually what conservatives like Mitch and Ted Cruz want to happen
They have been systematically fighting to defund the IRS for decades, to cause dysfunction, and for people to turn on it.
The result of that is tens of thousands of call center jobs lost, audits on the wealthy have become less frequent, now low-income people are disproportionately targeted, they can't attract and keep enough skilled accounts to properly audit rich folk and businesses, and it blocks the IRS from being capable of moving to a system where they do people taxes for them. Defunding the IRS is a very bad idea unless you want to give the wealthy a massive win.
Most IRS workers are a bunch of regular joe civil servants as well.
A properly funded IRS would proved tons of good middle-class jobs and could be unleashed to go after the biggest tax dodgers.
It is like you are pissed about how the management of a supermarket ripped you off, so you take your anger out on the cashiers. You are falling for the classic right-wing trap.