"Balancing the budget" is a right wing lie. Fiat currency is basically unlimited, with few caveats. Running a deficit doesn't cause inflation automatically. It can, and in a few specific instances it has, but that's not the only reason that causes inflation. If that were the case, Japan would be suffering from hyperinflation, but in reality they actually are fighting deflation as hard as they can.

Taxes, as you have correctly pointed out, are the most straightforward anti-inflationary device that is available to the federal government. The current issue, in the US, is who is getting taxed, and who the government is handing trillions of dollars to.