The U.S. Federal Reserve Bank is about as popular as Nancy Pelosi at a lawn party in conservative and libertarian circles. It's hardly beloved on the left either, but that's a story for another day. Big banks, in general, have never been popular institutions. The new term, "banksters,” describes an old feeling. Over the years, no one bank has attracted more negative attention—sometimes veering into the realm of the conspiracy theory—than the U.S. Federal Reserve. Ron Paul popularized the "audit the Fed" rallying cry, and his adherents won't be satisfied until our national bank is abolished.
The Fed oversees the nation's payments systems, works to maintain the stability of the financial system, supervises and regulates banks (poorly), and—this is the one that really gets people worked up—conducts monetary policy. As the economy cools off or heats up, the Fed sells and buys Treasury securities to achieve a targeted overnight interest rate. This is not "printing money" (a term that’s outgrown its usefulness to the point of being almost meaningless now), nor is it controlling the money supply, despite what you'll hear.
The Fed was established by a 1913 law intended to mitigate financial panic, and its mandate has grown over time. Building "mission creep" discomfort went into full bloom when Richard Nixon took the U.S. off the gold standard in 1971. Libertarians, dedicated to "small government," are uncomfortable with a fiat currency system, unmoored by gold, where the supply of dollars is theoretically unlimited. They, along with conservatives in general, are concerned about "big government" controlling interest rates or bailing out auto Detroit auto firms, which they prefer had failed.
Ron Paul is the grandfather of the current anti-Fed movement. He’s stated the Fed is more powerful than Congress. His group is nostalgic for the gold standard, which they’re certain would stabilize the economy if we were returned to it, despite the lessons of history. The U.S. experienced recession after recession while on the gold standard, yet Paul and Co. have conjured up a mythological memory of an economy that hummed along on a steady forward pace.
The standard thread you'll see through the arguments to abolish the Fed is an "Armageddon mentality" based on emotion-driven fears. Paul Ryan is a perfect example. He's always in high dudgeon over the coming collapse, which never happens. Ryan laughably said we'd be the next Greece, and also thinks that the Fed should raise rates to help an ailing economy, which worked out poorly when it did that in 1931, or when the European Central Bank did it in 2008. Ryan has publicly stated that he gets his ideas on the Fed from Francisco D'Anconia's speech from Atlas Shrugged. I don't think it's necessary to say any more about Ryan's economic theories.
Suspicion of the Fed is rooted deep in American consciousness. Hamilton and Jefferson skirmished over it in 1791 in debating the need for a central bank. It was anti-federalist vs. federalist—a schism that continues to define American politics—and the federalist won that round. A central bank was created, but Congress let its charter expire in 1811.
A fear that about half of the American electorate has always had is that of a too-powerful central government. Americans were colonized by a foreign power, leaving them skeptical about allowing any government that much control over their lives. The rural dwellers at the heart of the founding of this nation distrusted the city slicker bankers whose updated manifestation is the Fed. The political narrative from the right has held that rural values are noble and prudent while urban values tend toward the wicked and profligate—sinful even. To put it in 2016 GOP presidential debate terms, Christian values vs. New York values.
It was inevitable that the Fed became such a fixation, especially after the gold standard was dropped in 1971. If you hate regulating the economy, there must be something else to blame. If you have an irrational fear of inflation and a lack of concern about the huge human toll of unemployment, it shapes your thinking.