David Brin's latest post mentions creating the office of Inspector General of the United States. He provides more details here:
Only now picture this. What if we made a very simple change, by appointing and assigning and paying all of the inspectors through a civil service unit completely separated from each department's political chain of command? Indeed, separate from the legislative, executive and judicial branches?(Emphasis added.) I don't think it's constitutionally possible to create a governmental entity that doesn't fall into one of the three branches of government--the Coast Guard, NOAA, and the Public Health Service, for instance, are all part of the executive branch--but the underlying idea is intriguing. It also raises some difficult questions that go to the heart of how you create a democracy.
A uniformed service, with its own elite career path like the Coast Guard and NOAA and the Public Health Service... so that the word "general" has real meaning, encouraging higher-than-normal traditions and standards of conduct.
The first is independence: how do you create a governmental entity that can investigate the government itself without being unduly influenced by the very government it's investigating? This is the problem Congress faced when it created the Office of the Independent Counsel, someone who could investigate the President even though he or she was part of the Executive Branch but was appointed by judges upon certain triggering conditions. In Morrison v. Olson, the Supreme Court held the scheme constitutional because the independent counsel reported to the Attorney General, even though the Attorney General could remove him or her "only for good cause, physical disability, mental incapacity, or any other condition that substantially impairs the performance of such independent counsel's duties." (See Justice Rehnquist's opinion for a good explanation of how the statutes fit together.)
The second is checks and balances: how do you appropriately restrain the power of that independent entity? Remember Ken Starr's investigation of the Monica Lewenski affair? The power to investigate usually includes the power to conduct discovery, to force whomever or whatever you're investigating to turn over very large numbers of documents. That document search is an expense the party under investigation must bear regardless of whether or not there is ever a trial, and it's usually a substantial expense. How can you keep that independent investigator from conducting investigation after investigation of insubstantial issues, pursuing its own political agenda or simply to justify its own existence, all the while running up legal costs for whatever part of the government it's investigating? This problem may be a big part of why Congress let the independent counsel law lapse and we don't have independent counsels any more.
The Constitutional Convention solved the checks and balances problem by breaking the government into three pieces, instead of balling it all up in one autocrat, and designing a system in which no one piece could determine the extent of its own power. (Except maybe the Supreme Court, because they have the power to say what the words of the Constitution mean.) Ultimately, theoretically at least, the electorate retains the trump card of voting them out of office.
I don't have a clean answer to this problem. Both the executive branch agencies and Congress have an incentive not to be investigated because they don't want inefficiencies made public. That incentive suggests the power of any body of inspectors general would erode over time unless there's a balancing counter-incentive. One option might be some sort of citizen suit provision, where whistleblowers or other concerned citizens could trigger an investigation and receive a reward for evidence leading to a corruption conviction. That reward may give the citizenry an incentive to keep the office strong. Even that approach, though, doesn't solve the "runaway inspector general" problem or get around the fact that Congress will likely still have the power to cut the budget.
 The Constitution creates a government of limited power. That means the federal government has only those powers the Constitution gives it, and no others. Because the Constitution creates only three branches of government, it seems unlikely the Supreme Court would allow the government to give itself a fourth branch without something in the Constitution giving it the power to do so. You can do it if you go outside the government, which is the basic idea behind the Freedom of Information Act, but the government seems to have whittled away at that act's powers over the years.
 Who appoints the person is significant because the Constitution (Article II, section 2, clause 2) says this:
[The President] shall nominate, and by and with the Advice and Consent of the Senate, shall appoint . . . Officers; but the Congress may by Law vest the Appointment of such inferior Officers, as they think proper, in the President alone, in the Courts of Law, or in the Heads of Departments.(Emphasis added). The Court determined the independent counsel was an inferior officer and therefore could be appointed by Courts of Law.
 Give or take gerrymandering. The ultimate trump card, where the government torques off everyone so much that a supermajority votes them out, remains despite gerrymandering.
 I've just described a kind of qui tam action. For one example of this approach, see the new IRS Whistleblower laws (text available at whistleblowers.org).
Update: I wonder if it would help to involve the state governments. They're also elected, are independent, and their involvement might sit well with an originalist approach to the Constitution. Hmm. This idea needs further gnawing upon.