Any basic form of democracy has the critical problem of incentivizing getting and staying elected over absolutely everything else. By definition, it selects for leaders that will garner votes, and thus any actions or policy that seemingly help people other than those in power are only selected for insofar as they serve that final purpose. This can still result in benevolent leaders, who garnered votes by voicing their intent to improve their voters’ lives in this or that way, and were honest about it. But such candidates are often overshadowed by ones who use tactics to manipulate how voters vote, often towards governances that do not, in fact, support them.
This problem, as pointed out by Brennan, is exacerbated by large numbers of voters who are very susceptible to this form of manipulation, but even with a well-informed electorate (and ignoring the problems of vetting its members), that critical incentive still stands. An epistocracy would merely select for a politician that is better at manipulating educated people.
Another issue is lack of error correction. When a bad candidate is elected, they hold office for the duration of their term. During that time, they are essentially free to enact whatever policy they want unless they do something deemed illegal and are held criminally responsible. The so-called system of checks and balances in the United States seeks to limit this, but it operates through elections too, and thus is susceptible to the same kinds of manipulation. This means that bad selections, even when recognized, cannot be effectively removed and replaced with better ones, especially in positions without term limits.
These problems are not easily soluble, however, and the obvious solutions to all of them each have significant drawbacks.