I haven't studied modal logic, but it seems that what is "logically possible" is rather arbitrary. Logically possible in our physical world is likely not what philosophers think it is; some of these "logically possible" assertions strike me as similar to Descartes' claim that "the more perfect -- that is to say, that which contains in itself more reality -- cannot be a consequence of and dependent upon the less perfect. This truth is not only clear and evident..." (Third Meditation). ADHR criticizes all of the thought experiments on this basis.
There's another criticism for Frankfurt thought experiments. As a compatibilist, Harry Frankfurt accepts determinism. Yet if determinism implies that there is a single future path which rules out metaphysical responsibility, as Dennett, Pereboom, Honderich, Inwagen, Kane, and others believe, then Frankfurt has to explain how the "free will" necessary for moral (metaphysical) responsibility exists. His thought experiments assume "free will" because people make a free choice. He attacks what he calls the Principle of Alternate Possibilities, focusing this attack on actions -- but what really matters is choices. In Frankfurt's imaginary world, people can still make "free" choices for which they are ultimately responsible. (Technically, the agent in Frankfurt's experiment is only responsible for his action when the controller did not have to act.) What I'm saying is that Frankfurt's thought experiments don't get him any closer to the moral responsibility he wants; he still hasn't addressed the regular problems with "free will" moral responsibility. He won't be able to, either, because the metaphysical free will which allows ultimate metaphysical responsibility is incoherent. We are not responsible for our positions, and thus we are not ultimately responsible for our actions. We are only responsible in the sense that we have to be responsible (responsibility keeps people's behaviors in check). This critique is obvious, but somehow it hasn't gained traction in the philosophical literature; I imagine most philosophers would reject it out of hand simply because they don't want to admit that they've studied Frankfurt counterexamples these past 30+ years without noticing.
If we can get past this focus on ultimate responsibility, however, we can start to think about taking control of our lives and acting with at least some personal responsibility -- you aren't responsible for the position you're in, but goddamnit, you're gonna have to make the best of it. At the same time, this position recognizes (as we do legally) mitigating and aggravating factors in actions, and advocates that we study the causal factors behind behavior to maximize our ability to exert rational control over our bodies. Dennett struggles with this idea in Freedom Evolves, but this is Nietzsche's entire goal, actually, although he pushes people to do it without science (while recognizing that ultimately it may come only through science):
"learning to see -- habituating the eye to repose, to patience, to letting things come to it; learning to defer judgement (italics mine), to investigate and comprehend the individual case in all its aspects. This is the first preliminary schooling in spirituality: not to react immediately to a stimulus, but to have the restraining, stock-staking instincts in one's control. Learning to see, as I understand it, is almost what is called unphilosophical language 'strong will-power'. (Twilight of Idols, What the Germans Lack Section 6.)
Nietzsche makes the distinction between the causa sui -- the self-creating "free will" which we continually debate, and the self's "will to power", the causa prima which we really need to focus on. We can't be create ourselves, but ideally we can act with our rational self rather than our animal instincts, and avoid being jerked around by 'stimuli'.
UPDATE: Thinking more about the Frankfurt example -- in the linguistic sense, yes, Frankfurt's agent is responsible when he chooses to act as the controller wants him to act despite the fact that he cannot do otherwise. But this is a counterexample, and an outlandish one at that. It's really restricted to just this case -- when the agent acts differently than the controller wanted and must be controlled, the agent is no longer responsible. How is it supposed to apply in the real world?