Where's the beef?

On the matter of risk control, let me ask a simple question. My apologies to deconstructionists everywhere, but I have to be a crass, pragmatic, empiricist for a moment: WHERE ARE THE DATA?

What hard evidence is there that the REB review process reduces "problems"? I will argue there is none whatsoever, but if someone has something, other than hunches and impressions, I would be delighted to see it.

This evidence might come from something straight-forward, such as how many "problems" arising from research were reported in 1950? 1960? and so forth. (So adjust it for growth, whatever.) I bet those data would show progressively MORE problems over the last 50 years, during which time there has been more and more REB screening. (see reason 1 below)

Ooops, we're doing more and more screening, but the problem rate is not going down, instead it is going up? Simple-minded non-academics might well respond that we must not be doing the right thing if the problem rate is getting worse? I agree, and there are two main reasons, and I do not think anyone can deny either of these:

(1) Anybody can complain about anything, no matter how much screening and no matter how trivial, and find someone to string them along for a legal fee. REBs can't have any influence on this aspect of our sue-happy, "I'm a victim" society (though law schools cutting back enrollment might help).

(2) The "bad guys" are not going to come asking REB permission.

One analogy on (2) is with gun control "reducing crime": sorry, but criminals don't register their guns, so it's a subterfuge. Granted, that hasn't stopped an REB-like industry from arising to help "reduce crime" in that way, but it's as bogus as expecting Dr. Frankenstein to apply to an REB.

So, rather than ask how much risk is the university willing to take, I would ask how much unnecessary and INEFFECTIVE red-tape are we willing to impose on innocent folks?

And if you think it can't get worse, the best guess is that there is a lot more regulation on the way.

I repeat, there is no HARD evidence that REB screening has had ANY effect in reducing problems. If someone can prove me wrong, I will gladly recant. I think we need to take an honest look at the possibility that all this time and effort is not accomplishing anything in the way of reducing complaints from the experimental subjects, whether valid complaints or not, certainly nothing commensurate with the magnitude of the effort. So why do we persist IF, as I presume, there is no evidence of success?

On the matter of what constitutes supporting evidence for the proposition that REB activities avoid problems arising in the real world conduct of the research, I gather that there is the impression that when a reviewer identifies something that is allegedly a problem, then some people see this as justifying the review process, that is, a problem found is a problem avoided.

But it doesn't work that way, and this assumption needs to be made explicit and challenged: "Revision requested by REB" does not constitute a "problem that would have bounced back from the outside". This is a fallacy, or at best a half-truth, certainly nothing that would serve you well during a tax audit.

By analogy, let's consider a company that is obliged to institute an accident prevention program for the workplace. Someone dutifully goes around and identifies alleged hazards, and amasses an impressive count of things "fixed." Is this relevant? No, and in the non-academic world it would seem preposterous to accept this hazard count as indicative of success of the intervention. The only acceptable evidence would be whether the actual rate of accidents declined.

And that should be the case for REB activity as well. For REBs specifically, the "problem found" count is flawed for a couple of reasons.

(1) That something is identified as a problem by an REB reviewer does not mean the subject in the experiment will see it as a problem. That is, there is far from perfect overlap between the "professional" and the "public" perception of a problem. This is supported by the fact that occasional problems arise from projects that reviewers approved as clean. There is no reason to believe that sword doesn't cut both ways, in other words, that things that reviewers see as potential problems would be non-events to the public.

"Revision requested by REB" may speak to the creative abilities of the reviewers, but it is not a barometer of the success of the REB process at avoiding risk.

(2) The review process seems to go for a "worst case" scenario, and then proceeds as if the worst case will be the NORM, which of course is nonsense! Just because something "could" happen does not mean it "will," and when the worst case is an improbable event this confusion becomes more wasteful.

I might be hit by a truck as I leave my office, but it would be unwarranted for my wife to book an appointment with the undertaker this afternoon on that presumption. I might win the lottery this weekend, but it would not be prudent to hit my boss in the face with a pie this afternoon. That's why the original concept of "everyday risk" was somewhat useful. Unfortunately the REB process seems to have evolved to a condition whereby the review assumes that the worst case will be the norm, some strange delusion of achieving "zero risk" having replaced the acceptance of everyday risk.

The accident metaphor that may be appropriate is flight insurance. The experiment is a discrete interval of time, like a flight -- does a problem occur during that specific interval of time? Life insurance for your lifetime involves an unfortunately high and definite probability of death, whereas flight insurance is whether you die during a discrete interval of time. I believe that most financial advisers have long considered flight insurance to be grossly over-priced, precisely the argument I am making about the REB review process. Confusion of different kinds of risk is quite useful to the insurance industry, but expensive to the consumer (experimenter). So for whom is it useful to confuse varieties of risk in the REB process?

And, no, considering institutional risk to be the collection of all experimenters working doesn't convert it to a cumulative risk, because each experiment (flight) is an independent risk.

Could vs. Will is a statistical odds problem. When you "correct" an unlikely "problem" the odds are that it wouldn't have come up in an actual experiment anyhow, and so it would not effect any change in the accident rate. "Revision requested" cannot be a metric for the success of the REB review process at avoiding risk in the experimental setting, and its imperfection just increases when the risk in question is unlikely. In addition to the per-decade incident-rate analysis mentioned above, here are two other ways to assess the success of the REB review process.

Consider an experiment: for a year a random half of the applications to the REB are approved without review (gasp!), whereas the other half get the conventional REB review. At the end of the year we look at the number of problems-arising in the actual experiment in each group. Anyone care to bet that the number of problems-arising in the unreviewed group would be any different than in the reviewed group? I think you would be foolish to expect any difference. Yet that "problem-actually-arising rate" is the only true measure of the success of risk avoidance by REB activity.

Another experiment begging to be done would be to take proposals approved at one research site and submit them elsewhere. My best guess is that the prospect of approval at the second (third, etc.) would not be too different from 50:50. Likewise take proposals rejected at one research site and have them reviewed elsewhere, again the outcome would be a crapshoot. Perhaps the strongest test of this would be to take the method section from published articles and submit them for review to various review sites, and again I suspect the repeat reliability would be distressingly close to chance. Analogous research has been done before (e.g., Peters, D., Ceci, S. Peer-review practices of psychological journals: The fate of submitted articles, submitted again. Behavioral and Brain Sciences. 1982, 5: 187-255), and the results were not popular, as "conventional wisdom" took a major hit. This procedure begs to be applied to the REB review process.

Yes, I know what you're thinking, that this last would have to be submitted to the REBs and they would never stick their necks out that far? But, to the contrary, I don't think this one requires "ethical" review at all, here's why: The alleged purpose of the REB review is to "protect the public," and this project would never involve the public. Of course, if the purpose of the REB review is to protect the REB, then maybe we should say that out loud?

There are some further considerations, but my point is sufficiently supported by these, namely that identifying a lot of alleged problems does not indicate the REB process is successful at avoiding risk in the experimental setting. If risk avoidance is the goal, declining problem-arising rate in the actual experiment is the ONLY valid measure of success, alleged hazard-fixed counts are bogus. And if we aren't interested in measuring that accident rate as it actually occurs in the experiment, in terms that would satisfy accountants and tax auditors and insurance agencies as opposed to post-modernists, then I think it is fair to speculate on what it is that is the real agenda of REBs.

As to what the alternative agenda might be, a good argument can be made that the screening process is simply another effort at social engineering, another mechanism for nurturing politically correct thinking, that is, another extra-legal control mechanism for harassing people into right-thinking. I find this bothersome for several reasons. One is that an acquaintance in Montreal was hounded using an REB into committing suicide in 1994, and another colleague is being hounded in the same way up in Edmonton as we speak. Consider also the case of an academic who was hounded for blowing the whistle on bad drug test results, because it suggests that the need for ethical enhancement lies somewhere else within the system than in the research proposal process. Of course, it can't happen "here".

Do we have hard evidence? Do we care? If not, to either question, why not? These are fair and just questions.