Trial and error

On Wednesday I gave a talk at the Ryerson University Daphne Cockwell School of Nursing's Research Day. The other three papers in the session in which I presented were all very interesting papers, each dealing in some way with the challenges of research ethics review processes.

The three papers in the session were on: the use of qualitative interviews and participants' ideas of risk; conducting research with Aboriginal persons and the use of innovative community-based research designs with potentially vulnerable participants.

One thing that almost every speaker raised was that ethics review boards sometimes make less-than-great decisions. Sometimes ethics review boards have unrealistic expectations. Sometimes they grossly overestimate the level of risk. Sometimes they overstep what others see as their clear jurisdiction. And sometimes they make requests of researchers that are felt to be inappropriate.

A few individual stories were shared (with collective gasps from the audience) about the kinds of less-than-stellar decisions ethics review boards have made. I cringed at some of them along with everyone else.

I know that these kinds of things happen. My experiences on a variety of ethics review boards over the past years has demonstrated that, yes, sometimes, ethics review boards or rather the fallible people who sit on these boards can make errors or mistakes. They can certainly overstep their boundaries and they can make unrealistic requests on researchers sometimes. Yes, these things all happen. But without sounding defensive, I think a "step back" needs to be taken.

I could start talking here about how research ethics is an underpaid, undervalued part of research that researchers "love to hate" (Case in point: I've never been to a lecture on how great research ethics processes are but I can easily recall many lectures focusing on the drawbacks of ethics review boards and processes). I could talk about how many researchers publicly view ethics review boards as "barriers" to conducting research, and the processes of ethics review as simply more "hoops" through which to jump — unnecessary and bureaucratic processes that are neither meaningful or helpful. I could also mention the fact that ethics review boards are the very last committee anyone in the faculty wishes to join, and that most boards are functioning without enough people, time, support or funding. I could mention that working in research ethics is looked at by many as a kind of hobby or "passion" and an "add-on" to one's "real academic work". I could even talk about how some feel that, in Canada, ethics review boards work without clear and explicit guidelines but are instead charged with the responsibility of interpreting (very good but) somewhat vague principles embodied in our federal guidelines, the Tri Council Policy Statement. All these things are, to some degree, true - and anyone who serves on, administers or chairs an ethics review board will be happy to sit down and discuss these issues at length with you.

But this isn't what occurred to me as I listened to these clearly smart and experienced researchers talking about the kinds of things ethics review boards don't do well. What did occur to me is the way that ethics review boards learn and the difficulty it seems researchers have with acknowledging that errors in judgement can be made and learning should and can then occur on the part of ethics review boards. Then perhaps changes in the processes that they require others to go through can subsequently be made. Much of the way we set norms in the research ethics review process, while grounded in policies and principles, occurs through building a body of relevant cases and referring back to these previous cases for guidance as new challenges arise.

Ethics review board members learn through more than simply reading and applying policy. For example, while many researchers in the social sciences and humanities believe that boards apply mostly biomedical Western-centric guidelines to all research without flexibility and without consideration of non-medical research or non-Western norms, I'd say that I think ethics review boards are doing a pretty good job of learning from experience as more and more protocols are reviewed, not just from what is considered to be overly rigid policies. Most ethics review board members and chairs that I talk to don't just talk about learning from guidelines. In fact, they talk much more about applying guidelines and trying to think about guidelines within the context of the kinds of research that they review. Most say that they would never apply the same kind of review to a biomedical trial as they would, for example, to an ethnographic study. Nor would they force Western norms on projects, say, involving Aboriginal partners as participants. This doesn't mean that many ethics review boards haven't made the kinds of mistakes that researchers are citing and that were addressed in the papers yesterday and other talks that I've attended. But I think it does mean that ethics review boards are willing to learn from their mistakes and also willing to be flexible as they can, while still meeting their mandate of protecting participants, by using common sense and an open mind when approaching research of all kinds. I agree that grievous errors in judgement have occurred - yes, risk is often overestimated, researchers may be asked or told to do things that might not make sense within the confines of their research. But I think that positive changes are occurring.

What I hear from ethics review boards is that they want the ethics review process to be a conversation, instead of a lecture. They want for mutual and iterative learning to occur, with researchers informing ethics review boards and vice versa. Maybe I have rose coloured glasses on here, but I think ethics review boards are getting better at reviewing research in more contextual ways. I think policies are improving - the Tri Council Policy Statement second draft, along with other supporting Canadian documents, better addresses issues like risk, social science research, qualitative research and research with Aboriginal persons. It's still nowhere near perfect, but I'd challenge you to find a process as demanding, variable and specific as this, that is perfect. I'm hoping that the kinds of anecdotes about bad individual decisions on behalf of ethics review boards that are told for effect can eventually be replaced by stories of how an ethics review board learned from an error in judgement, revised processes and worked together with researchers to make the processes and outcomes of ethics review more relevant and meaningful.

Facebook and research

Yesterday at the annual meeting of the Canadian Society for Bioethics, we presented a paper on the use of social media networks like Facebook (FB) in the conduct of research. The goal of the talk was twofold: first, to describe the ways that researchers use networks like FB and second, to make colleagues aware of the challenges that the use of these social networks create for ethics review boards.

In many disciplines, social networks are used as both a source of recruitment and a source of data. If you want to study persons between the ages of about 15-40, FB and other social networks are arguably a good way to reach people and a rich source of demographic and contextual data about those people. Between the three ethics boards that I'm involved with, we've reviewed a significant number of research protocols that involve social media networks and the numbers are increasing. Here's an example: those who conduct research on children who are chronically ill are looking at ways of engaging participants using the social media networks that make up much of the daily life of sick or hospitalized kids. Many kids with chronic illnesses such as cystic fibrosis or childhood leukemia are often very physically isolated yet they are able to maintain an active social life through the internet and networks such as Facebook, Second Life or Twitter. These active social networks represent an opportunity that researchers are realizing and starting to access. This kind of research is not going away, but likely only going to become more common and ethics review boards need to be aware of some of the unique issues that arise in this context.

There's lots to talk about here, but for this entry, I'll focus on a couple of issues that arise out of the recruitment of potential participants using social media networks, specifically FB.

Social media networks are often used for recruitment of potential participants. A researcher can easily access FB groups that are created to provide a place for those with common interests or common problems congregate and connect. For example, there are, as of today, 137 FB groups for breast cancer survivors, 8 FB groups for survivors of childhood abuse and 22 FB groups for persons living with AIDS. For many researchers, these are a gold mine of opportunity to recruit research participants, by posting a recruitment notice on the FB group page or targeting all the group members with a Facebook email message inviting them to consider participating in a study.

Is this different from posting an information notice on the bulletin board of a community centre or cold-calling people? I think it is. FB is a community. That's why people like it there. It is a virtual community in which people have friends and sustainable connections, sometimes with FB "friends" who they have never met in real life. Disclosure is the norm, a community value. It may even be more than a norm — there may be subtle or more obvious social pressures to disclose information about oneself. Post a profile without a profile picture? You'll risk being inundated with messages from FB friends urging you to post a flattering picture of yourself. Post a profile without much information? Again, you'll find your profile stark and even uninteresting, compared to those peoples' profiles who post their relationship status, age, birthday, work, favourite movies, books and tv shows and photos of their life events. The temptation is very strong to adhere to the norm of disclosing plenty of personal information to your network of friends. So for a researcher to send a recruitment notice to you through FB might mean that researcher already has a significant or even a copious amount of information about a potential participant, as opposed to the researcher stapling a recruitment notice on a bulletin board and waiting to see who might respond.

For those of you who might not be familiar with FB and how much one can or cannot see about others on their profile, you might ask, "How can a researcher see information about that person's permission?". FB has various gradients or levels of privacy. When you create your profile, you set your privacy settings to control who sees what about you and how much they can see. Some people allow no information to be seen by persons other than those with whom they are FB friends, thus restricting access to their information to persons they know. However, some other people leave their profile "wide open", i.e. with all their posted information visible to anyone who might happen upon it. In between these two extremes are various privacy configurations which can allow certain people to see certain things and not other things, etc. etc.

Two important points about this understanding of privacy. Our understanding of privacy outside of a social media network is reasonably clear. We close our bedroom curtains when we enter our bedroom in order to prevent neighbours from seeing in. However when we're relaxing in our backyard, we are fully aware that our neighbours can see us then. We understand the difference, in many aspects of our daily lives, between what we are putting out there publicly and what we are holding back and keeping private. To use another example, we share information about private things like problems in our personal relationships or our opinions on sensitive issues with our very close friends quite easily but might hold back details about these kinds of private issues with acquaintances or more distant friends. We're clear on how much we want to share with each category of friends and over time, people get very practiced at managing that. It's clear in research that some kinds of information are private — things like your medical information and opinions are private and researchers must have explicit and clear permission before accessing those kinds of data. However on FB, it's much less obvious what is private and what is public. It may even be confusing — confusing not just for ethics review board members who might have little insight into the community values and norms of social media networks, but also to novice FB users and overly-enthusiastic but less computer-savvy FB members who may not even be aware of how "public" their personal information really is. If this understanding of privacy is unclear to members of the social network, chances are it's just as unclear to researchers (who may be accessing profile information without consent and without thinking that they might need consent) and to members of ethics review boards, who are already mandated with knowing about many kinds of special groups, unique methodologies and special contexts in which research occurs. Social media networks and ideas about privacy within these networks constitute yet another thing that REB members must start to understand.

In my next blog entry, I'll write a bit more about the use of FB in research and some more implications for how research ethics reviews are conducted.

Reasonable Expectations for Institutional Ethics Protections

One of the hallmarks of this thing we call "civilization" is the fact that the enforcement of norms isn't just left entirely up to the judgment and resources of individuals: we implement our norms by building various kinds of institutions. But sometimes the existence of the most visible of those institutions makes it too easy forget just how much work is actually done by less-formal, less-visible institutions. A case in point is the way people look to formal institutions — things like journals and ethics boards — as panaceas in the realm of research ethics.

Last Sunday, Nancy blogged about the Kuklo case, which involved research falsification by a former US Army surgeon. The NY Times has this update: Discredited Research Study Stuns an Ex-Army Doctor’s Colleagues

What most interested me from this update was this line:

The Walter Reed episode also shows how medical journals may fail to conduct adequate due diligence on the studies they publish — information that other doctors rely on for guidance.

I'm not sure that's right: it's not at all clear to me that the journal involved in this case "failed". This brings to mind a very good point that Nancy made with regard to the Coast IRB sting. Nancy pointed out that ethics boards necessarily have to trust researchers submitting protocols in a number of ways. Ethics boards are charged with exercising oversight, but they simply cannot verify every single detail. It's literally impossible. The same goes for journals like the British Journal of Bone and Joint Surgery, which published Kuklos's falsified research. By and large, the academic publication system runs on trust, as it must. And that trust is generally warranted: people are by-and-large honest. The exceptions, of course, can be very worrisome indeed. But in terms of the role of institutions such as journals and ethics boards, two specific points need to be made:

1) The first is that there is a sense in which the role of institutions is actually very small. So a bad end result doesn't necessarily mean that result was attributable to a failure of formal institutions. In this case, there may have been a failure on the part of the Journal; but it may also have been a failure of training, mentoring, professional socialization, or even parenting. Maybe all of the above.

2) Second, fixing problems through trying to achieve perfection in the functioning of formal institutions is difficult, costly, and potentially invasive. By analogy: the only way to reduce crime to near zero is to put a cop on literally every street corner — an enormously costly and disruptive solution. So the key is not necessarily to leap from bad outcome to calling for overhaul of the institutions involved. The key is to look carefully at those institutions, ask what roles that are and are not likely to be good at playing, and do what we can to help them achieve an approximation of best practice.

Outsourcing clinical trials: Further thoughts

A story on the outsourcing of clinical trials appeared recently in the National Post. We've written about this topic recently here and the National Post article reiterates a few of the key concerns about outsourcing. This article also addresses the issue of outsourcing from the perspective of some Canadian researchers who have taken their projects elsewhere.

Here's the story: Outsourced side effects

Here are a couple of things that I found most interesting about this article:

The comments from researchers interviewed for the story seem to demonstrate, to some degree, limited understanding of the usefulness of research ethics review. One researcher from the University of Toronto claims that the kinds of processes required in order to be allowed to conduct research are "bogging down" researchers so they are turning to other locales to conduct research. Canadian review processes are supposedly part of the problem. Outsourcing clinical research doesn't necessarily solve that problem, however. Research being conducted in other countries still requires ethics review in Canada at the institution at which the researcher is affiliated or in the case of a pharmaceutical company, with an independent or private Canadian ethics review board to ensure that the research processes adhere to our federal guidelines. The kind of processes that "bog" researchers down can actually help to avoid the problems that are reported by researchers elsewhere in the article. A Canadian ethics review process can't always anticipate the kinds of local problems identified in the article (like falsification of data or research being conducted on persons without consent), but it's a good start to avoiding them. While the bureaucracy to which this researcher refers is a source of much frustration for researchers, these processes can help to facilitate the rigourous and ethical conduct of research — wherever it's being conducted.

The model requiring "local ethics review", mandated by the Tri Council Policy Statement also means that some kind of review needs to happen wherever the research is outsourced to. This is a necessary part of gaining access to other locales and helps researchers to begin to understand local issues, standards and concerns.

Some researchers may claim that the process of an ethics review in a country where research is outsourced to would lack the same kind of rigour or validity of the processes we have here. The article quotes one researcher, commenting on a kind of cultural divide he experienced when problems occurred with the data from his research project, outsourced to Iran:

"It's a different culture, a culture with a lack, perhaps, of a long-standing research tradition dedicated to high standards and integrity."

It's dangerous to make broad negative generalizations about the "lack" of research integrity in other countries while claiming a kind of moral high ground here in the developed world. We've certainly had our share of research scandals. We've seen (and blogged about) researchers falsifying research to make themselves look better, to make a product or medicine appear more favorable than it really is or to promote a product or person. We've read about researchers ignoring conflicts of interest, forging signatures of colleagues and claiming research as their own that in fact is not. The list goes on. What's really important both here and overseas is good training and mentoring for researchers, and vigilant oversight from ethics boards.