WRAP is Certified as ‘evidence based’ by SAMHSA, but is it?

WRAP (Wellness Recovery Action Plan) has been certified by SAMHSA National Registry of Evidence Based Practices and Programs (NREPP) as an evidence-based intervention. This certification encourages states to implement it. But the evidence is not clear that WRAP improves any meaningful measure like decreasing hospital days, decreasing incarceration, decreasing homelessness or that it is for people with serious mental illness. The certification of unproven programs leads states to waste money.

WRAP was invented by and is the property of Mary E. Copeland. WRAP is a workshop run by a paid WRAP-trained facilitator who is a “peer” stable in recovery. (In the mental health industry, the word “peer” refers to someone who has had experience in the mental health system and may or may not believe mental illness exists). There is a manual for the instructor. The workshop is presented to groups of 8-12 individuals 26-55 years old with mental illness over eight weekly two-hour sessions. The purpose of WRAP is to help participants develop a wellness plan and

• Teach participants how to implement the key concepts of recovery (hope, personal responsibility, education, self-advocacy, and support) in their day-to-day lives.
• Help participants organize a list of their wellness tools–activities they can use to help themselves feel better when they are experiencing mental health difficulties and to prevent these difficulties from arising
• Assist each participant in creating an advance directive that guides the involvement of family members or supporters when he or she can no longer take appropriate actions on his or her own behalf
• Help each participant develop an individualized post-crisis plan for use as the mental health difficulty subsides, to promote a return to wellness

SAMHSA says their review found WRAP had a positive impact on
1: Symptoms of mental illness
2: Hopefulness
3: Recovery from mental illness
4: Self-advocacy
5: Physical and mental health

100% of the studies on WRAP used by SAMHSA to certify it were by WRAP’s founder and owner: Mary E. Copeland.

SAMHSA’s Review was extremely faulty

• Review based on a single test: SAMHSA’s certification took place in 2010, and they claimed to have used two studies to prove the intervention is evidence based.
  • While eventually there would be two different ‘studies’, they were both based on the exact same test on the exact same people at the exact same time. Colloquially, they were two reports on the exact same single test population.
    • One was a preliminary brief study that was published prior to SAMHSA approving the program in 2010.(FN1)
    • The second study was more complete but was not published until 2011 and therefore not likely peer reviewed before SAMHSA relied on it (FN2).

• Study had Conflict of Interest:  Mary Ann Copeland, who created the program and two others who work on it were authors on both studies.

• No review of long-term efficacy – The studies SAMHSA relied on were 6 month studies.

• Important outcomes not measured – WRAP does not claim to reduce hospitalization, violence, incarceration, homlesseness, awareness of illness and other important outcomes. So these were not measured.

• SAMHSA certified the program in spite of negative findings
  • No statistically significant differences were found between the two groups across time on somatization, obsessive-compulsive, hostility, and psychoticism subscales.
  • No statistically significant difference was found between the two groups across time on the subscale for ability to generate routes by which goals may be reached.
  • ‘Participation in WRAP may have had negative effects on empowerment

• SAMHSA still reports findings that are not likely true

SAMHSA, relying on the non peer-reviewed study still say

“WRAP participants had a significantly greater reduction in the severity and number of symptoms across time (from baseline to posttest to 6-month follow-up) relative to control group participants, as indicated by scores on the BSI Global Severity Index (p = .023); Positive Symptom Total (p = .027); and subscales measuring interpersonal sensitivity (p = .023), depression (p = .022), anxiety (p = .022), phobic anxiety (p = .034), and paranoid ideation (p = .009). No statistically significant differences were found between the two groups across time on somatization, obsessive-compulsive, hostility, and psychoticism subscales.”

None of those claims remained in the study after it underwent peer review.

Impact
In the United States, local and regional WRAP programs sponsored by mental health agencies and peer-run centers exist in every State, and over 25 States have integrated statewide WRAP initiatives. (Per SAMHSA)

Spending money on WRAP
In 2002, SAMHSA gave a Community Action Grant to the Minnesota Consumer/Survivor Network to deliver WRAP education and educator training to people in recovery around the state and to engage in a two-year consensus building process to encourage the adoption of WRAP. SAMHSA is also gave $70K to Vermont Psychiatric Survivors who’s director Linda Correy works to spread WRAP and promote it.

Conclusion
WRAP likely makes people who go through it feel better as do most interventions that involve and recognize the suffering of people with mental illness.   But clearly this has not been rigorously evaluated. The desired outcomes may or may not be the most important. The results are self-reported and not independently verified, there is no data showing efficacy beyond six months, no data comparing it to other interventions (say, non-formalized support or horse riding).  There is not information about which mental illnesses it helps more than others.

Other SAMHSA-funded Copeland Initiatives
SAMHSA contracted with WRAP owner, Mary Ann Copeland to write, so SAMHSA could publish and distribute the following brochures. We were unable to identify any evidence that any of them are evidence based or help improve the lives of people with serious mental illness:  Building Self-Esteem, Making and Keeping Friends, Speaking Out for Yourself, Developing a Recovery and Wellness Lifestyle, Recovering Your Mental Health, Action Planning for Prevention and Recovery, Dealing With the Effects of Trauma, Alternative Therapies.

POTENTIAL LIMITATIONS OF THE TWO STUDIES SAMHSA USED TO CERTIFY WRAP AS “EVIDENCE BASED”

Following are details on the two studies SAMHSA used to certify WRAP as evidence based

SAMHSA certified WRAP (Wellness Recovery Action Plan) as an evidence-based intervention.

Mental Illness Policy Org did not call on any scientists to review the studies. However, following are concerns we have about the two studies that warrant closer scrutiny.

1. Initial Outcomes of a Mental Illness Self-Management Program Based on Wellness Recovery Action Planning

This is the only study that had been published prior to SAMHSA certifying WRAP as being evidenced. Three of the authors of the study are associated with the developing and distribution of the program and therefore may lack the independence to accurately review the efficacy. The study found

“Those who attended the groups “revealed significant improvement in self-reported symptoms, recovery, hopefulness, self-advocacy, and physical health; empowerment decreased significantly and no significant changes were observed in social support.”

This could have been a placebo effect. The study did not use potentially more valuable measures incarceration rates, hospitalization rates, homeless rates, etc. It only looked for results one month after intervention. That was clearly too short a period for SAMHSA to rely on. The only measurements of progress were self-measurements by the patients that were not independently verified.

The measuring tools may have been inadequate.

• Brief Symptom Inventory
• Recovery (as defined by Illinois Office of Mental Health)
• Hopefulness as defined in the Journal of Personality)
• Self-advocacy and Empowerment (As co-developed by anti treatment advocate Judi Chambelin
• Social support
• Perceived physical health.

The study claimed

“A statistically significant decrease was found in global symptom severity. Scores on several symptom subscales—psychoticism, depression, phobic anxiety, obsessive-compulsive, interpersonal sensitivity, paranoid ideation, and general anxiety— also decreased significantly, indicating improvement.”

The sentence above is included on the SAMHSA site. But when the second study was published (a more detailed description of the same population as the first study (that underwent extensive peer review) that sentence does not appear. (See below).

2. Results of a Randomized Controlled Trial of Mental Illness Self-management Using Wellness Recovery Action Planning

This study had been submitted to SAMHSA in pre-publication (pre-review?) form and used by SAMHSA to evaluate WRAP in CA 2009. However it was not peer reviewed and published until June 2012. The delay between 2009 and 2011 may suggest reviewers had problem with the version presented to SAMHSA. Following are statements in this study that raise concern or deserve scrutiny.

• Three of the 11 authors had conflicts of interest in that they were connected with developing and distributing WRAP.
• Twice as many who received WRAP were lost to follow up compared to the control group: Of the 276 experimental subjects, 233 (84%) received the intervention and 43 (16%) did not. Eleven control subjects and 25 intervention subjects were lost to follow-up with reasons including death or ill health, moving away from the area, and formal withdrawal
• The study contains no data after 6 months
• The WRAP program served people with diverse diagnosis but did the study did not report results on efficacy by diagnosis (ex. 21% had SZ 38% were BP, etc.)
• As time went on the WRAP made less of a difference in symptom severity:  “On the other hand, the experimental vs control differences in symptom severity were larger between T(ime)1  and T2 and seemed to attenuate over the long term, even though WRAP participants were still doing better at T3 in the multivariate analysis.”
• The control group also improved: “Also noted in these results was improvement among control-condition subjects on all 3 outcomes.”
• Changes in hopefulness were modest: “Another finding inviting further explication is that regarding participants’ degree of hopefulness given that ob- served changes in raw scores were relatively modest.”
• “There were no differences by study condition in subjects’ self-perceived ability to construct successful plans of action, as measured by the ‘‘pathways’’ subscale.”
• “Regarding QOL (Quality of Life), again changes in raw scores were somewhat modest.”

Study limitations the researchers reported did not prevent SAMHSA from relying on results. Researchers reported:

“There are a number of study limitations that should be considered when interpreting these results.

• The first major caveat to our findings is that the study’s subjects were not drawn from a national probability sample of individuals with severe and persistent mental illness, which limits the generalizability of our results.
• A second caveat is the fact that all subjects came from a single Midwestern state, preventing an assessment of potential US regional variations in WRAP implementation and outcomes.
• A third caveat concerns the design of the study using a waitlist control condition. Use of an attention-control placebo would have allowed us to assess whether 8 weeks of peer interaction alone, and not the specific features of the WRAP intervention, caused the observed outcomes.
• A fourth caveat is that the study relied on participant self-report data that were uncorroborated by clinicians or objective observers such as research staff.” “Future studies using external raters and attention-control placebo interventions will offer a more rigorous evaluation of WRAP’s efficacy.
• A fifth caveat is that fidelity assessment was limited to WRAP facilitator self-report, while the additional use of direct observation to verify the validity of self-reports would have added credibility to fidelity assessment.
• Another potential confound is the high level of study subjects’ participation in peer-led programs and support groups, which may have exposed control-condition subjects to some of the same active ingredients as those contained in the WRAP intervention. As a result, the study may have underestimated the effects of WRAP relative to its impact in communities with low levels of peer support, as is typical in many areas of the United States.
• Finally, a longer time period of data collection might have revealed different findings than those attained at the end of the 8 months tracked in this study. All these limitations suggest that caution should be applied to interpretations from study results.”
• SAMHSA partially Funded the study: Cooperative Agreement H133B050003 and H133B100028.

 

FN 1: Initial outcomes of a mental illness self-management program based on Wellness Recovery Action Planning Cook, J. A., Copeland, M. E., Hamilton, M. M., Jonikas, J. A., Razzano, L. A., Floyd, C. B., et al. (2009).. Psychiatric Services, 60(2), 246-249 available at  http://www.illinoismentalhealthcollaborative.com/news/WRAP_Initial_Research_Findings.pdf

FN 2: Results of a Randomized Controlled Trial of Mental Illness Self-management Using Wellness Recovery Action Planning Judith A. Cook, Mary Ellen Copeland, Jessica A. Jonikas, Marie M. Hamilton, Lisa A. Razzano, Dennis D. Grey, Carol B. Floyd, Walter B. Hudson, Rachel T. Macfarlane, Tina M. Carter1, and Sherry Boyd available at http://www.mentalhealthrecovery.org/wrap/documents/SchizophrBull-2011-Cook-schbul_sbr012.pdf