Friday, May 18, 2012

Open Access Mandates: Ensuring Compliance

Three years ago, I wondered aloud how we should judge the success of an Open Access (OA) mandate. In particular, I wondered what level of compliance it was reasonable to expect a mandate to attain, and pointed out that OA advocates believe it is possible to achieve a 70% compliance rate within two years, if the mandate is compulsory.

It is therefore striking that, although the Wellcome Trust has had a compulsory mandate in place since 2006, The Times Higher Education reported in March that the Trust is still only achieving a 55% compliance rate. As a result, The Times Higher added, Wellcome’s head of digital services Robert Kiley wants to get tough on those who do not comply. The current situation, said Kiley, is “simply not acceptable”.

Elsewhere, Nature reported that of the 55% of researchers who comply with Wellcome’s mandate, 85% do so by means of Gold OA. One benefit of Gold OA, presumably, is that the author can hope to pass responsibility for compliance over to the publisher. While this does require paying a Gold OA fee, the Trust will pay this fee for the researcher. However, that 85% figure might seem to suggest that authors are more reluctant to comply with a mandate than some assume.

In my 2009 blog post I also reported that when, in 2008, the US National Institutes of Health (NIH) upgraded its request that researchers deposit their published papers into PubMed Central (PMC) into a requirement, compliance accelerated to 49% by the end of the year (from 19%).

In order to establish what level of compliance the NIH is achieving today, I contacted its Office of Extramural Research (ORE). Below is the (slightly edited) transcript of the email conversation I had, which was split over a number of emails.

The upshot is that the NIH is now achieving 75% compliance. 

This invites a question: Why is NIH proving more successful in achieving compliance than the Wellcome Trust? Here are some quick thoughts:

  • NIH is the largest public funder of medical research in the world, and so perhaps has greater authority than the Wellcome Trust
  • NIH has been more proactive in ensuring compliance
  • Where the Wellcome Trust insists that any embargo imposed by a publisher is no longer than six months, the NIH still permits embargoes to be up to 12 months (so perhaps publishers are more willing to post NIH-funded papers in PMC?)

As I say, these are just initial thoughts. I welcome comments from others.

One thing I find noteworthy is that, while the NIH clearly takes a close interest in compliance levels, it does not collect statistics on enforcement actions. With the growth in compliance now apparently levelling off at NIH, this would surely be useful information — and not just for NIH itself, but also for other funders like the Wellcome Trust and Research Councils UK  (RCUK), both of whom are currently looking to beef up their open access policies.

Q&A with NIH

RP: Can you give me some information on the current compliance rates for the NIH Public Access Policy, and explain how the NIH ensures compliance?

NIH: OSTP has submitted a report to Congress that included a current overview of the NIH policy, compliance level and impacts on Page 12.

Our current compliance efforts have been focused on outreach and engagement with institutions and publishers, as outlined on pages 3 and 4 of David Lipman's testimony to Congress in 2010.

RP: Ok, so after reading these documents I conclude the following:

1. The initial voluntary policy saw compliance rates of 19%.

2. The mandatory policy (introduced in April 2008) saw compliance grow to 49% by the end of 2008, 70% by the end of 2009 and currently stands at 75%.

3. In order to increase compliance the NIH has a) engaged in awareness programs; b) improved its ability to track papers resulting from NIH research awards; c) developed new systems to assist sponsored research offices at universities and medical research centres in tracking their compliance; d) improved the submission process for authors by i) co-opting publishers to deposit for authors and ii) introducing the NIH Manuscript Submission System.

NIH: To be clear, we introduced the NIH manuscript submission system in 2005. We have made modifications and improved our guidance, but it is not new.

We don't think it is accurate to say that we have 'co-opted' publishers, as the NIH policy does not apply to them.

RP:  Why do you think publishers agree to deposit papers for authors?

NIH: We generally hear three reasons why some publishers take an active role:

1) They want to be on PMC because of its high visibility, integration with other resources, and secure and free archiving — many journals do this for all their content, regardless of whether it falls under the NIH public access policy

2) They want to provide a service to their authors

3) They want greater control over the version posted to PMC (some want only the published version posted, and others want only the final peer-reviewed manuscript posted)

Green or Gold?

RP: Can you say what percentage of the papers deposited in PubMed Central are papers that have been published in a subscription journal and then self-archived (i.e. Green OA) and what percentage have been published in an OA journal for a fee (i.e. Gold OA)?

NIH: We do not characterize PubMed Central (PMC) deposits in terms of green and gold Open Access (OA). PMC is more than a repository for papers resulting from NIH-funded research.  See here.

All 2.4 million PMC articles are available without charge on PMC. This number includes about 225,000 author manuscripts, and the rest are final published versions that have been provided to PMC by publishers.

We do not have a simple way of generating counts of which papers are from subscription based journals and which are not. However, to put the above numbers in context, note also that close to half of the articles in PMC are from the digitized (scanned) issues of journals from the period between the mid-1800s and the mid to late 1990s — predating the birth of OA journals and the NIH public access policy.

About 450,000 articles make up the PMC OA subset. The term Gold OA generally refers to articles that are available under a very open license (such as CC-BY) and involve an open access fee charged to authors. The articles in the PMC OA subset come with a variety of licenses, not just CC-BY.

In addition, some of these OA articles are from journals that, to our knowledge, don't charge an OA fee — the Rockefeller Press journals, for example here


RP:  I came across a message on the Liblicence mailing list recently from someone who appeared to have concluded that there is no real force behind the NIH policy. Would it be accurate to say that the NIH currently has no way of enforcing compliance?

NIH: The NIH Public Access policy is a term and condition of award.  As we state in our FAQ: A grantee's failure to comply with the terms and conditions of award may cause NIH to take one or more enforcement actions, depending on the severity and duration of the non-compliance. NIH will undertake any such action in accordance with applicable statutes, regulations, and policies.

NIH generally will afford the grantee an opportunity to correct the deficiencies before taking enforcement action unless public health or welfare concerns require immediate action.

However, even if a grantee is taking corrective action, NIH may take proactive action to protect the Federal government's interests, including placing special conditions on awards or precluding the grantee from obtaining future awards for a specified period, or may take action designed to prevent future non-compliance, such as closer monitoring. See Enforcement Actions in the NIH Grants Policy Statement (11/03).

RP: Can you tell me how many researchers have received research funds for the last five years, and then give me annual figures on enforcement actions that have been taken against researchers who have  failed to comply with the public access policy. E.g. how many enforcement letters have been sent out, how many researchers have had special conditions placed on their awards, and how many have been precluded from obtaining future awards for a specified period (and for how long a period)?

NIH: We funded over 228,000 grants in the past 5 years, which support hundreds of thousands of investigators. NIH public access policies have been developed centrally, and compliance activities are carried out by individual NIH institutes and centers.

There are currently terms and conditions which require institutions to comply with the public access policy. When an investigator is found out of compliance with the NIH public access policy, the NIH institute or center in which the award resides will send out letters to bring the investigator into compliance.

NIH does not currently collect statistics on compliance efforts.


Bill Hooker said...

Thanks for following up on this, Richard. (I think my request for actual data was bounced off the list, dunno why.)

The link to the OSTP report (first A of Q&A) isn't working.

Richard Poynder said...

Thanks for pointing out the dead link Bill. It should be fixed now.

Your message was posted on Liblicense, and indeed your guess seems to have been pretty accurate!

Richard Poynder said...

Cameron Neylon posted this comment on Google+:

Basically the reason for higher compliance with NIH as far as I can tell is that grant reporting is coupled into deposition. So you can't include a paper in your grant report unless it's in PMC. I haven't actually seen this in action myself but it's consistent with the fact that places like Liege get very high compliance with institutional mandates when reporting and review is technically tied into compliance.

Stevan Harnad said...

How to Maximize Compliance With Funder OA Mandates: Collaborate With Institutional Mandates

Here are my hypotheses as to why neither NIH nor the Wellcome Trust (WT) has a compliance rate of 100% (and why NIH has a somewhat higher compliance rate):

1. How To Comply. Both the NIH and WT mandates designate Gold OA publishing as one of the means of fulfilling the mandate, instead of uniformly designating fundee self-archiving as the sole means of compliance (whether or not the fundee publishes in a Gold OA journal.

2. Who Complies. Funder mandates only apply to fundees: only fundees are bound by them. Yet fulfillment can be done by either fundees or non-fundees (publishers, especially in the case of WT), instead of uniformly designating fundee self-archiving as the sole means of compliance.

3. When To Comply. The designated timing for compliance with both mandates is not immediately upon publication -- instead of uniformly designating fundee self-archiving immediately upon publication as the sole means of compliance (even if the self-archived draft is not made immediately OA). As noted, it is in publishers' interests to make compliance as delayed as possible, and to leave it in their hands rather than the fundees'.

4. What Version To Deposit. It contributes to the delay in compliance and the ambiguity as to who is fulfilling the mandate (the fundee or the publisher) if compliance can wait for the publisher's PDF instead of uniformly designating fundee self-archiving of the refereed final draft immediately upon publication as the sole means of compliance (even if the self-archived draft is not made immediately OA and the publisher's PDF is optionally deposited later).

5. Where To Deposit. Both NIH and WT grants stress direct deposit in PubMed Central (PMC), instead of uniformly designating fundee self-archiving of the refereed final draft in the fundee's own institutional reposiitory immediately upon publication as the sole means of compliance (even if the self-archived draft is not made immediately OA and the publisher's PDF is optionally deposited later), thereby recruiting fundees' institutions to monitor and ensure compliance with the fulfillment conditions of the grant (as institutions are always very eager to do!).

Institututional ID/OA Mandates Work. None of these delays, ambiguities or uncertainties applies to (effective) institutional mandates such as U. Liege's model ID/OA (immediate-deposit/optional-access) mandate. Not only can author self-archiving in the institutional repository be designated by institutions as the sole means of submitting research for institutional reporting and performance assessment (as Cameron Neylon correctly points out), but institutions are in a position to monitor deposits continuously, not just when a research project grant (which may last for years) has elapsed.

Mutual Reinforcement Between Institutional and Funder Mandates. In addition, designating institutional repository self-archiving as the means of compliance for both funder and institutional mandates motivates institutions to adopt self-archiving mandates of their own, for all of their research output, in all disciplines, not just NIH- or WT-funded research. (Institutions are the universal providers of all published research, funded and unfunded.) Funder mandates designating institutional deposit make institutional and funder mandates convergent and mutually reinforcing -- rather than divergent and competitive, as funder mandates requiring direct institution-external deposit in PMC (instead of just automated harvesting or export from institutional repositories) do.

Effective Institutional Mandates Can Generate 100% OA Globally. The Liege model institutional ID/OA mandate really works. If funders and institutions worldwide collaborate, 100% OA can be reached not just for NIH and WT funded research but for all research.

Mike Taylor said...

I have to say I find it incomprehensible that any mandate's compliance rate is not 100%.

If I pay a merchant to deliver a new oven to my home and install it, but they just dump it outside my front door and walk away, then the very least sanction I will impose is never giving them my money again -- that's if I don't seek legal redress.

So if NIH or Wellcome pays someone to carry out a research programme and deposit the resulting papers in PubMed, and if they don't bother to do that, then the very least sanction the funder imposes should be to never give that group money again.

I don't see how this could be controversial. If you're paid to do something, you do it.

dave said...

Thanks for the follow up. This sounds like what many expected-- "NIH does not currently collect statistics on compliance efforts."

In my mind I was contemplating whether a FOIA request would do any good here, but it sounds like with the lack of record keeping would mean it would produce little additional information.

Alicia Wise @wisealic said...

Hi Richard,

Apologies for coming late to this discussion.

We estimate that more than 60% of manuscripts posted on PMC are published in journals where the publishers deposit manuscripts on behalf of authors.

* What percentage of NIH compliance comes from publisher deposit?

* What is the compliance level for authors depositing on their own?

Thanks for your insight,


Dr Alicia Wise
Director of Universal Access