Skip to main content

Rigor and Reproducibility [NIH Guideline]

Rigor and Reproducibility

Purpose: NIH research grant and mentored career development award applications must address rigor and transparency requirements outlined in the application instructions. Research Performance Progress Reports (RPPR) must emphasize rigorous approaches to ensure robust and unbiased results.

A. Eight steps to Rigorous and Reproducible Experiments in Biomolecular Research at UNC

  1. If using a core facility, consult with the core staff in the planning stage. Consult with a statistician if you need help developing a Power Analysis to assure that your results will be adequately powered.
  2. Design your experiment with sufficient controls (rigor) and replicates (reproducibility).
  3. Assure that ALL of your reagents (antibodies, cell lines, mice) are fully validated (see below).
  4. Have a clear and detailed protocol (SOP) and data analysis plan. Assure that the protocol is strictly followed or that any deviation is well documented.
  5. Assure that the staff or students performing the experiment are well trained and understand each step and the importance of performing them precisely.
  6. Use only well-maintained instrumentation, preferably maintained and operated in a core facility with expert staff (see #1 above).
  7. Document all steps, reagents, equipment and data analysis methods used in the experiment. Assure that the both the documentation and the data itself are properly stored in a safe data management repository.
  8. Acknowledge the Cancer Center Support Grant (P30 CA016086) (if applicable), other grants that support the core, the core (by name), and core staff in publication

B. Guide to Rigor and Reproducibility for UNC Chemistry Department X-ray Core Laboratory

  1. Consult with the Core Staff in the planning stage.
  2. XCL provides SC-XRD structural determination which requires compound in the form of single crystals.  Single crystals samples must be provided with some solvent remained in the vials or isolated/protected in the mineral oil (Paratone N). It is very common that solvents used for recrystallization to co-crystallize with the desired compound.  A dry sample may enable solvent molecules to escape from the crystal, therefore breaking crystals apart and losing crystallinity.  Additionally, compounds that are extremely sensitive to air and moisture must be transported in dry ice to reduce the rate of decomposition.
  3. All reagents used at XCL are properly labeled with chemical grades and are fully compliant with OSHA standards. 
  4. Some of the most commonly used recrystallization methods include slow evaporation, slow cooling, and vapor diffusion.  Proper recrystallization method allows quality crystals to be produced that are suitable to the X-ray diffraction experiments.
  5. All independent users must be approved by the laboratory staff to ensure proper and safety experiments conducted at the XCL.
  6. XCL houses a single crystal X-ray diffractometer, Bruker SMART ApexII, which is properly maintained by the XCL director on a regular basis.  Maintenance includes calibration of the instruments and cold stream, cooling fluid for the CCD detector chiller, as well as Haskris filter change for cooling the X-ray generator.
  7. A standard X-ray diffraction experiment is conducted by mounting a single crystal onto a MiTeGen pin and then onto the goniometer.  The crystal is protected under appropriate temperature that prevents the sample from decomposition.  3 sets (orthogonal from one another) of 12 frames are initially collected, allowing unit cell to be calculated using the reflections obtained from these frames.   A full data is then collected based on the information of unit cell parameters and diffraction intensity (associated with crystal size and quality).  Once the data collection is complete, the reflections are consolidated and their intensities corrected.  Space group is determined based on the crystal systems, diffraction intensity distribution, and systematic absences.  A structure connectivity is determined via various structure solution methods available, and refined to publishable quality.  Successful structural refinement yields Crystallographic Information File (CIF), structure factor file (FCF), and a written report for publication purposes.   All saved data are securely stored in a safe data management repository.
  8. Please provide acknowledgement to the Cancer Center Support Grant (P30 CA016086) in publications.  For Core director and staff acknowledgement, please follow Laboratory Policies.

C. Additional Resources

Learn about the NIH Initiative to Enhance Reproducibility through Rigor and Transparency. (Video)

Resource Authentication Plan: https://grants.nih.gov/reproducibility/faqs.htm#V

What Kind of Information Should I Include in My Application’s Resource Authentication Plan? Check out instructions on NIH Nexus Blog.

What are ‘Key Biological and/or Chemical Resources’ that should be addressed your application’s authentication plan? Key biological and/or chemical resources include, but are not limited to, cell lines, specialty chemicals, antibodies and other biologics. More on NIH website

FASEB report on enhancing research reproducibility identifies three main gaps to research reproducibility:

  • Lack of uniform definitions to describe the problem
  • Insufficient reporting of key experimental details
  • Gaps in scientific training

Recommendations for research using antibodies (page 9-10) ‘Although vendor-supplied technical information may help investigators select reagents such as antibodies, this information is insufficient for validation’.