On this page · 9 sections
A study can be carefully conducted and still leave an important workplace question unanswered. A training study may measure knowledge, while a manager needs to know whether workers can recognize changing conditions during maintenance. A survey may describe employee perceptions, while the decision concerns whether a protective control performs as intended.
This article turns internal validity, external validity and peer review into questions safety professionals can use when reading research. The workplace examples and checklist are my practical interpretation of the cited methods literature. They are not a validated appraisal instrument or findings from a field study.
Start with the decision you need to make
Write the workplace question before judging the paper. Identify the people and task, the change being considered, the current alternative and the outcome that matters. Include the conditions under which the change must function. “Does this safety program work?” is too broad to guide a useful comparison.
A more specific question might be: would adding coached practice to an existing hazard-recognition lesson improve maintenance workers’ ability to identify changing conditions during the tasks they perform? That question distinguishes a learning outcome from an injury outcome. It also identifies a comparison: the existing lesson without the added practice.
Keep the purpose visible while reading. A descriptive study can help identify concerns or generate questions. An intervention study may help assess a proposed change. Neither should be expected to answer a different question simply because its title contains the same safety topic.
Separate internal validity from workplace applicability
Patino and Ferreira (2018) distinguish internal validity, concerning the credibility of findings within the study, from external validity, concerning their application beyond it. Their article addresses clinical research; the distinction is useful here as a methodological concept, not as evidence that a particular safety intervention works.
Ask two connected questions: does the study support its stated conclusion, and is that conclusion relevant to the people and conditions in your decision? Similarity to your workplace cannot repair a serious measurement or comparison problem. Strong methods also cannot make an unstudied outcome appear in the results.
| Question | What to inspect | Workplace example |
|---|---|---|
| Is the finding credible within the study? | Recruitment, measurement, comparison, missing data and other changes that could explain the result. | Did a training comparison also change supervision or staffing? |
| Does the finding apply to this decision? | Relevant people, tasks, conditions, delivery and outcomes. | Does recognition in a classroom exercise inform performance during the maintenance task being considered? |
Read the methods before extending the conclusion
Identify what the researchers actually did. Was this a cross-sectional survey, a before-and-after comparison, a controlled trial or a qualitative investigation? Check who participated, how they were selected, what was measured and when. Read the limitations alongside the results.
For a claim that a change caused an improvement, examine the basis for that comparison. Could a simultaneous equipment change, different workload, selective participation or altered reporting explain part of the difference? A before-and-after change alone leaves those possibilities open. Random assignment can strengthen a causal comparison, but does not remove every concern about measurement, missing responses or implementation.
Qualitative interviews can provide detailed accounts of work and possible explanations. A survey can describe associations or perceptions. Assess each study against its question and design. Do not turn a theme into a population percentage or an association into a demonstrated intervention effect.
Compare the conditions that could change the result
Lesko and colleagues (2020) explain why the target population and the action being considered matter. A result may differ when factors influencing the effect, the version of an intervention or interactions between people differ across settings. Their framework concerns causal inference in epidemiology; applying it to the following workplace questions is my interpretation.
Compare meaningful conditions rather than matching an industry label. Two manufacturers may use different equipment, languages, employment arrangements and maintenance practices. A lesson delivered by a researcher with repeated coaching may differ from a short recording viewed alone. Those differences deserve investigation when they could affect the intended result.
Record what matches, what differs and what is unknown. Explain why a difference matters to the decision. A different location does not automatically make a paper irrelevant, and a familiar job title does not automatically make its findings transferable.
Keep the outcome and follow-up intact
Write down the measured outcome in the researchers’ terms. A knowledge score, observed task performance, reported safety climate, exposure measurement and injury record answer different questions. If a paper measured knowledge immediately after instruction, describe that finding without converting it into a lasting reduction in injuries.
Check the unit and denominator behind a result: workers, tasks, observations, locations or hours. Identify the follow-up period and how uncertainty was reported. Avoid treating repeated observations of a small group as though each observation came from a different worker. A precise-looking percentage is difficult to interpret when the underlying count or comparison is unclear.
The companion article What Safety Audit Scores Can—and Cannot—Tell You applies this same discipline to rating criteria and aggregate scores. In both cases, the number needs its evidence and boundaries.
Worked example: a training result and a maintenance decision
This is an original fictional example. It does not describe a published study, an actual employer or a verified improvement. Suppose a hypothetical report follows 100 volunteer production workers. Forty meet a hazard-recognition criterion before a revised lesson and 60 meet it three weeks afterward. The revision adds both a lesson and supervisor coaching, with no separate comparison group.
The observed proportion increases from 40% to 60%: a 20-percentage-point increase, or a 50% increase relative to the original 40%. Both descriptions use the same invented counts. Neither establishes that injuries fell by 20% or 50%, and neither isolates the effect of the lesson from coaching or other changes.
| Evidence in the example | Supported description | Question still open |
|---|---|---|
| 40 of 100 before; 60 of 100 afterward. | More participants met the stated recognition criterion at follow-up. | How consistent and meaningful was the measurement? |
| Lesson and coaching changed together. | The observed change followed the combined arrangement. | What caused the change, and which component contributed? |
| Volunteer production workers; three-week follow-up. | The finding describes those participants and that period. | Would it apply to maintenance, contractors, other shifts or longer periods? |
| No injury or exposure outcome was measured. | The example provides no measured injury or exposure effect. | What evidence would address the intended protective outcome? |
A maintenance manager could use this report to develop a question about coached practice. Before adopting the approach, the manager would need to examine task differences, involve affected workers and determine how task performance would be evaluated. The example does not justify replacing required controls or claiming that the lesson has a demonstrated injury-prevention effect.
Use peer review as one quality check
Drozdz and Ladomery (2024) describe peer review as a quality-control process and discuss limitations, including bias and inconsistent review. Peer review can identify weaknesses and improve a manuscript. It does not guarantee that every finding is correct or that an intervention will work in another workplace.
Check the publication’s actual review process, author disclosures and any corrections associated with the paper. A journal’s reputation or an author’s credentials can help establish context, but the methods and evidence still need examination. A list of references does not establish that every claim accurately represents those sources.
The current Jenkins Industry Safety Journal collection is not peer reviewed. This applied methods guide was developed and edited with AI assistance. Its fictional example and checklist have not undergone independent validation or peer review.
Document an applicability judgment before acting
Use the questions below to make the reasoning visible. This is an author-developed discussion aid, not a numerical scoring system. Do not total the answers into a percentage of validity or treat a majority of favourable answers as approval.
| Record | Question to answer |
|---|---|
| Decision | What specific change, people, task and intended outcome are under consideration? |
| Finding | What did the study actually measure, over what period and with what uncertainty? |
| Credibility | What design, measurement or comparison issues could change the interpretation? |
| Relevant conditions | Which important study conditions match, differ or remain unknown here? |
| Delivery | Can the intervention and its supporting resources be delivered as studied? |
| Evidence gaps | What further research, worker input or technical assessment is needed? |
| Verification | What evidence would show that the proposed change functions in this work? |
| Decision record | Who decides, on what evidence, with which limitations and review triggers? |
A defensible record might state: “The paper supports investigating coached hazard-recognition practice. Its participants, task and short follow-up differ from our maintenance work. It provides no measured injury effect. Further task assessment and a defined evaluation are needed before making an effectiveness claim.” That statement preserves useful evidence while identifying what it cannot yet answer.
If a change proceeds, define responsibilities, resources and checks for its intended result. OSHA’s hazard-prevention and control guidance recommends planning how control effectiveness will be verified after implementation. The related Safety Management System Action Plan connects those checks with decision authority and follow-up.
Research can inform a decision without supplying all the evidence needed to implement it. Seek appropriate technical assessment for consequential gaps, retain necessary protections and revisit the decision when work conditions or evidence change. Local evaluation should examine the proposed change without exposing people to uncontrolled hazards.
References and source record
The methods papers below inform the conceptual discussion; they do not report tests of the fictional safety-training example. This is a focused explanatory article, not a systematic review of safety-intervention research.
- Patino, C. M., & Ferreira, J. C. (2018). Internal and external validity: Can you apply research study results to your patients? Jornal Brasileiro de Pneumologia, 44(3), 183.
- Lesko, C. R., Ackerman, B., Webster-Clark, M., & Edwards, J. K. (2020). Target validity: Bringing treatment of external validity in line with internal validity. Current Epidemiology Reports, 7(3), 117–124.
- Drozdz, J. A., & Ladomery, M. R. (2024). The peer review process: Past, present, and future. British Journal of Biomedical Science, 81, 12054.
- Occupational Safety and Health Administration. (n.d.). Hazard prevention and control. Recommended Practices for Safety and Health Programs.