Ohio Turnpike and Infrastructure Commission
[Stay on top of transportation news: Get TTNews in your inbox.]
In 2017, after nearly a year of rigorous study and expert testimony, a National Academy of Sciences academic panel found what members believed was a better way for federal regulators to separate safe from high-risk motor carriers. It was known as the Item Response Theory.
It took nearly six years of exhaustive research and evaluation by the Federal Motor Carrier Safety Administration to reject the complex data-driven method primarily for one reason: The agency believed the method was too complex for truckers to understand.
In February, FMCSA said it was instead proposing to make improvements in the agency’s safety measurement system based on percentile ratings that ultimately are guided by the Compliance, Safety, Accountability system. CSA has been the target of criticism by some motor carriers since it was first introduced in 2010.
A sampling of early reaction from a number of transportation stakeholders showed that some in the industry agree with the regulators, but some data experts, including members of the academic panel and longtime trucking data guru, Steve Bryan, view the IRT method’s rejection as a lost opportunity.
However, the agency decision, according to safety executive Brandon Wiseman, blocked the adoption of a system that was a potential “monster” for trucking.
“It’s just pretty unwieldy from what I can see,” said Wiseman, a trucking attorney and owner of Trucksafe Consulting. “I understand it works pretty well for other industries, but I believe FMCSA when they say it’s not going to work for us.”
“IRT showed promise in better identifying carriers for crash risk,” said Dan Horvath, vice president of safety policy for American Trucking Associations. “But as with everything, details matter, and FMCSA identified several concerns related to IRT, the primary being its complexity. I think it’s important to be able to explain any methodology in a simplistic way, and FMCSA indicated this was not possible.”
“From an academic aspect, IRT had a lot of potential,” said Sean Garney, co-director at Scopelitis Transportation Consulting. “But I think FMCSA probably made the right decision because it requires a lot of data, some of which FMCSA couldn’t get. Plus, it’s extremely difficult to understand. We already have challenges describing CSA.”
In a combined statement, three members of the Academy of Sciences academic panel that studied FMCSA’s past rating system called FMCSA’s tweaks to the SMS system a “positive step.”
The three members, Joel Greenhouse, Sharon-Lise Normand and Michael Cohen, said: “However, by not fully implementing the IRT approach the FMCSA does not benefit from the structure that the IRT model would provide that systematically and transparently combines the individual data analytic inputs into a coherent system, accounts for the accumulated uncertainty in the data, and facilitates principled, scientifically based decision-making.
“It is unfortunate that FMCSA decided not to take advantage of the abundant expertise in the use of IRT modeling. As noted in our report these models have been successfully implemented in equally complex settings such as in health care and education. By deciding to do the IRT modeling in-house, FMCSA did not benefit from the wisdom and experience of experts who have successfully developed these models and have a keen understanding of both the implementation and computational challenges in the use of IRT models in practice.”
“I did see the Federal Register notice, and I understand the practical challenges they list, in particular that IRT is not readily understandable by the general public and the runtime for the model is excessively long,” said Brenda Lantz, a committee member and program co-director of the Upper Great Plains Transportation Institute at North Dakota State University. “I was pleased to see that they did gain some insights during the testing of IRT that helped lead to their proposed changes that I believe will improve SMS.”
“I was a fan of IRT,” said Bryan, CEO of Bluewire, a company that helps motor carriers protect their reputation and defend against frivolous lawsuits. “I thought IRT was an excellent solution. When I was at SambaSafety we built an IRT model, and it worked.”
He added, “Certain people at FMCSA didn’t understand the science behind IRT. So they just didn’t think they could explain it to the truckers. They must have told us this 50 times. What I’m afraid they’ve done is fix the complaints. But just fixing the complaints is not a new scientific approach to this problem.”
Bryan said that most don’t understand the science behind credit scores, IQ scores, or how educators compute grades.
“All of those are IRT models. Nobody explains to a person how they compute their IQ score,” Bryan added. “They just trust it because it’s good science.”
