New CSA Scoring System “Item Response Theory” to be Tested in September
posted in Alerts by Brian Gray
New CSA Scoring System “Item Response Theory” to be Tested in September
FMCSA will begin testing a new scoring system for its Compliance, Safety, Accountability program now that they filed a report with Congress outlining changes to the Safety Measurement System (SMS).
Use of Item Response Theory (IRT) approach to identifying at-risk carriers for intervention. FMCSA hopes to run a larger test in April and have further evaluated the full program by June 2019.
“While this type of model has not been widely used within the transportation industry, it may provide [FMCSA] with tools to make decisions about the importance of different variables within SMS, such as which violations to include in a BASIC, severity weights and others,” NAS wrote in its initial report.
In addition, it will work with states and carriers to better calculate data on vehicle miles traveled and crashes. It is all part of the larger goal to add a larger scoring system, rather than comparing similar carriers with peers as the CSA program previously had done.
In July 2017, the National Academies of Science studied ways to make the rankings more accurate in the assessment of fleets’ risk of future crashes.
The chief criticisms of the CSA program from the report include:
- Reconfigure the SMS statistical model (percentile ranking) with an “item response theory” (IRT) model that more accurately targets at-risk carriers
- Make the scoring system more transparent and easier for carriers to understand and replicate
- Depart from using relative metrics as the sole means for targeting carriers
- Better collaborate with state partners and other data providers to collect higher quality data, such as miles traveled and number of power units
FMCSA expects the new system will allow carriers to more easily understand and calculate their safety scores.
One former Police and roadside DOT Officer wrote in a comment that he “responded to many crashes. For the most part, it was pretty easy to determine violation (or fault) and assign the most responsible party as Unit 1 and the not responsible party as Unit 2. Police Officers can determine violation at a crash scene but they may not be aware of any FMCSA/DOT issues.”
He went on saying that, “CSA should recognize this nationwide system of determining violation/fault and consider that before placing a crash on the company’s BASIC score. While the newer method of being able to Data Q is at least better, it has not been and continues to not be fair to carriers and drivers. If the accident lists the carrier as Unit 2 and shows no violations, it should not be included in the BASIC score.”
If you need any help challenging a violation through the DataQ process, give Glostone a call at 503-607-1088.
