NTDB Risk Adjustment Symposium

Dear Colleagues,

We have reserved time on the busy ACS/COT schedule for a “Symposium on Risk Adjustment”.  As with our previous symposium on missing data and imputation, this session is intended to produce an exchange of ideas and opinions among colleagues with interest and experience in this area, including involved statistical consultants.  In this way, NTDB Subcommittee members and staff can obtain valuable advice to help with our data analyses and the resources we can provide to contributing institutions, the COT, and the wider community of injury researchers.

On this occasion, our group will include two specially-invited participants:  Dr. Howard Champion, whose well-known leadership in the development of MTOS and TRISS will give unique perspective to our current efforts; and Dr. Turner Osler, whose innovative ideas about risk-adjustment have led to a recent grant from the AHRQ to apply these methods to NTDB hospitals.

A symposium, by definition, should be a relatively unstructured affair (the Greek roots mean “drinking party”).  However, I would appreciate your looking over the following pages and considering your responses to the assertions and questions they contain.  Anyone who would like to make a more formal 5- or 10-minute presentation at some point will be welcome to do so.  Otherwise, I will lead us through an informal but sober and goal-oriented discussion based on the outline below.

We look forward to having you join us!

David Clark
Chairman, NTDB Subcommittee

GENERAL POLICIES

• Methods and terminology should be consistent with standard clinical epidemiology.
• Methods should be compatible with standard injury categorizations.
• Methods should be replicable by outside researchers.
• Modeling concepts should be understandable by clinicians or public health professionals with basic statistical training (e.g., MPH).
• Analysts proposing new models should be blinded to identity of subjects.
• Models should be selected for their ability to explain as well as predict.
• A variety of outcome analyses should be provided to institutions and regulators.
• Hospitals not in the reference database should still be able to use its models to evaluate their own performance.

INCLUSION/EXCLUSION AND "MESSY DATA" ISSUES

Which cases should be excluded?

Transfers out

to other acute-care hospitals
to SNF, rehab, psych, etc.

Short stays (<2 days?, <3 days?)

only if sent home?
only if not dead in hospital?

Populations inconsistently included

DOA – how defined?
Hip fractures in the elderly
Low falls
Low severity (AIS<3, ISS<9?)
Burns, drownings, suffocation, poisoning

Which institutions should be excluded?

Low volume of cases (<1 per week?)
Low volume of serious cases
High proportion of cases with missing or invalid data (%?)

What periods of observation should be excluded?

Not contemporary (more than 1 year, 5 years?)
Not admitted

What to do with missing or invalid data (if institution included)?

Rational substitution (e.g., GCSmotor=6 if GCS=15)
Imputation (multiple, hot-deck?)
Publish both provided and imputed values
Analyses with/without imputation

What to do with calculated variables provided by institution?

Recalculate if possible, algorithms which to use if they disagree
Publish both provided and recalculated values

DATA MANAGEMENT ISSUES

What outcomes to measure?

Death in hospital

(exclude transfers out?)
(early vs. late, hazard rates?)

Complications (which?)
Functional status at discharge
Length of stay

What anatomic categories or scores to utilize?

AIS provided by institutions (version?)
AIS recalculated from ICD-9 provided by institutions (ICDMAP-90, other?)
ISS calculated from one of the above
Worst AIS, overall or by body region, derived from one of the above
ICISS-type (worst “SRR”, 2 worst “SRRs”, all “SRRs”, which reference?)
Others (Anatomic Profile, NISS, HARM, etc. etc.)
New “empiric” models with multiple categories

What physiologic categories or scores to utilize?           

GCSmotor (ED vs. scene vs. “best”)
BP (ED vs. scene)
Avoid those often unmeasurable (GCSverbal, Resp Rate)
Is “intubation status” a measure of physiology or treatment?

What host factors to distinguish?

Age provided by institutions or recalculated
Age functions or groups
Comorbidity categories or scores (Charlson, Elixhauser)

What mechanisms to distinguish (if not excluded)?

Blunt/Penetrating, or more detailed
CDC categories
“Low falls”?

MODELING ISSUES

What subpopulations should be analyzed separately?

Patient-level:  Mechanism, age, etc. (see potential exclusion criteria)
Institution-level:  Designation/verification status, size, etc.

One big model or multiple smaller models?

Different models by subpopulation
Indicator terms for subpopulation
Indicator terms with selected interactions by subpopulation
Point/interval estimation after combining multiple model

What to do about nonlinear continuous predictors?

Categorization
Splines, fractional polynomials, etc.
Centering (to hospital mean, or grand mean?)

Should hospital effects be random or fixed?

Should random intercepts include hospital-level factors?
Random slope(s)?
Modeling methods, software         

Summary statistics

Difference measures (e.g., “W statistic”, Risk Difference)
Ratio measures (odds vs. probabilities, O/E, “risk-adjusted mortality”)
Logarithmic transformations of ratio measures?
Stratification/standardization
Report based on less frequent binary outcome (mortality, not survival)
Labeling “outliers”

Trauma Programs