Three Steps To Improve How You Measure Patient Outcomes
Simple steps to improve a surprisingly difficult to collect data point.
Survival outcomes are perhaps the most important metric to measure in most Global Health treatment centers.
How do we know how effectively we are delivering care if we cannot measure the outcomes? If 100% survival is the target, we have to know how far we are from it to improve our accuracy.
Unfortunately, survival outcomes are very difficult to measure accurately! As a pediatric hematologist-oncologist who supports monitoring and evaluation for treatment programs in Africa, I know this all too well. There is one primary reason why, even in clinical trials, these are so difficult to measure:
Survival outcomes change over time
An alive person can relapse or their disease can progress at any time. A patient may die, but the treatment program is unaware of the event. Patients may suffer financial or social hardships during treatment and stop attending clinic visits completely.
There are many reasons these data may not be captured well:
Insufficient staffing to actively collect data
No ability to efficiently identify patient statuses that need to be updated
Inconsistent variable definitions for disease status (remission/relapse/etc.) and patient status (alive/dead/abandoned treatment/etc.)
Data storage is confusing with repeated observations on the same unit over time
Thankfully, all of these problems have sustainable solutions if you design a system that respects the nuances of time-varying data.
Let me share three essential steps you can take to improve your data quality.
These are minimal recommendations for general program monitoring and evaluation. Realize if you’re doing research, you need this and more to assure accuracy.
Step 1: Prioritize accurate data collection in the organization.
It is easy to produce survival numbers (60% 5-year OS; see I just did it), but it is exquisitely difficult to produce accurate survival numbers. To do it, your entire Global Health organization has to understand the value of data management and commit to doing it well.
That means people with the proper tools have to be given time and support to create frameworks and workflows to record the outcomes over time. It also means the leadership has to be willing to commit enough money and resources to make this happen.
Step 2: Organize the people, tools, frameworks, and workflows to optimize data collection
- People - Depending on the size of the program, the number of patients to be followed, and the frequency of follow-up, you need enough people with dedicated time to do it. Some patient statuses can be captured when they come to the clinic, some have to be called on the phone or tracked down another way. Estimate how long it takes to update 10 statuses for each scenario, apply that rate (total time/10 patients in the scenarios) to the total number of patients to be followed, and estimated the number of person-hours required per month.
- Tools - Find a data recording tool that handles repeated observations on the same unit easily and that allows strict enforcement of variable options and date conventions. REDCap is a nice, free, and popular option that does this well. Spreadsheets can do it, but they are limited. Other fancier tools exist that you can explore. The key is to have a clean picture of repeated values for each patient arranged and to have control over data entry.
- Frameworks - To measure “outcomes”, create two variables: “Disease status” and “Patient status”. Define each in a few categories, and maintain a data dictionary that explicitly defines each category. Share the dictionary with all people collecting data and refer to it often. Also at a minimum have a variable for “date of change in status” to capture the event time.
- Workflows - Create a schedule for when patients in different phases of therapy need status updates. For example, patients actively on treatment need status updates every month, within two years of the end of treatment every quarter, and yearly after that. Now figure out a way to automate it, so that you can run a report each month that shows you who needs updates. I wrote code (#rstats!) to do it for my organization. There are software tools that can do it as well. Or you can see if you teach Excel to do it, I’m sure it can be done with macros.
Step 3: Check the accuracy of the data over time
Another person, perhaps a medical provider or a data manager, should conduct periodic reviews of the data. Biopsy a certain percentage of the medical charts, say 5-10%, for each quarter and compare what is in the chart or what is confirmed via a patient phone call with what is recorded in the system. Categorize types of errors into classification (e.g. “relapse” should have been classified as “progressed”), date (dates recorded incorrectly), or missed event (an event happened but not recorded). Study the pattern or errors and tune the elements in Step 2 above to reduce these errors.
Survival outcomes are hard to measure accurately but it can be done!
With organizational support, with the right people, tools, frameworks, and workflows, and with accuracy checks over time, your program can produce high-quality data that you can use to improve the care you provide to your patients!