3 Tips to Conducting an Effective CME Activity Evaluation

The accuracy of your CME activity evaluation is critical in the development of your ACCME Self-Study report. The integrity of your data determines the validity of your assessments—and ultimately influences your ability to win accreditation or reaccreditation. In this post, we’ll offer three tips for conducting an effective CME activity evaluation.

Master ACCME Standards for CME Curriculum Development

As a CE coordinator, you work hard to help your association achieve its mission of providing CME to professionals as a way of improving the overall quality of healthcare in their communities. This matches the ACCME mission, and every aspect of your program is most likely designed around improving learner outcomes and maintain a high-quality, relevant curriculum.

Regardless of when you received your accreditation (or if you are in the midst of a first Self-Study), it’s smart to review curriculum standards as you develop new activities and evaluate existing ones. Here is a quick review of some of the curriculum criterion that the ACCME requires your program to meet or maintain compliance:

Criterion 2. Your CME activities should address the unique educational needs (defined as gaps in knowledge) of your learners.

Criterion 5. The educational formats that you choose for activities should incorporate well with their setting and advance the objectives of the activity.

Criterion 6. The activities that you create should be designed to develop “desirable physician attributes [eg, Institute of Medicine (IOM) competencies, Accreditation Council for Graduate Medical Education (ACGME) Competencies].”

Criterion 7. Activities should be created independently of commercial interests. (SCS 1, 2, and 6).

It is critical that every step of your curriculum development process is well-documented, and that activities—from completion rates to knowledge retention evidence—are constantly monitored.

That’s because your achievements in meeting ACCME compliance criterion aren’t really “there” unless they’re demonstrable with data. That means truly comprehensive data which reflects the positive changes that your CME programming has created with respect to professional knowledge gaps. If you are using an all-in-one CME LMS like EthosCE, then you won’t have to toil with documentation—everything from course completion rates to PARS submissions is managed from a single dashboard. If you’re on your own, you’ll need to be especially attentive to the data collection process, in addition to, properly segmenting the data that you are receiving.

Always Verify Activities Data Collection Standards.

The goal of every activity is to address professional knowledge gaps and enhance knowledge retention. In order for your program’s achievements in these two areas to be demonstrable in your Self-Study, you’ll need to have a ready supply of fresh, “clean” data which gives evidence of your program’s success in meeting its benchmarks. Here are three questions to ask when looking at the way data is collected for activity evaluation:

1. Are activities monitored continuously or at intervals?
   Interruptions in activities monitoring can mean that important shifts in learner knowledge retention or course completion rates— may be missed. This data may hold valuable insights that can help you improve your programming and if they aren’t included in your evaluations then your findings may be inaccurate. Make sure that your LMS offers seamless activities monitoring to avoid gaps in data.
2. Is activity reporting comprehensive?
   If your LMS provides constant activities monitoring, it’s important that it also provides access to a range of information—such as course completion rates, learner outcomes, and learner satisfaction data. The ACCME requires a full, accurate portrait of your program’s educational impact, and incomplete reports—those with only learner satisfaction statistics and not learner outcome data— are not considered compliant. Verify that the data your LMS collects includes all of the metrics that matter.
3. Can activities data be contaminated through human error?
   When your numbers are off, even slightly, your projections and your assessments will be too, and by extension, they won’t bear scrutiny when they’re reviewed. Missing or inaccurate data is the kind of Kryptonite that no CME team—not even with your staff’s superpowers— can keep from undoing months of hard work. Choose an LMS such as EthosCE that automates data collection and prevents the kind of unseen errors that can ruin even the most meticulously prepared report.

If you are handling data collection manually, then one of your most challenging tasks is slogging through all of that information and determining which metrics are the most important to your evaluations.

Know Which Metrics Matter

When it comes to explaining which metrics matter, the ACCME is exceedingly clear in its explanation of Criterion 11:

The provider is asked to analyze the overall changes in competence, performance, or patient outcomes facilitated by their CME program using data and information from each CME activity. Providers who only measure the change in knowledge in all their activities will not have any data on change in competence, performance, or patient outcomes to analyze.

That means that your activities data reporting must be focused on learner outcomes with reference to specific knowledge retention goals. The learner’s entire journey from the start of the activity to the finish should be evaluated. That means looking data which includes at:

• Improvements in knowledge retention over time (quizzes and test scores)
• Learner satisfaction data
• Expected outcomes in relation to documented outcomes
• This data must be provided for each CME activity.

Staying on track with data monitoring and activities evaluations is an important part of your ACCME compliance strategy. If compiling and analyzing data becomes a time and resource-intensive undertaking, then you’ll have fewer opportunities to support the development of your curriculum. That means the growth of your program—and the quality of educational components—may be compromised simply because there’s no time to implement needed improvements.

Seem overwhelming? It doesn’t have to be. Choosing the EthosCE LMS gives you access to a platform handles data collection, metrics, and PARS-compliant reporting while making it easy to develop a curriculum that meets ACCME standards.

If you’re interested in frictionless activities data collection and evaluation through your LMS, we’ve got your solution ready. Contact us here to learn more.