AETC Cross-Region Evaluation

Date of Report: 03/2010
Source: AETC National Evaluation Center

In fall 2007, the AETC National Evaluation Center (NEC) began facilitating a process to develop a set of standard evaluation questions for use by regional AETCs. Collaboratively, regional evaluators developed four quantitative evaluation questions to measure learners' ratings of the quality of training, applicability of training content, knowledge prior to training, and knowledge immediately after training. Most regional AETCs incorporated the four immediate post-training questions into their evaluations of level 1, 2, and 3 trainings. And, because more intensive levels of training require additional evaluation, evaluators also worked to enhance measurement strategies for level 2 and 3 trainings.

The Behavior Change Pilot (Pilot) evaluation resulted from these discussions, which focused on how to measure practice changes trainees made as a result of AETC training. The project was based on a successful model used by the Midwest AETC (MATEC), which included an online open-ended questionnaire that gathered information and examples on how trainees made changes to their clinical practice as a result of training. To test whether this approach would work in other regions, the Northwest, Pacific, and Texas/Oklahoma AETCs worked with the AETC NEC to develop a similar online open-ended questionnaire.

In November 2008, the AETC NEC launched its website ( http://aetcnec.ucsf.edu/ ), which includes an AETC Only section featuring survey building tools. The open-ended Pilot questionnaire was programmed and administered through the website. Local Performance Site (LPS) staff and/or regional evaluators were asked to collect the email addresses of trainees, notify trainees that they would be receiving an email survey in six to eight weeks, and log the training event along with the email addresses into the AETC NEC website's survey center.

As of December 2009, over 200 AETC trainings have been evaluated with the Pilot online questionnaire. AETC trainees were able to express the following types of changes made as a result of AETC training programs:

	changes in attitudes toward HIV patients,
	changes in attitudes on how to conduct patient/client care,
	improved diagnostic and patient education skills, and
	changes in clinic policies and procedures.

Trainees noted several aspects about the training (e.g., useful training materials) and their workplace (e.g., opportunity to apply information learned) that helped facilitate change. Some of the obstacles trainees mentioned were the lack of time to implement the information learned, unsupportive colleagues, and the costs associated with some of the training recommendations (e.g., cost of labs). Trainees who did not apply the information learned cited lack of time and the fact that they did not currently provide care to patients.

An aim of the Pilot was to generate information upon which quantitative questions could be created and used by regional AETCs. A quantitative online questionnaire, with answer options based on the responses previously provided by trainees, is currently used in place of the open-ended online questionnaire. For more information or to participate in the follow-up project entitled AETC Cross-Regional Evaluation (ACRE), email Kevin Khamarko at kevin.khamarko@ucsf.edu .