Τετάρτη 22 Απριλίου 2015

Contextualizing SEGUE: Evaluating Residents’ Communication Skills Within the Framework of a Structured Medical Interview

Abstract

Background

The SEGUE (Set the stage, Elicit information, Give information, Understand the patient's perspective, and End the encounter) Framework is a checklist-style rating scale to facilitate the teaching and assessment of communication skills in medical learners. It has been used for over 15 years, and it is recommended in the Accreditation Council for Graduate Medical Education toolbox of assessment methods for resident training. When it was developed, its ability to provide objective scoring was a substantial improvement over global ratings.

Methods

In this article we describe the strengths and weaknesses of the SEGUE Framework. We highlight one residency program's experience with using the SEGUE Framework to evaluate residents' communication skills. Specifically, we cite previous studies and describe our own analysis of resident interviewing performance that demonstrates how the SEGUE Framework did not distinguish between different levels of interviewing skill level in our sample.

Results

Two case examples illustrate how the SEGUE Framework is not an ideal instrument to measure either the quality or the process of medical interviews.

Conclusion

Therefore, we propose a new method of contextualized assessment that builds on the SEGUE Framework. Our system evaluates discrete interviewing behaviors within the context of an ambulatory medical interview. We describe our interview structure, as well as a new instrument (the Wy-Mii, pronounced “why me”), to assess both communication and interpersonal skills. We expect that our new method of contextualized assessment will better differentiate between beginning and advanced levels of medical interviewing skills for residents.

Background

The Accreditation Council for Graduate Medical Education (ACGME) established 6 core competencies that residents and fellows are expected to demonstrate. Residency programs are expected to fairly and effectively evaluate residents' performance in each of these competency areas. The evaluation procedures are expected to be competency-based, valid and reliable, efficient, and relevant to the resident's medical practice.
One of the core competencies is interpersonal and communication skills (ICS). ICS involves “effective information exchange and teaming with patients, their families, and other health professionals.” In addition to the ACGME, many other governing medical bodies have also declared the importance of ICS for resident training. For example, the Institute of Medicine identified the importance of physician-patient interaction. The Federation of State Medical Boards and the National Board of Medical Examiners have also included a standardized patient component to assess communication and interviewing skills in the United States Medical Licensing Examination since 2004. Furthermore, the Medical Council of Canada has been evaluating residents' communication skills in their standardized patient examination for more than 10 years.
Although there is consensus that ICS is an important competency for residents to master, there has been little agreement in the literature about how these skills should be assessed. The ACGME website recommends 5 methods of assessment, including Objective Structured Clinical Examinations (OSCEs), standardized patients, 360-degree evaluations, patient surveys, and checklists. There have been several articles describing appropriate protocols for OSCE and standardized patient evaluations., Patient surveys and 360-degree evaluations are also a standard part of most hospital and business practices. Perhaps the biggest question that residency programs have entertained is how to use checklists to effectively measure resident interpersonal and communication skills.
One of the more popular instruments used by faculty to evaluate medical interviewing skills has been the SEGUE Framework. SEGUE stands for Set the stage, Elicit information, Give information, Understand the patient's perspective, and End the encounter. This checklist was originally designed to evaluate medical students, but it has been used for a broad range of medical learners for approximately 15 years. To support its use in evaluating residents, the ACGME has included the SEGUE Framework in its toolbox of assessment methods. When this instrument was developed many years ago, it offered an important method for evaluating medical communication; it focused on specific, objective communication “tasks” (eg, greet patient appropriately) rather than a subjective global rating system (eg, rate the resident's communication skill on a 1–5 scale). It employed a binary “yes/no” response scale to indicate whether the learner completed a particular communication task.
The SEGUE Framework has several demonstrated strengths and weaknesses. In addition to its longevity, other strengths include good internal consistency, good interrater reliability, and easy usability., It also reasonably adheres to both Kalamazoo Consensus Statements (KCS)., However, there is also a significant weakness of the SEGUE Framework. There have been several empirical studies demonstrating that instruments using binary response scales (yes/no), such as the SEGUE Framework, cannot effectively measure differences in medical interviewing skill.,
Because this seemed like an important weakness for evaluation in residency training, the faculty of the Wayne State University Family Medicine Residency Program questioned the validity of our own SEGUE scores. For many years we had been conducting OSCEs at the beginning of each academic year. These OSCE interviews were scored with the SEGUE Framework by medical school employees whose full-time job was to conduct OSCEs. These employees had been specifically trained to use the SEGUE Framework up to a 90% level of interrater reliability.
For program development purposes we decided to examine our SEGUE scores from 3 recent family medicine residency classes. There were 8 residents per class, for a total of 24 residents. Because our purpose was to evaluate discriminant validity, we followed the recommendation of Makoul that it was acceptable to use summary scores for psychometric evaluation. Therefore, we reviewed the SEGUE Framework summary scores for each resident's OSCE interviews from their first through third year of training.
We had hoped that use of the SEGUE Framework would detect improvements in medical interviewing skill over time during the residents' 3-year training. However, our results showed no discrimination; SEGUE was unable to detect a significant longitudinal improvement in communication scores in any of our residents. This was particularly surprising because qualitative ratings from our faculty preceptors had noted marked improvements in interviewing skills through our monthly video review. Even comments from community preceptors indicated that our residents were improving their communication and interpersonal skills.
We also compared communication skills between resident classes. Again, SEGUE summary scores were unable to detect differences between even our first- and third-year residents. Overall, our results were similar to other previously published studies; the SEGUE Framework seemed to be unable to identify differences in medical interviewing performance.
Because our preceptors were able to detect qualitative differences in interviewing skills between postgraduate year (PGY)-1, PGY-2, and PGY-3 residents, why was the SEGUE Framework unable to distinguish between residents' performance levels? The SEGUE Framework follows the KCS recommendations fairly closely, and it measures tasks that are obviously important for a medical interview (eg, explains rationale for diagnostic procedures). Would we not expect advanced residents to perform these important tasks more consistently than interns?
As previously stated, the SEGUE Framework assesses only the presence or absence of the interviewing tasks (yes/no). It does not measure quality, nor does it assess the context in which the “task” is applied. These may be shortcomings of the instrument. Perhaps it is not simply whether or not residents apply an interviewing task that best distinguishes beginning from advanced medical interviewers. It is possible that the primary difference in skill level lies in the quality, the timing, or the manner in which the SEGUE tasks are used in a medical interview. Maybe this is why global ratings have been more successful in distinguishing between beginning and advanced interviewers. Despite their subjectivity, global ratings are able to consider the interview context more effectively than a binary checklist such as the SEGUE Framework.
As one example, the SEGUE checklist does not effectively account for medical interviews in which the patient has multiple complaints. We will briefly consider Dr. Miller, a second-year family medicine resident who interviewed a patient with 2 chief complaints: hypertension and foot pain. Dr. Miller did an excellent job of eliciting the patient's ideas and concerns about his high blood pressure, but she did not inquire about any personal concerns about his foot pain. How would SEGUE item 6, “eliciting patient's view of health problem and/or progress (ideas/concerns),” be appropriately scored? Using the binary scale, the only scoring options are “yes” or “no.” According to the scoring rules, this item should be scored “yes” because the behavior occurred “at least one time during the encounter.” Therefore, Dr. Miller will receive full credit for this item, even though she did not obtain the patient's illness experience for every complaint. This result seems unsatisfactory; the SEGUE Framework renders a score that does not reflect the resident's whole performance during the interview.
Another weakness is that the SEGUE Framework does not allow for faculty to evaluate the process (or coherence) of a medical interview. As an example, we will briefly consider Dr. Lopez, a first-year internal medicine resident who is interviewing a patient with chronic back pain and an acute sinus infection. Dr. Lopez completes his interview for today's sinus infection, and he begins to inquire about the patient's back pain. In the middle of the back pain interview, he suddenly remembers that he did not ask a couple of key questions about the sinus infection. He politely, but without explanation, asks the sinus questions and then returns to his inquiry about the patient's back pain. According to SEGUE scoring protocol, this resident might be given a perfect score for this interview, even though abrupt changes like this are disruptive. Because the SEGUE Framework encourages (but does not require) a linear structure for the medical interview, there is no way for residency faculty to offer an appropriately lower score for this resident's abrupt change in topic. In fact, this resident could have changed topics 3 or 4 times without a scoring penalty as long as he included all of the required SEGUE tasks. This result also seems unsatisfactory, even if residency faculty were only using the SEGUE Framework for teaching purposes rather than for evaluation.
These case examples not only demonstrate weaknesses of the SEGUE Framework, but they also indicate an important need in the assessment of medical interviewing skills. They demonstrate the need to assess communication and interpersonal skills within the context of a structured medical interview. The interviewing tasks of the SEGUE Framework do not occur in a vacuum; they occur within a doctor-patient relationship, at a particular time, in a specific part of the medical interview. Only when faculty and residents use the same medical interviewing protocol can rating systems and feedback be useful. It does not make conceptual or practical sense to evaluate communication skills outside the interview structure.

Methods

When we reviewed the pros and cons of the SEGUE Framework, overall it seemed to be an incomplete tool for assessment of resident communication skills. Fifteen years ago, its development was a significant advancement toward objective rating of medical interviewing skills. Yet the current empirical data and our qualitative observations indicated that more work is still required. What is lacking is the opportunity for ratings of quality (how well each task was performed) within the context of a structured medical interview.
In order to find another instrument that might fill these gaps, we reviewed the ACGME's toolbox of assessment methods. We found that several of these instruments measure humanistic qualities or interpersonal skills without mentioning any content from the medical interview (eg, agenda setting, history of present illness). Three of the instruments were designed for patients to evaluate their doctors rather than for faculty observation, and 2 were coding systems seemingly developed for research purposes rather than resident evaluation. In addition to the SEGUE Framework, the only instrument in the ACGME toolbox that appeared to be viable for faculty rating of resident medical interviews was the Calgary-Cambridge Observation Guides (CCOG).
In order to examine the value of the CCOG and potentially identify other instruments, we reviewed the medical education literature. We discovered that in a recent article, Schirmer and colleagues had evaluated 15 of the most commonly used instruments for the assessment of interpersonal and communication skills. Some of the instruments included the Kalamazoo Communication Checklist, the CCOG, the Macy Model Checklist, and the SEGUE Framework. An expert panel had rated each instrument on several important characteristics, including consistency with the first KCS, psychometric properties, and usability/practicality.
The authors concluded that none of the instruments received high ratings in all of their categories. Those instruments that closely followed the KCS did not necessarily have strong psychometric properties or usability. Very few of the instruments measured interviewing with more than one family member in the room, and several instrument did not assess interview efficiency. This study concluded that the SEGUE Framework closely followed the KCS, but it did not assess the interviewing of family members or interview efficiency. Only the CCOG attempted to measure both the process and content of the medical interview. However, this instrument seemed to artificially separate the content of the medical interview (eg, history of present illness, diagnosis, management) from the process (eg, building relationship, gathering information). Two of its forms (“guides”) measure the process issues and the third form allows for comments on the traditional medical content. Furthermore, this instrument received the lowest usability score. The CCOG is probably not practical for everyday use in an ambulatory setting because of its length; it includes a total of 74 items plus the section for additional comments.

Results

Our review of the medical education literature and the ACGME toolbox did not yield any instruments that measure interpersonal and communication skills within the context of a structured ambulatory medical interview. Therefore, the faculty at the Wayne State University Family Medicine Residency Program decided to develop a model of medical interviewing that guides a resident through a patient encounter from start to finish. We followed recommendations from book chapters, empirical studies, and both KCS.,Additionally, the GME-Today web site (http://www.gme-today.com) may be helpful. Here is our recommended medical interview structure:
  1. Take time for self-reflection and/or self-care.
  2. Briefly review the patient's electronic medical record (EMR) to remind yourself of his/her last visit, to review the nurse's documentation, and to review the patient's vital signs.
  3. Introduce yourself appropriately to the identified patient and anyone else in the room. If others are present, be sure to understand their relationship to the identified patient and the reason(s) they are here.
  4. Elicit the chief complaint(s) with an open-ended question (eg, “What brings you in to see me today?” or “How can I help you today?”)
  5. Elicit any other complaints by asking the patient (or family) whether there are other issues to discuss during today's visit. Keep asking until there are no more issues, or you reach 5 agenda items.
  6. Add any medically necessary items to agenda (“We also need to discuss your blood pressure, which was high today.”)
  7. Set the agenda collaboratively with the patient/family. The doctor informs the patient if there is a medically necessary issue that must be addressed first (eg, high blood pressure or chest pain). Then the doctor asks which of the other issues the patient/family would like to address. Generally the agenda for a 20-minute visit should not include more than 2 issues (or 3 issues for an advanced resident).
  8. Explore the first agenda item collaboratively with the patient/family.
    • Start with open-ended questions and gather details through closed-ended questions.
    • Explore and clarify the patient's symptoms, past/current treatments, and related history for the first agenda item in an organized manner.
    • Inquire about the context (family, culture, gender, age, socioeconomic status, spirituality), and the patient's experience of his/her illness.
  9. Remind the patient about the second agenda item, and explore this issue in the same manner as the first agenda item.
  10. Screen for psychosocial problems (eg, domestic violence, depression, anxiety, childhood abuse, alcohol/drug use) if necessary, as prompted by the patient's history or behavioral presentation.
  11. Wash your hands, and talk the patient through the physical examination, if any.
    • Ask permission or explain what you are going to do before doing it (eg, “I am going to listen to your heart now, is that ok?”).
  12. Mentally develop a preliminary diagnosis, but only share information with the patient based on postgraduate year status (PGY).
    • First-year residents should obtain supervision before sharing information about diagnosis or treatment with the patient.
    • Advanced residents might give the patient some basic diagnostic information or treatment possibilities before precepting, depending on experience and medical knowledge.
  13. Briefly explain your need for supervision to the patient, leave the room, and present the case to a faculty physician, and psychologist, if necessary.
    • Present the case to faculty in an accurate, organized (and efficient for PGY-3) manner.
  14. Return to the patient's room and present the diagnoses (“Here is what I think is going on…”) in an organized, sensitive manner.
  15. Work collaboratively with the patient (ie, find common ground) in order to develop a treatment plan for each agenda item.
    • Check for patient understanding and agreement with the treatment plan, as well as potential barriers, and/or important side effects.
  16. Discuss follow-up and provide a rationale and time frame for the next visit.
  17. Complete the documentation in the patient's EMR immediately after the visit, if possible.
Following ACGME guidelines, we sought to develop a method of assessment that was consistent with our medical interview structure, curriculum, and available technology. The Wayne State Family Medicine Center is paperless; everything is organized through our EMR system. There is a digital camera system in every clinic room so that physician and psychologist faculty can observe resident performance remotely (with patient consent) rather than watching over the resident's shoulder. We wanted to design a rating scale that was compatible with this technology and incorporated our belief that communication skills should be evaluated in the context of the medial interview.
To meet these needs, we developed the Wayne State Medical Interviewing Inventory (Wy-Mii, pronounced “why me”), a 27-item instrument designed to assess residents' communication and interpersonal skills in the context of a real or standardized ambulatory medical interview. The first 22 items measure communication tasks, which are assessed within the context of our recommended medical interview structure (described earlier). Similar to the SEGUE Framework, faculty raters identify whether the resident completed these interviewing tasks, but they also provide a quality rating for each task on a 10-point scale ranging from 0 to 9. The scale includes the following descriptions of quality: 0 (Not Done), 1 to 3 (Poorly Done), 4 to 6 (Adequately Done), 7 to 9 (Well Done), and Not Applicable. We are currently developing a scoring manual to provide descriptions and examples for each item to enhance standardization and interrater reliability.
The last 5 items of the Wy-Mii measure interpersonal skills. The Kalamazoo II Report defined interpersonal skills as “relational and process-oriented.” Conceptually, these skills transcend the medical interview; they should be evident through the entire patient encounter, not just during certain parts of the interview or examination. Therefore, a 10-point scale ranging from 0 to 9 was developed to assess how consistently the resident applies these skills during the encounter: 0 (Absent), 1 to 3 (Not Consistently Applied), 4 to 6 (Consistently Applied), and 7 to 9 (Exceptional Application of this Skill). Additionally, for teaching purposes we recommend a section for comments after every item so that faculty raters can provide explicit rationale for their rating, as well as offer helpful feedback to the resident for future interviews.

Discussion

The Wayne State University Family Medicine Residency Program has been using the new interview format and the Wy-Mii instrument for approximately 6 months. During that time we have noticed several important changes that have occurred as a result of these new procedures. First, the faculty physicians have stated that this new interviewing structure makes it easier to teach and evaluate resident skills, including medical knowledge. At first, a couple of the physicians did not “buy in” to the new procedures. They did not oppose the new interview format, they simply did not enforce it when supervising residents' cases. However, after they became more familiar with the Wy-Mii structure, they realized that this interview format helped them identify resident deficiencies more rapidly. One physician even commented that using the Wy-Mii instrument helped to evaluate the residents' interviews because “it describes how a good interview should flow.” In retrospect, the program should have instituted a couple of faculty development sessions to train the faculty about the Wy-Mii and the new interview procedures.
Second, the new procedures have helped first-year residents to feel less overwhelmed by the prospect of learning interviewing skills. In the past, residents were expected to keep too many communication skills in mind, particularly with the pressure to learn new medical information and accurately treat their ambulatory patients. These new interviewing procedures have helped the faculty to focus on a smaller number of teaching points for each residency year. For example, in the first year we now focus primarily on introductions, agenda setting, and documentation with EMR.
Third, using the Wy-Mii instrument has increased the interest of our senior residents in learning more advanced communication skills. Once they mastered agenda setting, they began to spontaneously ask for guidance about skills such as understanding the patient's perspective of their illness and psychosocial screening. Additionally, learning the structure of the Wy-Mii form has also helped the senior residents identify and correct their own interviewing deficits.
Lastly, the implementation of these new interviewing procedures has increased the co-supervision of residents' cases by physicians and psychologists. The in-house psychologists were assigned to evaluate each resident's interviewing skills once per month by administering the Wy-Mii. Even though their presence was not required, the physician faculty have made a sincere effort to attend the evaluation sessions, particularly if the resident was their advisee. The faculty all agreed that the increased co-supervision has made the experience more robust and meaningful for the residents.

Conclusions

In summary, we have reviewed the pros and cons of the SEGUE Framework to evaluate interpersonal and communication skills. We highlighted how its measurement of discrete interviewing behaviors (tasks) was an important breakthrough 15 years ago, but that in today's learning environment, the SEGUE Framework is an incomplete tool for assessing resident communication and interpersonal skills. Research has shown the inability of the SEGUE Framework to differentiate between high and low levels of interviewing skill level, and our case examples demonstrated how it is not an ideal instrument to measure or teach medical interviewing to residents. At present, there seem to be no other instruments identified in the medical education literature or the ACGME toolbox that would fill these gaps.
Therefore, we proposed a new method of contextualized assessment that builds on the SEGUE Framework. We have developed the Wayne State Medical Interviewing Inventory (Wy-Mii), a rating scale designed to assess residents' communication and interpersonal skills in the context of an ambulatory medical interview. We have already begun to formally investigate its psychometric properties, as well as develop another brief instrument to assess resident's written communication skills through our EMR system. We expect that the contextualized assessment of medical interviewing skills, as well as the structured assessment of medical documentation, will better differentiate beginning from advanced levels of resident skill. Further research will be necessary to confirm this hypothesis.

References

1. Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties. General competencies: Minimal program requirement language. September 1999. 2009.http://www.acgme.org/outcome/comp/compMin.asp. Accessed July 8,
2. Lynch D. C., Swing S. R. Key Considerations for Selecting Assessment Instruments and Implementing Assessment Systems. 2009. Available at: http://www.acgme.org/outcome/assess/keyConsider.asp. Accessed July 8,
3. Institute of Medicine. Improving medical education: enhancing the behavioral and social science content of medical school curricula. Washington, DC:: Institute of Medicine; 2004.
4. Epstein R. M. Defining and assessing professional competence. JAMA. 2002;287(2):226–235. [PubMed]
5. Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties. Toolbox of Assessment Methods, version 1.1. 2000. Available at:http://www.acgme.org/Outcome/assess/ToolTable.pdf. Accessed July 8, 2009.
6. van der Vleuten C. M., Swanson D. B. Assessment of clinical skills with standardised patients: state of the art. Teach Learn Med. 1990;2(2):58–76.
7. Bergus G. R., Woodhead J. C., Kreiter C. D. Trained lay observers can reliably assess medical students’ communication skills. Med Educ. 2009;43(7):688–694. [PubMed]
8. Makoul G. The SEGUE Framework for teaching and assessing communication skills. Patient Educ Couns. 2001;45(1):23–34. [PubMed]
9. Schirmer J. M., Mauksch L., Lang F., et al. Assessing Communication Competence: A Review of Current Tools. Fam Med. 2005;37(3):184–192. [PubMed]
10. Brunett P. H., Campbell T. L., Cole-Kelly K., et al. Participants in the Bayer–Fetzer Conference on Patient–Physician Communication in Medical Education. Essential elements of communication in medical encounters: the Kalamazoo Consensus Statement. Acad Med. 2001;76(4):390–393. [PubMed]
11. Duffy F. D., Gordon G. H., Whelan G., et al. Participants in the American Academy on Physician and Patient’s Conference on Education and Evaluation of Competence in Communication and Interpersonal Skills Assessing competence in communication and interpersonal skills: The Kalamazoo II report. Acad Med. 2004;79(6):495–507. [PubMed]
12. Hodges B., Regehr G., McNaughton N., Tiberius R., Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999;74(10):1129–1134. [PubMed]
13. Hodges B., McIlroy J. H. Analytic global OSCE ratings are sensitive to level of training. Med Educ.2003;37(11):1012–1016. [PubMed]
14. Kurtz S., Silverman J., Benson J., Draper J. Marrying content and process in clinical method teaching: enhancing the Calgary-Cambridge guide. Acad Med. 2003;78(8):802–809. [PubMed]
15. Kalet A., Pugnaire M. P., Cole-Kelly K., et al. Teaching communication clerkships: models from the Macy initiative in health communications. Acad Med. 2004;79(6):511–520. [PubMed]
16. Bickley L. S., Szilagyi P. G. Bates’ guide to Physical Examination and History Taking. 2007:23–63. 9th ed. Philadelphia: Lippincott Williams, & Wilkins;
17. Rodriguez H. P., Anastario M. P., Frankel R. M., et al. Can teaching agenda-setting skills to physicians improve clinical interaction quality? A controlled intervention. 2010. Available at:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2245937/. Accessed February 9, [PMC free article][PubMed]
Articles from Journal of Graduate Medical Education are provided here courtesy of Accreditation Council for Graduate Medical Education
στις

Δεν υπάρχουν σχόλια:

Δημοσίευση σχολίου

Εγγραφή σε: Σχόλια ανάρτησης (Atom)