Simulation Training and Skill Assessment in Anesthesiology

Alexander Rothkrug; Sohail Mahboobi

Simulation Training and Skill Assessment in Anesthesiology

Free Review Questions

Introduction

The anesthesia patient safety foundation (APSF) was founded in 1985 with the timeless mission to ensure that “no patient shall be harmed by anesthesia.”[1] This mission is still widely pursued, and progress has certainly been made in the last 30 years with simulation curriculum and advancement playing a significant role in the training of anesthesiologists. In its first year of operation, the APSF awarded four grants to patient safety research, to achieve its goal of improvement in patient safety. In the 1999 report by the Institute of Medicine, To Err is Human, the APSF was recognized as an organization that has made demonstratable positive impacts on patient safety.[2] The APSF can be credited with facilitating the growth of anesthesia simulation as a tool for education, training, research, and assessment.[3] In 1988, the APSF took part in organizing a conference on medical simulation followed by an anesthesia simulator curriculum conference a year later in 1989. After that, a national conference on anesthesia simulation and its role in education took place in 1995.

Anesthesiology’s stake in simulation dates as far back as the 1950s when anesthesiologist Dr. Peter Safar demonstrated the ability to perform mouth-to-mouth ventilation on a chemically paralyzed human volunteer.[4] Dr. Safar, together with Dr. Bjorn Lind and toy-maker Asmund Laerdal, went on to create the first resuscitation mannequin, Resusci Anne, in the 1960s.[5] Its use today in BLS education is a testament to the importance of the earliest simulation research and development. Then, in the late 1960s, medical student Stephen Abrahamson and anesthesiologist Dr. J. Samuel Denson, along with engineers from Aerojet-General Corporation, developed the Sim One, a mannequin simulator controlled remotely by a computer, which has been used for educating anesthesiology residents in airway management.[6][7][8] Gaba and colleagues developed the virtual anesthesiology training simulation system in 1986, a true high-fidelity simulator.[9][10] Unlike its predecessors, this model integrated with the actual operating room (OR) monitors, allowing a systems operator working behind the scene to simulate the physiologic response to hands-on input from trainees. This simulator was used for training in intubation and handling endotracheal tube complications as well. A simulation instructor could run through various scripts with trainees. The focus on anesthesia Crisis Resource Management (CRM) began coming into its own in the 1990s, drawing inspiration from other industries such as aviation’s Crew Resource Management.[11] The advent of high-fidelity simulators allowed for further studies evaluating their practicality in supporting the CRM curriculum.[10][12][13]

Function

Low-fidelity mannequins used for cardiopulmonary resuscitation (CPR) training in basic and advanced life support (BLS/ACLS) courses differ significantly from high fidelity mannequins that have the capability of recording tidal volume, chest compression depth, and rate as well as respiratory rate.[14] In stressful or emergencies, the ability to retrieve knowledge stored in long-term memory becomes impaired, even in the most competent practitioners. Simulation is particularly helpful for the practice of various clinical scenarios when anesthesia is performed outside the main hospital setting with less support, such as at an outpatient surgery center, where the staff may not have significant clinical experience in managing medical emergencies. Identifying weaknesses in a hospital system or areas where workflow may be impaired is another objective of simulation. The benefit of simulation, in this case, is uncovering and addressing unsafe or inefficient aspects of the work environment within a safe learning environment.[15] The higher the acuity of the situation, the more apparent these inefficiencies can become.[16] For instance, in obstetric anesthesia, an “eclampsia box” was conceived through simulated drills when it was determined that essential items needed to treat an eclamptic seizure promptly were challenging to find in a timely manner.[17] It is well known from other high-risk industries such as aviation and nuclear energy that simulation exercises within the work environment are most effective, but considering the staff and facility requirements, are often the most costly as well.[18]

In certain cases, scenarios enacted on simulation models may not truly reflect the actual time course for the disease and the treatments utilized.[19] Thus it remains difficult to assess the consequences and patient care outcomes from failing to complete certain treatment tasks.

In a review of 26 studies where augmented and mixed reality technology was used for medical education beyond surgery, the most common application of simulation exercises was to anatomy and anesthesiology procedures.[20] Augmented reality (AR) has been used for simulation of central vein catheterization, lumbar puncture, spinal needle insertion, and ultrasound examination for trauma.[21][22][23][24][25][26][27] While all of these studies demonstrated benefits over conventional learning techniques, more studies are needed to account for the variability in reporting and study designs, among the existing literature, before AR can gain wider acceptance as a complementary educational tool.

Continuing Education

Simulation has been in use to earn credits for the maintenance of certification in anesthesia (MOCA). Weinger et al. examined the use of mannequin based simulation by 263 board-certified anesthesiologists during the simulation component of their maintenance of certification, which includes two 20 minutes duration emergency situations, to identify performance gaps and to establish psychometric properties of the assessment methods.[28] Successful study of simulation with summative aims may eventually shape the simulation curriculum. With formative and educational goals, the most common deficiencies identified in practicing anesthesiologists can be the topics of emphasis in the training of medical students and anesthesia residents. What this study did differently from preceding studies was that it evaluated both the technical and non-technical aspects of clinical decision making and judgment.[29] The authors did not evaluate physicians individually and used shorter scenarios weighted more towards the assessment of technical performance. In this way, the authors were able to view subjects in a more clinically realistic environment, which integrates a team and the necessity for communication.[30] This also allowed them to identify key gaps in performance during crisis management, such as failing to escalate care when the first-line treatment modality failed, calling for help, interacting with and delegating roles to other team members, and following evidence-based guidelines. Specific examples of deficiencies in care observed in both simulation and actual clinical settings include failure to transfuse for hemorrhage or requesting the surgeon convert to open procedure, cardioversion for unstable arrhythmias, etc.[31][32][33][34] When the anesthesiologist in the “hot seat” called a second physician for help, the performance was more likely to be better. Limitations to this form of study include physicians being unfamiliar with and having to adapt to the simulation environment. Real crisis outcomes may be superior to simulation if the physician is more familiar with their office/OR/team. Regardless, the simulation portion of MOCA has been viewed as realistic and applicable to practice by thousands of board-certified anesthesiologists.[35][36] A drawback of current simulation evaluation is inter-rater variability or evaluator bias, which can contribute to disagreement in the ratings of technical and behavioral performances. There is currently no gold standard evaluation technique or method to achieve consistent, reliable ratings. The ability to video-tape and review simulation exercises may lead to future studies that develop a standardized method of evaluation. It remains difficult to determine if the addition of a simulation component to the MOCA every ten years will result in real-life improvements to how anesthesiologists manage emergency events.[37] When we look at the aviation and nuclear industries, they can practice simulation as frequently as every two months to annually. However, feedback gathered by the American Society of Anesthesiologists (ASA) Simulation and Editorial Board (SEB) has been overwhelmingly positive concerning the utility of the MOCA simulation courses, which began in 2010.[35] Perhaps the most important figure is that as of 2014, after over 2700 participants have completed the course thus far, 94% reported changing at least one thing in their practice, and that correlates to real changes being made as a result of these simulation courses.[38][39]

Clinical Significance

The anesthesiology milestones project created by the American Board of Anesthesiology (ABA), together with the Accreditation Council for Graduate Medical Education (ACGME), consists of 6 general core competencies and several sub-competencies as well as serves as an anchor for post-graduate anesthesia education.[40] Performance is reported by training programs to the ACGME biannually. Some sub-competencies, especially those within the realms of crisis management, professionalism, ethical dilemmas, and communication, may sometimes be difficult to assess in the clinical setting. Simulation offers an outlet to educate and assess performance. One way this has been implemented is through Objective Structured Clinical Examinations (OSCEs). In 2018, the ABA implemented an OSCE component in the applied exam taken by anesthesiology residents. In a survey of 66 anesthesiology residency programs, roughly one in three programs were incorporating OSCEs into their curriculum in preparation for the ABA exam.[41] 75% of those programs yet to do so had plans to create an OSCE curriculum of their own at some point. Several academic institutions have shown success in preparing residents for the ABA exam as well as identifying deficiencies in certain competencies through OSCEs, despite the limitations that challenge programs in general, such as resources, faculty, and time.[42][43][44]

Enhancing Healthcare Team Outcomes

COGNITIVE AIDS

The use of cognitive aids or algorithms is helpful in the retrieval of information during high-acuity situations that demand quick, yet efficient action by a healthcare provider and/or team of providers. In industries such as aviation and nuclear energy, the use of cognitive aids is practically mandated, given the nature of regulations in these industries. Limitations to the use of cognitive aids in healthcare include, but are not limited to, the complexity of steps listed in the aids, unclear or incomplete instructions, the stigma that utilizing a cognitive aid implies a suboptimal fund of knowledge and the extra steps involved in using the aid in a crisis.[45] The utility of a cognitive aid not only relies on the clinician's willingness to use it but largely on the design and ease of use of the aid itself. Experimental aids geared towards reducing the time to find information through color-coding, content labeling, and clustering have demonstrated to be more favorable versus traditional step-wise aids in low-fidelity simulation.[46] [Level III] In a randomized control trial examining performance as well as perceived task load and anxiety in medical students learning difficult airway techniques using one of two cognitive aids, the performance was better in those that used the simpler vortex algorithm instead of the ASA difficult airway algorithm.[47] [Level II] Another pilot simulation study shows how the use of an obstetric anesthesia checklist in high-fidelity cesarean section simulation improved the management of peripartum hemorrhage and pre-eclampsia.[48] [Level III]

An extensive review of the existing literature involving the use of cognitive aids in anesthetic emergencies demonstrates improvement in technical skills. Although the impact of these aids on team coordination is not clearly established and requires further study.[49] Simulation is beneficial in allowing physicians in training to practice treating diseases such as malignant hyperthermia (MH), which has an incidence of 1 in 62,000 general anesthesia cases where succinylcholine is used, and other rare but potentially fatal conditions.[50] In high-fidelity simulations of malignant hyperthermia in adult patients, first and second-year anesthesia residents had significantly better treatment scores with more frequent use of cognitive aids.[19] [Level III] This result has been reproducible in cohorts of registered anesthesiologists using checklists to aid in the treatment of malignant hyperthermia as well.[51] [Level III] With respect to MH, cognitive aids could be improved by stressing the importance of administering dantrolene as early as possible and how to mix this medication properly. Berkenstadt et al. developed an on-line electronic help system that includes cognitive aids to be used in point of care anesthesia scenarios, and it was shown to improve physician's performances in screen-based anesthesia simulations.[52] [Level III]

DEBRIEFING

Debriefing can be learner-guided or facilitator-guided, the latter being more commonly used and better studied.[53][54] This may be because having a trained facilitator-led discussion allows learners to stay on task, promotes efficient use of the time, and to focus on the learning objectives.[55] Several 3-phase and multiphase conversation structures utilized in facilitator-guided post-event debriefing have been studied. While no single conversation structure can be definitively considered superior to the others, each has its benefits, and choosing which to use should be tailored to the clinical scenario.[53] A feature common to most structures is that each allocates a time to discuss the events that took place during the simulation and provides a summary of the key take-away points. Where some of these structures differ is that not all incorporate a phase to tackle any emotions experienced by the learners. While taking the time to address emotions can pave the way for a targeted and efficient discussion of the exercise, some models assume that healthcare professionals often deal with high-acuity scenarios and thus should already be equipped to recapitulate on the events objectively. For example, Rudolph et al.'s debriefing with a good judgment model consists of a reaction, analysis, and summary phase. The reaction phase is typically started by asking learners how they felt.[56] Conversely, the diamond debriefing method does not include a specific phase dedicated to reaction; however, it does uniquely include an application phase that asks learners how they will apply the acquired skills and lessons to their clinical practice.[57] A 4-phase method, promoting excellence and reflective learning in simulation, better known as PEARLS, builds off of the 3-phase debriefing with good judgment model but includes an additional description phase. The description phase focuses on key events and/or issues that transpired during the simulation with an overall aim to have the learners and facilitators develop a shared mental model before further debriefing conversation.[58] Another widely used framework known as the team-guided team self-correction, advocacy-inquiry, and systemic-constructivist (TeamGAINS) model, has seven steps that are more team-centric, focusing on an individual within their system than in isolation.[59]

In one study that looked at the rates of debriefing that occurred after critical events at a large academic medical center, roughly half of the events were debriefed.[60] [Level III] Of those, events that were associated with communication breakdowns amongst team members were less likely to be debriefed than critical events, largely secondary to more focus on patient's pathophysiology. The authors noted that fewer than half of critical events truly undergo a comprehensive productive debriefing process. This study not only suggests that there may be an underlying apprehension to address communication barriers amongst team-members but further emphasizes the crucial role simulation plays as a primary means to prevent communication breakdown from occurring.

COMMUNICATION

Teamwork is paramount to anesthesia crisis resource management, especially when it relies on multidisciplinary decision-making. In response to the institute of medicine's report, to err is human, as well as the public demand for improvement in patient safety, the agency for healthcare research and quality (AHRQ) and the department of defense (DOD) have developed a curriculum dubbed TeamSTEPPS (team strategies & tools to enhance performance and patient safety) in 2006.[61] Based on several years of research into teamwork as it pertains to healthcare workers, TeamSTEPPS can be likened to a meta-analysis, put into practice. Teamwork, knowledge, skills, and abilities (KSAs) such as closed-loop communication, shared mental models, and adaptability are among several included in the building blocks of TeamSTEPPS.[61] Weller et al. coded different aspects of team-focused communications and studied how communication differs between simulated events and actual practice in real clinical scenarios in the operating room (OR).[62] They showed no significant difference in the quality of communication during simulated vs. real routine events, however, observed that significantly less and lower quality communication occurred during real OR crises vs. simulated crisis events. This only strengthens the case that simulation is a practical tool for practicing teamwork and communication, not only for anesthesiology training but also can be used widely in other specialties.

Details

References

[1]

Eichhorn JH. The Anesthesia Patient Safety Foundation at 25: a pioneering success in safety, 25th anniversary provokes reflection, anticipation. Anesthesia and analgesia. 2012 Apr:114(4):791-800. doi: 10.1213/ANE.0b013e3182427536. Epub 2012 Jan 16 [PubMed PMID: 22253277]

[2]

Institute of Medicine (US) Committee on Quality of Health Care in America, Kohn LT, Corrigan JM, Donaldson MS. To Err is Human: Building a Safer Health System. 2000:(): [PubMed PMID: 25077248]

Level 2 (mid-level) evidence

[3]

Cooper JB. Towards patient safety in anaesthesia. Annals of the Academy of Medicine, Singapore. 1994 Jul:23(4):552-7 [PubMed PMID: 7979131]

[4]

Sinz EH. Anesthesiology national CME program and ASA activities in simulation. Anesthesiology clinics. 2007 Jun:25(2):209-23 [PubMed PMID: 17574186]

[5]

Grenvik A, Schaefer J. From Resusci-Anne to Sim-Man: the evolution of simulators in medicine. Critical care medicine. 2004 Feb:32(2 Suppl):S56-7 [PubMed PMID: 15043230]

[6]

Abrahamson S. Sim One--a patient simulator ahead of its time. Caduceus (Springfield, Ill.). 1997 Autumn:13(2):29-41 [PubMed PMID: 9509635]

[7]

Denson JS, Abrahamson S. A computer-controlled patient simulator. JAMA. 1969 Apr 21:208(3):504-8 [PubMed PMID: 5818529]

[8]

Abrahamson S, Denson JS, Wolf RM. Effectiveness of a simulator in training anesthesiology residents. 1969. Quality & safety in health care. 2004 Oct:13(5):395-7 [PubMed PMID: 15465945]

Level 2 (mid-level) evidence

[9]

Gaba DM, DeAnda A. A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology. 1988 Sep:69(3):387-94 [PubMed PMID: 3415018]

[10]

Gaba DM, DeAnda A. The response of anesthesia trainees to simulated critical incidents. Anesthesia and analgesia. 1989 Apr:68(4):444-51 [PubMed PMID: 2929977]

[11]

Howard SK, Gaba DM, Fish KJ, Yang G, Sarnquist FH. Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviation, space, and environmental medicine. 1992 Sep:63(9):763-70 [PubMed PMID: 1524531]

[12]

Reznek M, Smith-Coggins R, Howard S, Kiran K, Harter P, Sowb Y, Gaba D, Krummel T. Emergency medicine crisis resource management (EMCRM): pilot study of a simulation-based crisis management course for emergency medicine. Academic emergency medicine : official journal of the Society for Academic Emergency Medicine. 2003 Apr:10(4):386-9 [PubMed PMID: 12670855]

Level 3 (low-level) evidence

[13]

Curtis JR, Back AL, Ford DW, Downey L, Shannon SE, Doorenbos AZ, Kross EK, Reinke LF, Feemster LC, Edlund B, Arnold RW, O'Connor K, Engelberg RA. Effect of communication skills training for residents and nurse practitioners on quality of communication with patients with serious illness: a randomized trial. JAMA. 2013 Dec 4:310(21):2271-81. doi: 10.1001/jama.2013.282081. Epub [PubMed PMID: 24302090]

Level 2 (mid-level) evidence

[14]

Ritt RM, Bennett JD, Todd DW. Simulation Training for the Office-Based Anesthesia Team. Oral and maxillofacial surgery clinics of North America. 2017 May:29(2):169-178. doi: 10.1016/j.coms.2016.12.006. Epub 2017 Mar 6 [PubMed PMID: 28279584]

[15]

Wenk M, Pöpping DM. Simulation for anesthesia in obstetrics. Best practice & research. Clinical anaesthesiology. 2015 Mar:29(1):81-6. doi: 10.1016/j.bpa.2015.01.003. Epub 2015 Feb 7 [PubMed PMID: 25902469]

[16]

Henriksen K, Battles JB, Keyes MA, Grady ML. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools). 2008 Aug:(): [PubMed PMID: 21249925]

Level 3 (low-level) evidence

[17]

Thompson S,Neal S,Clark V, Clinical risk management in obstetrics: eclampsia drills. BMJ (Clinical research ed.). 2004 Jan 31; [PubMed PMID: 14751898]

[18]

Gaba DM. The future vision of simulation in health care. Quality & safety in health care. 2004 Oct:13 Suppl 1(Suppl 1):i2-10 [PubMed PMID: 15465951]

Level 2 (mid-level) evidence

[19]

Harrison TK, Manser T, Howard SK, Gaba DM. Use of cognitive aids in a simulated anesthetic crisis. Anesthesia and analgesia. 2006 Sep:103(3):551-6 [PubMed PMID: 16931660]

[20]

Gerup J, Soerensen CB, Dieckmann P. Augmented reality and mixed reality for healthcare education beyond surgery: an integrative review. International journal of medical education. 2020 Jan 18:11():1-18. doi: 10.5116/ijme.5e01.eb1a. Epub 2020 Jan 18 [PubMed PMID: 31955150]

[21]

Jeon Y, Choi S, Kim H. Evaluation of a simplified augmented reality device for ultrasound-guided vascular access in a vascular phantom. Journal of clinical anesthesia. 2014 Sep:26(6):485-9. doi: 10.1016/j.jclinane.2014.02.010. Epub 2014 Sep 6 [PubMed PMID: 25204510]

[22]

Huang CY, Thomas JB, Alismail A, Cohen A, Almutairi W, Daher NS, Terry MH, Tan LD. The use of augmented reality glasses in central line simulation: "see one, simulate many, do one competently, and teach everyone". Advances in medical education and practice. 2018:9():357-363. doi: 10.2147/AMEP.S160704. Epub 2018 May 10 [PubMed PMID: 29785148]

Level 3 (low-level) evidence

[23]

Rochlen LR, Levine R, Tait AR. First-Person Point-of-View-Augmented Reality for Central Line Insertion Training: A Usability and Feasibility Study. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2017 Feb:12(1):57-62. doi: 10.1097/SIH.0000000000000185. Epub [PubMed PMID: 27930431]

Level 2 (mid-level) evidence

[24]

Robinson AR 3rd, Gravenstein N, Cooper LA, Lizdas D, Luria I, Lampotang S. A mixed-reality part-task trainer for subclavian venous access. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2014 Feb:9(1):56-64. doi: 10.1097/SIH.0b013e31829b3fb3. Epub [PubMed PMID: 24310163]

[25]

Keri Z, Sydor D, Ungi T, Holden MS, McGraw R, Mousavi P, Borschneck DP, Fichtinger G, Jaeger M. Computerized training system for ultrasound-guided lumbar puncture on abnormal spine models: a randomized controlled trial. Canadian journal of anaesthesia = Journal canadien d'anesthesie. 2015 Jul:62(7):777-84. doi: 10.1007/s12630-015-0367-2. Epub 2015 Mar 25 [PubMed PMID: 25804431]

Level 1 (high-level) evidence

[26]

Sutherland C, Hashtrudi-Zaad K, Sellens R, Abolmaesumi P, Mousavi P. An augmented reality haptic training simulator for spinal needle procedures. IEEE transactions on bio-medical engineering. 2013 Nov:60(11):3009-18. doi: 10.1109/TBME.2012.2236091. Epub 2012 Dec 21 [PubMed PMID: 23269747]

[27]

Wang S, Parsons M, Stone-McLean J, Rogers P, Boyd S, Hoover K, Meruvia-Pastor O, Gong M, Smith A. Augmented Reality as a Telemedicine Platform for Remote Procedural Training. Sensors (Basel, Switzerland). 2017 Oct 10:17(10):. doi: 10.3390/s17102294. Epub 2017 Oct 10 [PubMed PMID: 28994720]

[28]

Weinger MB, Banerjee A, Burden AR, McIvor WR, Boulet J, Cooper JB, Steadman R, Shotwell MS, Slagle JM, DeMaria S Jr, Torsher L, Sinz E, Levine AI, Rask J, Davis F, Park C, Gaba DM. Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists. Anesthesiology. 2017 Sep:127(3):475-489. doi: 10.1097/ALN.0000000000001739. Epub [PubMed PMID: 28671903]

[29]

Murray DJ, Boulet JR, Avidan M, Kras JF, Henrichs B, Woodhouse J, Evers AS. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology. 2007 Nov:107(5):705-13 [PubMed PMID: 18073544]

[30]

Fletcher GC, McGeorge P, Flin RH, Glavin RJ, Maran NJ. The role of non-technical skills in anaesthesia: a review of current literature. British journal of anaesthesia. 2002 Mar:88(3):418-29 [PubMed PMID: 11990277]

[31]

Dutton RP, Lee LA, Stephens LS, Posner KL, Davies JM, Domino KB. Massive hemorrhage: a report from the anesthesia closed claims project. Anesthesiology. 2014 Sep:121(3):450-8. doi: 10.1097/ALN.0000000000000369. Epub [PubMed PMID: 25000278]

[32]

Mhyre JM, Ramachandran SK, Kheterpal S, Morris M, Chan PS, American Heart Association National Registry for Cardiopulmonary Resuscitation Investigators. Delayed time to defibrillation after intraoperative and periprocedural cardiac arrest. Anesthesiology. 2010 Oct:113(4):782-93. doi: 10.1097/ALN.0b013e3181eaa74f. Epub [PubMed PMID: 20808215]

[33]

Novick RJ, Lingard L, Cristancho SM. The call, the save, and the threat: understanding expert help-seeking behavior during nonroutine operative scenarios. Journal of surgical education. 2015 Mar-Apr:72(2):302-9. doi: 10.1016/j.jsurg.2014.09.009. Epub 2014 Nov 11 [PubMed PMID: 25451719]

Level 3 (low-level) evidence

[34]

Arriaga AF, Gawande AA, Raemer DB, Jones DB, Smink DS, Weinstock P, Dwyer K, Lipsitz SR, Peyre S, Pawlowski JB, Muret-Wagstaff S, Gee D, Gordon JA, Cooper JB, Berry WR, Harvard Surgical Safety Collaborative. Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams. Annals of surgery. 2014 Mar:259(3):403-10. doi: 10.1097/SLA.0000000000000342. Epub [PubMed PMID: 24263327]

Level 3 (low-level) evidence

[35]

McIvor W, Burden A, Weinger MB, Steadman R. Simulation for maintenance of certification in anesthesiology: the first two years. The Journal of continuing education in the health professions. 2012 Fall:32(4):236-42. doi: 10.1002/chp.21151. Epub [PubMed PMID: 23280526]

[36]

Steadman RH, Burden AR, Huang YM, Gaba DM, Cooper JB. Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program. Anesthesiology. 2015 May:122(5):1154-69. doi: 10.1097/ALN.0000000000000613. Epub [PubMed PMID: 25985025]

[37]

Weinger MB, Burden AR, Steadman RH, Gaba DM. This is not a test!: Misconceptions surrounding the maintenance of certification in anesthesiology simulation course. Anesthesiology. 2014 Sep:121(3):655-9. doi: 10.1097/ALN.0000000000000303. Epub [PubMed PMID: 24821072]

[38]

Domino FJ, Chopra S, Seligman M, Sullivan K, Quirk ME. The impact on medical practice of commitments to change following CME lectures: a randomized controlled trial. Medical teacher. 2011:33(9):e495-500. doi: 10.3109/0142159X.2011.599452. Epub [PubMed PMID: 21854144]

Level 1 (high-level) evidence

[39]

Mazmanian PE, Johnson RE, Zhang A, Boothby J, Yeatts EJ. Effects of a signature on rates of change: a randomized controlled trial involving continuing education and the commitment-to-change model. Academic medicine : journal of the Association of American Medical Colleges. 2001 Jun:76(6):642-6 [PubMed PMID: 11401812]

Level 1 (high-level) evidence

[40]

. The anesthesiology milestone project. Journal of graduate medical education. 2014 Mar:6(1 Suppl 1):15-28. doi: 10.4300/JGME-06-01s1-30. Epub [PubMed PMID: 24701262]

[41]

Isaak RS, Chen F, Arora H, Martinelli SM, Zvara DA, Stiegler MP. A Descriptive Survey of Anesthesiology Residency Simulation Programs: How Are Programs Preparing Residents for the New American Board of Anesthesiology APPLIED Certification Examination? Anesthesia and analgesia. 2017 Sep:125(3):991-998. doi: 10.1213/ANE.0000000000002189. Epub [PubMed PMID: 28632531]

Level 3 (low-level) evidence

[42]

Rebel A, Srour H, DiLorenzo A, Nguyen D, Ferrell S, Dwarakanatli S, Haas E, Schell RM. Ultrasound Skill and Application of Knowledge Assessment using an Innovative OSCE Competition-Based Simulation Approach. The journal of education in perioperative medicine : JEPM. 2016 Jul-Dec:18(1):E404 [PubMed PMID: 27957515]

[43]

Miller C, Toy S, Schwengel D, Isaac G, Schiavi A. Development of a Simulated Objective Structured Clinical Exam for the APPLIED Certification Exam in Anesthesiology: A Two-Year Experience Informed by Feedback from Exam Candidates. The journal of education in perioperative medicine : JEPM. 2019 Oct-Dec:21(4):E633 [PubMed PMID: 32123698]

[44]

Tanaka P, Park YS, Liu L, Varner C, Kumar AH, Sandhu C, Yumul R, McCartney KT, Spilka J, Macario A. Assessment Scores of a Mock Objective Structured Clinical Examination Administered to 99 Anesthesiology Residents at 8 Institutions. Anesthesia and analgesia. 2020 Aug:131(2):613-621. doi: 10.1213/ANE.0000000000004705. Epub [PubMed PMID: 32149757]

[45]

Morris AH. Decision support and safety of clinical environments. Quality & safety in health care. 2002 Mar:11(1):69-75 [PubMed PMID: 12078374]

Level 2 (mid-level) evidence

[46]

Clebone A, Burian BK, Tung A. The effect of cognitive aid design on the perceived usability of critical event cognitive aids. Acta anaesthesiologica Scandinavica. 2020 Mar:64(3):378-384. doi: 10.1111/aas.13503. Epub 2019 Nov 25 [PubMed PMID: 31709509]

[47]

Ambardekar AP, Rosero EB, Bhoja R, Green J, Rebal BA, Minhajuddin AT, Kosemund MS, Guttman OT, Mercier DW. A Randomized Controlled Trial Comparing Learners' Decision-making, Anxiety, and Task Load During a Simulated Airway Crisis Using Two Difficult Airway Aids. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2019 Apr:14(2):96-103. doi: 10.1097/SIH.0000000000000362. Epub [PubMed PMID: 30908420]

Level 1 (high-level) evidence

[48]

Alexander LA, Newton MW, McEvoy KG, Newton MJ, Mungai M, DiMiceli-Zsigmond M, Sileshi B, Watkins SC, McEvoy MD. Development and Pilot Testing of a Context-Relevant Safe Anesthesia Checklist for Cesarean Delivery in East Africa. Anesthesia and analgesia. 2019 May:128(5):993-998. doi: 10.1213/ANE.0000000000003874. Epub [PubMed PMID: 30379674]

Level 3 (low-level) evidence

[49]

Marshall S. The use of cognitive aids during emergencies in anesthesia: a review of the literature. Anesthesia and analgesia. 2013 Nov:117(5):1162-71. doi: 10.1213/ANE.0b013e31829c397b. Epub [PubMed PMID: 24029855]

[50]

Halliday NJ. Malignant hyperthermia. The Journal of craniofacial surgery. 2003 Sep:14(5):800-2 [PubMed PMID: 14501352]

[51]

Hardy JB, Gouin A, Damm C, Compère V, Veber B, Dureuil B. The use of a checklist improves anaesthesiologists' technical and non-technical performance for simulated malignant hyperthermia management. Anaesthesia, critical care & pain medicine. 2018 Feb:37(1):17-23. doi: 10.1016/j.accpm.2017.07.009. Epub 2017 Sep 20 [PubMed PMID: 28939466]

[52]

Berkenstadt H, Yusim Y, Katznelson R, Ziv A, Livingstone D, Perel A. A novel point-of-care information system reduces anaesthesiologists' errors while managing case scenarios. European journal of anaesthesiology. 2006 Mar:23(3):239-50 [PubMed PMID: 16430796]

Level 3 (low-level) evidence

[53]

Cheng A, Eppich W, Grant V, Sherbino J, Zendejas B, Cook DA. Debriefing for technology-enhanced simulation: a systematic review and meta-analysis. Medical education. 2014 Jul:48(7):657-66. doi: 10.1111/medu.12432. Epub [PubMed PMID: 24909527]

Level 1 (high-level) evidence

[54]

Tannenbaum SI, Cerasoli CP. Do team and individual debriefs enhance performance? A meta-analysis. Human factors. 2013 Feb:55(1):231-45 [PubMed PMID: 23516804]

Level 1 (high-level) evidence

[55]

Sawyer T, Eppich W, Brett-Fleegler M, Grant V, Cheng A. More Than One Way to Debrief: A Critical Review of Healthcare Simulation Debriefing Methods. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2016 Jun:11(3):209-17. doi: 10.1097/SIH.0000000000000148. Epub [PubMed PMID: 27254527]

[56]

Rudolph JW, Simon R, Dufresne RL, Raemer DB. There's no such thing as "nonjudgmental" debriefing: a theory and method for debriefing with good judgment. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2006 Spring:1(1):49-55 [PubMed PMID: 19088574]

[57]

Jaye P, Thomas L, Reedy G. 'The Diamond': a structure for simulation debrief. The clinical teacher. 2015 Jun:12(3):171-5. doi: 10.1111/tct.12300. Epub [PubMed PMID: 26009951]

[58]

Eppich W, Cheng A. Promoting Excellence and Reflective Learning in Simulation (PEARLS): development and rationale for a blended approach to health care simulation debriefing. Simulation in healthcare : journal of the Society for Simulation in Healthcare. 2015 Apr:10(2):106-15. doi: 10.1097/SIH.0000000000000072. Epub [PubMed PMID: 25710312]

[59]

Kolbe M, Weiss M, Grote G, Knauth A, Dambach M, Spahn DR, Grande B. TeamGAINS: a tool for structured debriefings for simulation-based team trainings. BMJ quality & safety. 2013 Jul:22(7):541-53. doi: 10.1136/bmjqs-2012-000917. Epub 2013 Mar 22 [PubMed PMID: 23525093]

Level 2 (mid-level) evidence

[60]

Arriaga AF, Sweeney RE, Clapp JT, Muralidharan M, Burson RC 2nd, Gordon EKB, Falk SA, Baranov DY, Fleisher LA. Failure to Debrief after Critical Events in Anesthesia Is Associated with Failures in Communication during the Event. Anesthesiology. 2019 Jun:130(6):1039-1048. doi: 10.1097/ALN.0000000000002649. Epub [PubMed PMID: 30829661]

[61]

Henriksen K, Battles JB, Keyes MA, Grady ML, King HB, Battles J, Baker DP, Alonso A, Salas E, Webster J, Toomey L, Salisbury M. TeamSTEPPS(™): Team Strategies and Tools to Enhance Performance and Patient Safety. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools). 2008 Aug:(): [PubMed PMID: 21249942]

Level 3 (low-level) evidence

[62]

Weller J, Henderson R, Webster CS, Shulruf B, Torrie J, Davies E, Henderson K, Frampton C, Merry AF. Building the evidence on simulation validity: comparison of anesthesiologists' communication patterns in real and simulated cases. Anesthesiology. 2014 Jan:120(1):142-8. doi: 10.1097/ALN.0b013e3182a44bc5. Epub [PubMed PMID: 23903023]

Level 3 (low-level) evidence