Mixed methods analysis of reflective statements of residents following the developmental-behavioral pediatric rotation
Introduction
Reflection is important in the professional practice of medicine, as it encourages trainees to reframe problems, question assumptions, and recognize multiple perspectives to identify gaps in their own knowledge to enhance the therapeutic relationship and their clinical decisions (1). Analyses of trainees’ reflection papers have been utilized to evaluate training outcomes in various disciplines, including pharmacy (2) and family medicine (3). Reflection has been used in surgery (4) with a summary reflective statement at the end of each week to assess the curriculum and in family medicine (3) using reflective exercises each week over the course of a year as part of residency curriculum. Qualitative analysis of reflection papers often involves thematic analysis (2,3,5). One framework by Castleberry et al. (2) utilized a five-point coding scale adapted from Kember (6) to quantify the degree of reflection: absent, non-reflection, understanding, reflection, and critical reflection.
The United States (US) Accreditation Council for Graduate Medical Education (ACGME) for pediatrics requires residents to undergo one ‘educational unit’ (4 weeks or 1 month block or a longitudinal experience) in developmental-behavioral pediatrics (DBP) (7). DBP is a pediatric subspecialty that focuses on the evaluation, treatment, and management of infants, children, and adolescents with a wide range of developmental and behavioral conditions. DBP has an emphasis on interdisciplinary practice (8), and the DBP rotation can often include community experiences (including schools and therapy centers) as well as experiences with other disciplines (psychology, speech-language therapy) in unidisciplinary or multidisciplinary formats (9). Leadership Education in Neurodevelopmental and Related Disabilities (LEND) is an interdisciplinary certificate program that provides graduate level training designed to improve the health and wellbeing of individuals with disabilities. Both DBP and LEND training occur under the umbrella of Maternal Child Health (MCH) Bureau and share many training characteristics. Twelve MCH Leadership Competencies, organized within a conceptual framework in a progression from self to wider community, were developed to be shared across the multiple MCH disciplines, unify the workforce on a common path to equip the MCH workforce to improve the health of MCH populations.
The Michigan LEND program developed a leadership (L), interdisciplinary (I), family-centered (F), and equity (E), aka LIFE framework which codifies the 12 MCH competencies (10) into an easy to remember and easy to apply structure. Table 1 shows the LIFE framework key questions/prompts for each of the domains used to facilitate reflective thinking.
Table 1
| Leadership | Interdisciplinary | Family-centered | Equity |
|---|---|---|---|
| What tools can be used to avoid or resolve conflicts? | What is my role in identification, assessment, and treatment? | Who are the members of the family? | Who is accessing/benefiting from our programs? Who is not? |
| What leadership strengths/styles can I bring to this situation? | What does each discipline bring to the care of a child and family? | What are their current roles? | Who is at risk for disparate outcomes in the health system? |
| What strategies can I use to facilitate creativity, innovation, collaboration, and leadership? | How do disciplines interact as a team? | How can family strengths be built upon to better support the child with a disability? | What are barriers, differential impacts? What can we do to change that? |
| How can I better facilitate learning between the classroom and the workplace? | How can I more effectively communicate with other professionals on my team? | How does the family want to engage with professionals? | Why are some people at greater risk? How can we reach and engage them? |
| How can I address identified gaps in the service system? | How can we demonstrate respect for other’s knowledge and approach to a problem? | How can community resources be used to supplement family resources to better support the child? | How are our actions relevant to specific populations? |
| How can the clinician work as a team with other providers and the family, even across geographic distance? | How can we improve our surveillance system and build the ones that collect the data we need? |
Reflection paper: expectation is for the resident to write a reflective paper based on observations in one of the clinic or community sites. The reflection paper should be turned in at the end of the rotation as part of the portfolio. No identifying information (e.g., child’s real name, identifying characteristics) should be included. As part of the reflection, answer one question from each of the four LIFE categories. LIFE, leadership, interdisciplinary, family-centered, and equity.
Additionally, the LIFE framework and its associated competencies are largely consistent with ACGME Pediatrics Milestones (11); acknowledging the importance of interprofessional communication and collaboration as well as family-centered care. However, with its greater focus on leadership and equity, the LIFE framework and its associated competencies arguably expand on the Pediatrics Milestones.
At the senior author’s institution, the DBP rotation is a 1-month block completed usually by first year (PGY-1) of Pediatrics (Peds) residency, though some Medicine-Pediatrics (Med-Peds) residents may experience the rotation in their 2nd year (PGY-2) for scheduling reasons. Residents complete clinical hours with DBP, psychology, unidisciplinary experiences (orthotics, audiology), and have the opportunity to see interdisciplinary practice with medically complex children in the clinics, while also observing community educational evaluations that employ multiple disciplines. With the onset of the coronavirus disease 2019 (COVID-19) pandemic, many of the rotation sites (including schools, community sites, and speech therapy, occupational therapy, physical therapy clinics) were closed or ceased allowing residents to participate due to social distancing rules. However apart from a brief hiatus of in-person care from March to May 2020, the clinical sites at the primary training program continued unabated. This served as the primary experience for the residents in that timeframe and included rotating with DBP, psychology, and the autism evaluation clinic.
At the beginning of the rotation, residents were required to view an introductory video that outlined the LIFE framework in relation to neurodevelopmental disabilities. At the end of the rotation, the residents are asked to reflect on their rotation experience using the LIFE framework by selecting questions from each of the content areas as it applies to either a clinic or community experience during the rotation (see Table 1). They were not provided any additional instructions, nor did they receive any additional training in reflective writing. Residents were informed that the reflection was not intended to be used to evaluate their performance during the rotation but to serve as a self-learning activity. While degree of reflection was scored using Castleberry ratings, this was done after residents completed the rotation and was not shared with the residents.
The purpose of the study was to determine the degree of reflection for the various LIFE categories in the Peds and Med-Peds residents’ reflective statements. The primary objective was to evaluate the extent to which residents reflect on the LIFE framework by reporting the depth of thinking of each of the four LIFE categories. A secondary objective was to determine whether residents have higher Castleberry ratings (corresponding to a higher level of reflection) depending on timing of the rotation (earlier or later in training year, or before or since the COVID-19 pandemic). We hypothesized that those who complete their DBP rotation later in their training will demonstrate a greater level of reflection than those who complete the rotation earlier in their training. We hypothesized that residents who completed their rotation during the COVID-19 pandemic when many interdisciplinary sites were closed will have higher Castleberry ratings on equity but lower ratings on interdisciplinary (as a consequence of their limited exposure to certain experiences). We present the following article in accordance with the SRQR reporting checklist (available at https://pm.amegroups.com/article/view/10.21037/pm-21-116/rc).
Methods
This is a retrospective study of reflection statements submitted by Peds and Med-Peds residents at the senior author’s institution, at the end of their DBP rotation. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The Western Michigan University Homer Stryker M.D. School of Medicine Institutional Review Board (IRB#00010682) deemed the study exempt from full review as all data were deidentified and could not be linked back to the participants and individual consent for this retrospective analysis was waived. One hundred percent of eligible Peds and Med-Peds residents completed reflection statements from July 1, 2017, through March 30, 2021, and were included in the study with no exclusion criteria. The DBP rotation coordinator deidentified the original reflection statements (in paper or electronic form). Each reflection paper was coded for level of reflection within each LIFE category using the Castleberry coding scheme; therefore, each reflection paper received four independent codes. Any “extraneous” or introductory reflection content had LIFE themes extracted for coding. Two independent coders, trained to 100% agreement on reflections of two medical students (not included in the resident analyses), rated the reflections with interrater agreement with a third coder ranging from 84.4–93.7%. Remaining disagreements were resolved through discussion. See Table 2 for the Castleberry coding scheme.
Table 2
| Level | Description | Example statement |
|---|---|---|
| 0: absent | Prompt was not answered | N/A |
| 1: non-reflection | No elaboration on or explanation of the concept | Physicians play an important role in identifying gaps in service delivery |
| 2: understanding | Elaboration on or explanation of the concept, but no application to personal practice and no examples provided | In order to provide the optimum care an interdisciplinary team needs to function in coordination which should include a PCP, some sort of long-term adult vocational schooling facility and employment providers for this cohort. This will enable them to earn a livelihood, be independent and to become a contributing member of the society reducing the burden that this medical condition brings |
| 3: reflection | Connection of concepts to personal practice through examples and/or discussion of personal role in relation to prompt | Each discipline has a unique role to play in the care of a child and family. Learning starts within the school system where the child spends majority of his/her days. This is where vital steps can be taken to address the weakness and getting testing done to better provide services. For example, obtaining an IEP or 504 plan. These same weaknesses can be worked on even more by involving PT/OT and/or speech therapy. PCP and specialist can help medically to monitor health of the patient. All these services can help alleviate the stress on the families along with providing support and regiments for helping to mold (sic) their child |
| 4: critical reflection | Explicit discussion of behavior or belief change related to prompt due to experience in the rotation | I learned a lot about interdisciplinary care by witnessing the coordination of care between different disciplines when attending the [name] clinic. With the number of appointments that these patients must attend, this model of care is one that (in my opinion) should be exercised in all fields of medicine, especially as pertaining to chronic illness/disease. Having behavioral health, orthopedic surgery, prosthetics, social work, nutrition, etc present at the same time was/is not only convenient for the patient, but also for the care providers. Not only did it help provide a comprehensive plan in a time-effective manner, but that it also helped in minimizing lapses in time that can occur during which errors can be made; period of time during which instructions can be ‘lost in transition and in translation’ |
N/A, not applicable; PCP, primary care physician; IEP, Individualized Education Program; PT/OT, physical therapy/occupational therapy.
Statistical analysis
We conducted data analysis in RStudio v.3.6.0 (Boston, MA, USA; 2019). Descriptive statistics for categorical variables included counts. Descriptive statistics for continuous variables included counts, percent missing data, means, and standard deviations (SDs). We calculated skewness and kurtosis values for the Castleberry ratings as the primary outcomes of interest. Visual inspection of histograms and skewness and kurtosis values suggests Castleberry ratings were not normally distributed. As such, non-parametric tests such as Wilcoxon rank-sum tests were used to test differences in Castleberry ratings for each facet of the LIFE framework by (I) those who complete the rotation before or after the onset of the COVID-19 pandemic and (II) for those who completed the rotation early or late in the training year (i.e., 8 Wilcoxon rank-sum tests).
Results
Thirty-six residents (25 Peds and 11 Med-Peds) completed the DBP rotation during the study period, 67% of whom were female, with 33 completing as PGY-1 and 3 as PGY-2. There were 21 (58%) residents who completed their DBP rotation prior to COVID-19 (March 2020) and 16 (44%) who completed the rotation in the first half of their academic year. Apart from two reflections that did not address the family-centered (F) component, reflections addressed all components of the LIFE framework, half of the reflections provided additional information.
Mean Castleberry ratings were 3.2 (SD =0.7), 2.7 (SD =0.8), 2.6 (SD =1.0), and 2.6 (SD =0.8) for LIFE, respectively. All Castleberry ratings were negatively skewed (range, −0.8 to −1.7) and had kurtosis values of 3.0 or higher (range, 3.0 to 5.8).
Table 3 shows the results of Wilcoxon rank-sum tests where the difference in Castleberry ratings for each facet of the LIFE framework were tested by variables reflecting pre/post-COVID-19 completion of the rotation and early/late in the training year completion of the rotation. Results suggest there were no differences in Castleberry ratings between those who completed the rotation before and after the onset of the COVID-19 pandemic. Results suggest residents who completed the rotation late in the training year had lower Castleberry ratings for the interdisciplinary (I) facet of the LIFE framework (W=214.50; P=0.02). There were no differences between residents who completed the rotation early versus late in the training year on other aspects of the LIFE framework.
Table 3
| LIFE category | N | M | SD | W | P |
|---|---|---|---|---|---|
| Leadership | |||||
| Pre-COVID-19 | 21 | 3.19 | 0.60 | – | – |
| Post-COVID-19 | 15 | 3.20 | 0.86 | 146.50 | 0.70 |
| Early | 16 | 3.38 | 0.50 | – | – |
| Late | 20 | 3.05 | 0.83 | 192.00 | 0.26 |
| Interdisciplinary | |||||
| Pre-COVID-19 | 21 | 2.86 | 0.57 | – | – |
| Post-COVID-19 | 15 | 2.53 | 1.06 | 176.40 | 0.43 |
| Early | 16 | 3.06 | 0.25 | – | – |
| Late | 20 | 2.45 | 1.00 | 214.50 | 0.02* |
| Family-centered | |||||
| Pre-COVID-19 | 21 | 2.48 | 1.03 | – | – |
| Post-COVID-19 | 15 | 2.73 | 0.96 | 129.00 | 0.32 |
| Early | 16 | 2.81 | 0.66 | – | – |
| Late | 20 | 2.40 | 1.19 | 190.50 | 0.29 |
| Equity | |||||
| Pre-COVID-19 | 21 | 2.62 | 0.81 | – | – |
| Post-COVID-19 | 15 | 2.67 | 0.82 | 148.00 | 0.75 |
| Early | 16 | 2.88 | 0.50 | – | – |
| Late | 20 | 2.45 | 0.95 | 200.50 | 0.15 |
*, P<0.05. DBP, developmental-behavioral pediatrics; COVID-19, coronavirus disease 2019; N, number; M, mean; SD, standard deviation; W, test statistic for Wilcoxon rank-sum test; P, P value associated with the test statistic.
Discussion
This study of residents’ reflections at the end of their DBP rotation revealed that the degree of reflection was highest for leadership (L) amongst the four components of the LIFE framework. This is likely because the coding scheme defined reflection based on the degree to which personal insight and practice application were incorporated into the response. Many of the leadership (L) prompts were specifically related to the residents’ personal role, whereas the interdisciplinary (I), family-centered (F), and equity (E) prompts were often focused on aspects outside of the residents’ direct role, such as systems-level considerations. Previous research has shown that the order in which questions are asked can influence participant responding on later questions (12); earlier questions can cause either an assimilation effect (responding more similarly to the first question) or a contrast effect (responding less similarly to the first question). Given that L prompts were first, and reflections on leadership were significantly higher than reflections on the three other LIFE components, it is possible that there was a contrast effect due to the order of the prompts. Residents had no significant difference in the degree in reflection along any of the LIFE components related to the COVID-19 pandemic period and were significantly different only for interdisciplinary (I) reflection being higher earlier in their training compared to later. One possible reason for this is that the DBP rotation is likely the earliest exposure to interdisciplinary practice and the novelty thus prompted residents to higher degree of reflection. As the training year goes on, it is possible that residents have greater appreciation for family centeredness and equity. An important implication for training is that reflection should be encouraged for rotations beyond DBP and implicit instruction provided on reflection strategies. Previous studies with Peds residents’ comments after their Adolescent Medicine rotation similarly articulated taking on a professional role and achieving self-awareness (13). Future directions could involve comparisons of reflections following DBP and other rotations along the LIFE themes, as well as determining if the sequence of other rotations with a high degree of interdisciplinary care (such as Adolescent Medicine) impact the degree of reflection on this area.
Residents had a choice between five to eight prompt questions per each LIFE component, and the prompt question selected could have potentially influenced their degree of reflection, based on the wording and the focus of the question. Previous research suggests that when given a choice between multiple prompts, people choose the prompts which are most clear, interesting, relatable, and eliciting of strong emotions (14). Additionally, participants tend to opt for prompts related to topics that are more specific and that they are more knowledgeable about and have stronger feelings about (15). It is heartening to note that in the sample of residents there was no difference in their reflections along the LIFE framework due to the pandemic. However, this was a small sample and the medical DBP and psychology experiences were generally uninterrupted during this time, which could account for some of the responses. There is concern about the impact of the pandemic on resident education in general (16); and little is known about the impact of COVID-19 on DBP resident education, particularly with closure of interprofessional and community experiences, and transition of many clinics to virtual. While the impact of telehealth for DBP training during COVID-19 is being explored (17), more investigation is needed around the impact of COVID-19 in general on DBP education, both at a resident and fellow level.
Reflective writing can be evaluated both qualitatively and quantitatively to assess both the depth and breadth of reflection. Reflection in clinical medicine helps to sharpen awareness of the multiple dimensions of the patient-physician relationship and fosters physicians’ abilities to acknowledge, identify, and express their strengths, vulnerabilities, limitations, errors, and areas of growth (3). Reflective writing is important to independent learning and learner autonomy, with the opportunity to be mindful learners (18). Faculty can play a key role in facilitating residents in the reflective process, though this might require the former to undergo development through faculty development to understand the theory and application of the reflective process (1). In our study, resident reflections were used not to evaluate resident performance, but to facilitate discussion and feedback with the DBP specialist who headed the rotation. Reflection can be a tool that can clarify professional values and help learners process complex subject matter (19), such as the DBP rotation content.
Limitations of our study include data from a single residency program in a small Midwestern city that conducts DBP rotation primarily in the first year of residency. As such, it may not be representative of all residency programs of all sizes and experiences across the US. Residents were not specifically trained in the creative or technical writing process for reflective statements; receiving (or not) training could be a covariate in analysis in future studies. For this preliminary study, we did not conduct thematic and sentiment analysis on the reflective statements. We could also use open-ended prompts such as, ‘discuss what you have learned related to leadership (L), interdisciplinary (I), family-centered (F), and equity (E) concepts during your rotation’ to allow for a more naturalistic analysis of themes that emerge within each domain. However we did use specific example prompts to encourage critical thinking, as recommended when reflective thinking is not explicitly taught (1).
This study lays the groundwork for broader use of the structure of the LIFE framework in all aspects of Peds and Med-Peds education, particularly given that it expands on the ACGME Pediatrics Milestones. We hoped to show feasibility of a reflective approach and analysis, to identify gaps in perception that can be addressed through pedagogical or experiential learning before or during the rotation for subsequent residents. Future research should compare reflections along the LIFE framework across different rotations and thematic and sentiment analysis can reveal opportunities for guiding residents on the reflection process.
Conclusions
Our study showed that reflection analysis can be feasibly used with residents on a pediatric subspecialty rotation. Peds and Med-Peds residents reflect on their DBP rotation to a moderately high degree, particularly on their own role or leadership (L) in the LIFE framework. There was no difference in reflection related to the COVID-19 pandemic, though residents rotating early in their training year reflected on interdisciplinary (I) principles to a greater degree.
Acknowledgments
The authors would like to thank Joyce Thomas for data management and the Michigan Leadership Education in Neurodevelopmental and Related Disabilities (Mi-LEND) for guidance and providing an evaluative framework.
Funding: None.
Footnote
Reporting Checklist: The authors have completed the SRQR reporting checklist. Available at https://pm.amegroups.com/article/view/10.21037/pm-21-116/rc
Data Sharing Statement: Available at https://pm.amegroups.com/article/view/10.21037/pm-21-116/dss
Peer Review File: Available at https://pm.amegroups.com/article/view/10.21037/pm-21-116/prf
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://pm.amegroups.com/article/view/10.21037/pm-21-116/coif). The authors have no conflicts of interest to declare.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Western Michigan University Homer Stryker M.D. School of Medicine Institutional Review Board (IRB#00010682) and individual consent for this retrospective analysis was waived.
Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.
References
- Plack MM, Greenberg L. The reflective practitioner: reaching for excellence in practice. Pediatrics 2005;116:1546-52. [Crossref] [PubMed]
- Castleberry AN, Payakachat N, Ashby S, et al. Qualitative Analysis of Written Reflections during a Teaching Certificate Program. Am J Pharm Educ 2016;80:10. [Crossref] [PubMed]
- Duggan AP, Vicini A, Allen L, et al. Learning to See Beneath the Surface: A Qualitative Analysis of Family Medicine Residents' Reflections About Communication. J Health Commun 2015;20:1441-8. [Crossref] [PubMed]
- Monaghan SF, Blakely AM, Richardson PJ, et al. The reflective statement: a new tool to assess resident learning. J Surg Res 2012;178:618-22. [Crossref] [PubMed]
- Schaub-de Jong MA, Cohen-Schotanus J, Dekker H, et al. The role of peer meetings for professional development in health science education: a qualitative analysis of reflective essays. Adv Health Sci Educ Theory Pract 2009;14:503-13. [Crossref] [PubMed]
- Kember D. Determining the level of reflective thinking from students' written journals using a coding scheme based on the work of Mezirow. International Journal of Lifelong Education 1999;18:18-30.
- ACGME Program Requirements for Graduate Medical Education in Pediatrics. 2020. [cited 2021 Feb 1]. Available online: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/320_Pediatrics_2020.pdf?ver=2020-06-29-162726-647
- Kelly DP. Interdisciplinary collaborative future for developmental-behavioral pediatrics: barriers and opportunities. J Dev Behav Pediatr 2013;34:523-8. [Crossref] [PubMed]
- Shahidullah JD, Kettlewell PW, Palejwala MH, et al. Behavioral Health Training in Pediatric Residency Programs: A National Survey of Training Directors. J Dev Behav Pediatr 2018;39:292-302. [Crossref] [PubMed]
- Maternal & Child Health. MCH Leadership Competencies. [cited 2021 Apr 16]. Available online: https://mchb.hrsa.gov/training/leadership-00.asp
- The Accreditation Council for Graduate Medical Education. Pediatric Milestones. [cited 2021 Aug 3]. Available online: https://www.acgme.org/portals/0/pdfs/milestones/pediatricsmilestones.pdf
- Carlson JE, Mason R, Saltiel J, et al. Assimilation and Contrast Effects in General/Specific Questions. Rural Sociol 1995;60:666-73.
- Albuhairan F, Leslie K, Goldberg E. Pediatric residents' experiences of a clinical rotation in Adolescent Medicine. BMC Med Educ 2010;10:88. [Crossref] [PubMed]
- Powers DE, Fowles ME. Test-takers' judgments of essay prompts: Perceptions and performance. Educ Assess 1999;6:3-22.
- Polio C, Glew M. ESL writing assessment prompts: How students choose. J Second Lang Writ 1996;5:35-49.
- American Board of Pediatrics. 2020 Annual Report. The American Board of Pediatrics, 2020. [cited 2021 Apr 16]. Available online: https://www.abp.org/sites/abp/files/pdf/annual-report-2020.pdf
- Huffman LC, Feldman HM, Hubner LM. Fellows Front and Center: Tele-Training and Telehealth. Acad Pediatr 2020;20:764-5. [Crossref] [PubMed]
- Kathpalia SS, Heah C. Reflective writing: Insights into what lies beneath. RELC J 2008;39:300-17.
- Winkel AF, Yingling S, Jones AA, et al. Reflection as a Learning Tool in Graduate Medical Education: A Systematic Review. J Grad Med Educ 2017;9:430-9. [Crossref] [PubMed]
Cite this article as: Fledderman NM, Slavin LJ, Drevon DD, Soares NS. Mixed methods analysis of reflective statements of residents following the developmental-behavioral pediatric rotation. Pediatr Med 2023;6:2.
