Ben Craft, BA

Patient experience has become a major component of the Center for Medicare and Medicaid Services Value‐Based Purchasing initiative.[1] Hospitals have therefore focused quality improvement (QI) efforts on this area.[2] Hospital performance in the realm of patient experience is generally determined using systematic surveys with closed‐ended questions, but patient‐generated narrative feedback can help hospitals identify the components of care that contribute to patient satisfaction and or are in need of improvement.[3] Online narrative responses posted by patients on rating websites or social media have been criticized because they may not be representative of the population,[4] but they also have some advantages.[5] Any patient may leave a comment, not just those who are selected for a survey. Patients may also experience benefits through the act of sharing their story with others. Moreover, most US hospitals use some form of social media,[6] which they can theoretically use to self‐collect narrative data online. To realize the full potential of patient‐generated online narratives, we need a clearer understanding of the best practices for collecting and using these narratives. We therefore solicited patient feedback on the Facebook page of a large tertiary academic medical center to determine whether it is feasible to use social media platforms for learning about and improving hospital quality.

METHODS

Baystate Medical Center (BMC) is a tertiary care medical center in western Massachusetts. We identified key BMC stakeholders in the areas of QI and public affairs. Noting that patients have expressed interest in leaving comments via social media,[7] the group opted to perform a pilot study to obtain patient narratives via a Facebook prompt (Facebook is a social media site used by an estimated 58% of US adults[8]). The BMC public affairs department delivered a press release to the local media describing a 3‐week period during which patients were invited to leave narrative feedback on the BMC Facebook wall. The BMC Institutional Review Board deemed that this study did not constitute human subjects research.

During March 2014 (March 10, 2014March 24, 2014), we posted (once a week) an open‐ended prompt on BMC's Facebook wall. The prompt was designed to elicit novel descriptions of patient experience that could help to drive QI. It read: We want to hear about your experiences. In the comment section below, please tell us what we do well and how we can improve your care. Because of concerns about the potential reputational risks of allowing open feedback on a public social media page, the prompt also reminded patients of the social media ground rules: there should be no mention of specific physicians, nurses, or other caregivers by name (for liability reasons); and patients should not include details about their medical history (for privacy reasons).

We collected all posts to preserve comments and used directed qualitative content analysis to examine them.[9] Two research team members[3, 10, 11] independently coded the responses. Starting with an a priori codebook that was developed during a previous study,[3] they amended the codebook through an iterative process to incorporate new concepts. After independently coding all blocks of text, the coders reviewed their coding selections and resolved discrepancies through discussion. We then performed second‐level coding, in which codes were organized into major pertinent themes. We reviewed the coded text after applying secondary codes in order to check for accuracy of coding and theme assignment as well as completeness of second‐level coding. We calculated percent agreement, defined as both raters scoring a block of text with the same code divided by total number of codes. We also calculated the Spearman correlation between the 2 reviewers. We used descriptive statistics to assess the frequency of select codes and themes (see Supporting Information, Appendix 1 and Appendix 2, in the online version of this article).[9, 12, 13]

RESULTS

Over a 3‐week study period, 47 comments were submitted by 37 respondents. This yielded 148 codable statements (Table 1). Despite limited information on respondents, we ascertained from Facebook that 32 (86%) were women and 5 (14%) were men.

From coded text, several broad themes were identified (see Table 1 for representative quotes): (1) comments about staff (17/37 respondents, 45.9%). These included positive descriptions of efficiency, caring behavior, good training, and good communication, whereas negative comments included descriptions of unfriendliness, apparent lack of caring, inattentiveness, poor training, unprofessional behavior, and poor communication; (2) comments about specific departments (22/37, 59.5%); (3) comments on technical aspects of care, including perceived errors, incorrect diagnoses, and inattention to pain control (9/37, 24.3%); and (4) comments describing the hospital physical plant, parking, and amenities (9/37, 24.3%). There were a few miscellaneous comments that did not fit into these broad themes, such as expressions of gratitude for our solicitation of narratives. Percent agreement between coders was 80% and Spearman's Rho was 0.82 (p<0.001).

A small number (n=3) of respondents repeatedly made comments over the 3‐week period, accounting for 30% (45/148) of codes. These repetitive commenters tended to dominate the Facebook conversation, at times describing the same experience more than once.

DISCUSSION

In this study evaluating the potential utility of social media as a hospital QI tool, several broad themes emerged. From these themes, we identified several areas that could be deemed as QI targets, including: training staff to be more responsive and sensitive to patients needs and concerns, improving patient and visitor parking, and reducing emergency department waiting times. However, the insight gained from solicited Facebook comments was similar to feedback gained from more traditional approaches of soliciting patient perspectives on care, such as patient experience surveys.[14]

Our findings should be viewed in the context of prior work focused on patient narratives in healthcare. Greaves et al. used sentiment analysis to describe the content of nearly 200,000 tweets (comments posted on the social networking website Twitter) sent to National Health Service (NHS) hospitals.[15] Themes were similar to those found in our study: (1) interaction with staff, (2) environment and facilities, and (3) issues of access and timeliness of service. Notably, these themes mirrored prior work examining narratives at NHS hospitals[3] and were similar to domains of commonly used surveys of patient experience.[14] The authors noted that there were issues with the signal to noise ratio (only about 10% of tweets were about quality) and the enforced brevity of Twitter (tweets must be 140 characters or less). These limitations suggest that using Twitter to identify QI targets would be difficult.

In contrast to Greaves et al., we chose to solicit feedback on our hospital's Facebook page. Facebook does not have Twitter's enforced brevity, allowing for more detailed narratives. In addition, we did not encounter the signal‐to‐noise problem, because our prompt was designed to request feedback that was relevant to recent experiences of care. However, a few respondents dominated the conversation, supporting the hypothesis that those most likely to comment may be the patients or families who have had the best or worst experiences. In the future, we will attempt to address this limitation and reduce the influence of repeat commenters by changing our prompt (eg, Please tell us about your experience, but please do not post descriptions of the same experience more than once.).

This pilot demonstrated some of the previously described benefits of online narratives.[5] First, there appears to be value in allowing patients to share their experiences and to read the experiences of others (as indicated in a few grateful patients comments). Second, soliciting online narratives offers a way for hospitals to demonstrate a commitment to transparency. Third, in contrast to closed‐ended survey questions, narrative comments help to identify why patients were satisfied or unsatisfied with their care. Although some surveys with closed‐ended questions also allow for narratives, these comments may or may not be carefully reviewed by the hospital. Using social media to solicit and respond to comments enhances existing methods for evaluating patient experience by engaging patients in a public space, which increases the likelihood that hospitals will attempt to improve care in response.

Notably, none of the identified areas for improvement could be considered novel QI targets for BMC. For example, our hospital has been very focused on training staff around patient experience, and emergency department wait times are the focus of a system‐wide improvement effort called Patient Progress.

This study has other limitations. We conducted this study over a 3‐week time period in a single center and on a single social media site whose members may not be representative of the overall patient population at BMC. Although we do not know how generalizable our findings are (in terms of identifying QI targets), we feel that we have demonstrated how using social media to collect data on patient experience is feasible and could be informative for other hospitals in other locations. It is possible that we did not allow the experiment to run long enough; a longer time or broader outreach (eg, a handout given to every discharged patient over a longer period) may be needed to allow patients adequate opportunity to comment. Of note, we did not specifically examine responses by time period, but it does seem, in post hoc analysis, that after 2 weeks of data collection we reached theoretical saturation with no new themes emerging in the third week (eg, third‐week comments included I heart your nurses. and Love Baystate but hate the ER.). More work is also needed that includes a broader range of social media platforms and more participating hospitals.

In conclusion, the opportunity to provide feedback on Facebook has the potential to engage and empower patients, and hospitals can use these online narratives to help to drive improvement efforts. Yet potential benefits must be weighed against reputational risks, a lack of representative respondents, and the paucity of novel QI targets obtained in this study.

Disclosures: Dr. Lagu is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number K01HL114745. The authors report no conflicts of interest.

Patient experience has become a major component of the Center for Medicare and Medicaid Services Value‐Based Purchasing initiative.[1] Hospitals have therefore focused quality improvement (QI) efforts on this area.[2] Hospital performance in the realm of patient experience is generally determined using systematic surveys with closed‐ended questions, but patient‐generated narrative feedback can help hospitals identify the components of care that contribute to patient satisfaction and or are in need of improvement.[3] Online narrative responses posted by patients on rating websites or social media have been criticized because they may not be representative of the population,[4] but they also have some advantages.[5] Any patient may leave a comment, not just those who are selected for a survey. Patients may also experience benefits through the act of sharing their story with others. Moreover, most US hospitals use some form of social media,[6] which they can theoretically use to self‐collect narrative data online. To realize the full potential of patient‐generated online narratives, we need a clearer understanding of the best practices for collecting and using these narratives. We therefore solicited patient feedback on the Facebook page of a large tertiary academic medical center to determine whether it is feasible to use social media platforms for learning about and improving hospital quality.

METHODS

Baystate Medical Center (BMC) is a tertiary care medical center in western Massachusetts. We identified key BMC stakeholders in the areas of QI and public affairs. Noting that patients have expressed interest in leaving comments via social media,[7] the group opted to perform a pilot study to obtain patient narratives via a Facebook prompt (Facebook is a social media site used by an estimated 58% of US adults[8]). The BMC public affairs department delivered a press release to the local media describing a 3‐week period during which patients were invited to leave narrative feedback on the BMC Facebook wall. The BMC Institutional Review Board deemed that this study did not constitute human subjects research.

During March 2014 (March 10, 2014March 24, 2014), we posted (once a week) an open‐ended prompt on BMC's Facebook wall. The prompt was designed to elicit novel descriptions of patient experience that could help to drive QI. It read: We want to hear about your experiences. In the comment section below, please tell us what we do well and how we can improve your care. Because of concerns about the potential reputational risks of allowing open feedback on a public social media page, the prompt also reminded patients of the social media ground rules: there should be no mention of specific physicians, nurses, or other caregivers by name (for liability reasons); and patients should not include details about their medical history (for privacy reasons).

We collected all posts to preserve comments and used directed qualitative content analysis to examine them.[9] Two research team members[3, 10, 11] independently coded the responses. Starting with an a priori codebook that was developed during a previous study,[3] they amended the codebook through an iterative process to incorporate new concepts. After independently coding all blocks of text, the coders reviewed their coding selections and resolved discrepancies through discussion. We then performed second‐level coding, in which codes were organized into major pertinent themes. We reviewed the coded text after applying secondary codes in order to check for accuracy of coding and theme assignment as well as completeness of second‐level coding. We calculated percent agreement, defined as both raters scoring a block of text with the same code divided by total number of codes. We also calculated the Spearman correlation between the 2 reviewers. We used descriptive statistics to assess the frequency of select codes and themes (see Supporting Information, Appendix 1 and Appendix 2, in the online version of this article).[9, 12, 13]

RESULTS

Over a 3‐week study period, 47 comments were submitted by 37 respondents. This yielded 148 codable statements (Table 1). Despite limited information on respondents, we ascertained from Facebook that 32 (86%) were women and 5 (14%) were men.

From coded text, several broad themes were identified (see Table 1 for representative quotes): (1) comments about staff (17/37 respondents, 45.9%). These included positive descriptions of efficiency, caring behavior, good training, and good communication, whereas negative comments included descriptions of unfriendliness, apparent lack of caring, inattentiveness, poor training, unprofessional behavior, and poor communication; (2) comments about specific departments (22/37, 59.5%); (3) comments on technical aspects of care, including perceived errors, incorrect diagnoses, and inattention to pain control (9/37, 24.3%); and (4) comments describing the hospital physical plant, parking, and amenities (9/37, 24.3%). There were a few miscellaneous comments that did not fit into these broad themes, such as expressions of gratitude for our solicitation of narratives. Percent agreement between coders was 80% and Spearman's Rho was 0.82 (p<0.001).

A small number (n=3) of respondents repeatedly made comments over the 3‐week period, accounting for 30% (45/148) of codes. These repetitive commenters tended to dominate the Facebook conversation, at times describing the same experience more than once.

DISCUSSION

In this study evaluating the potential utility of social media as a hospital QI tool, several broad themes emerged. From these themes, we identified several areas that could be deemed as QI targets, including: training staff to be more responsive and sensitive to patients needs and concerns, improving patient and visitor parking, and reducing emergency department waiting times. However, the insight gained from solicited Facebook comments was similar to feedback gained from more traditional approaches of soliciting patient perspectives on care, such as patient experience surveys.[14]

Our findings should be viewed in the context of prior work focused on patient narratives in healthcare. Greaves et al. used sentiment analysis to describe the content of nearly 200,000 tweets (comments posted on the social networking website Twitter) sent to National Health Service (NHS) hospitals.[15] Themes were similar to those found in our study: (1) interaction with staff, (2) environment and facilities, and (3) issues of access and timeliness of service. Notably, these themes mirrored prior work examining narratives at NHS hospitals[3] and were similar to domains of commonly used surveys of patient experience.[14] The authors noted that there were issues with the signal to noise ratio (only about 10% of tweets were about quality) and the enforced brevity of Twitter (tweets must be 140 characters or less). These limitations suggest that using Twitter to identify QI targets would be difficult.

In contrast to Greaves et al., we chose to solicit feedback on our hospital's Facebook page. Facebook does not have Twitter's enforced brevity, allowing for more detailed narratives. In addition, we did not encounter the signal‐to‐noise problem, because our prompt was designed to request feedback that was relevant to recent experiences of care. However, a few respondents dominated the conversation, supporting the hypothesis that those most likely to comment may be the patients or families who have had the best or worst experiences. In the future, we will attempt to address this limitation and reduce the influence of repeat commenters by changing our prompt (eg, Please tell us about your experience, but please do not post descriptions of the same experience more than once.).

This pilot demonstrated some of the previously described benefits of online narratives.[5] First, there appears to be value in allowing patients to share their experiences and to read the experiences of others (as indicated in a few grateful patients comments). Second, soliciting online narratives offers a way for hospitals to demonstrate a commitment to transparency. Third, in contrast to closed‐ended survey questions, narrative comments help to identify why patients were satisfied or unsatisfied with their care. Although some surveys with closed‐ended questions also allow for narratives, these comments may or may not be carefully reviewed by the hospital. Using social media to solicit and respond to comments enhances existing methods for evaluating patient experience by engaging patients in a public space, which increases the likelihood that hospitals will attempt to improve care in response.

Notably, none of the identified areas for improvement could be considered novel QI targets for BMC. For example, our hospital has been very focused on training staff around patient experience, and emergency department wait times are the focus of a system‐wide improvement effort called Patient Progress.

This study has other limitations. We conducted this study over a 3‐week time period in a single center and on a single social media site whose members may not be representative of the overall patient population at BMC. Although we do not know how generalizable our findings are (in terms of identifying QI targets), we feel that we have demonstrated how using social media to collect data on patient experience is feasible and could be informative for other hospitals in other locations. It is possible that we did not allow the experiment to run long enough; a longer time or broader outreach (eg, a handout given to every discharged patient over a longer period) may be needed to allow patients adequate opportunity to comment. Of note, we did not specifically examine responses by time period, but it does seem, in post hoc analysis, that after 2 weeks of data collection we reached theoretical saturation with no new themes emerging in the third week (eg, third‐week comments included I heart your nurses. and Love Baystate but hate the ER.). More work is also needed that includes a broader range of social media platforms and more participating hospitals.

In conclusion, the opportunity to provide feedback on Facebook has the potential to engage and empower patients, and hospitals can use these online narratives to help to drive improvement efforts. Yet potential benefits must be weighed against reputational risks, a lack of representative respondents, and the paucity of novel QI targets obtained in this study.

Disclosures: Dr. Lagu is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number K01HL114745. The authors report no conflicts of interest.

Patient experience has become a major component of the Center for Medicare and Medicaid Services Value‐Based Purchasing initiative.[1] Hospitals have therefore focused quality improvement (QI) efforts on this area.[2] Hospital performance in the realm of patient experience is generally determined using systematic surveys with closed‐ended questions, but patient‐generated narrative feedback can help hospitals identify the components of care that contribute to patient satisfaction and or are in need of improvement.[3] Online narrative responses posted by patients on rating websites or social media have been criticized because they may not be representative of the population,[4] but they also have some advantages.[5] Any patient may leave a comment, not just those who are selected for a survey. Patients may also experience benefits through the act of sharing their story with others. Moreover, most US hospitals use some form of social media,[6] which they can theoretically use to self‐collect narrative data online. To realize the full potential of patient‐generated online narratives, we need a clearer understanding of the best practices for collecting and using these narratives. We therefore solicited patient feedback on the Facebook page of a large tertiary academic medical center to determine whether it is feasible to use social media platforms for learning about and improving hospital quality.

METHODS

Baystate Medical Center (BMC) is a tertiary care medical center in western Massachusetts. We identified key BMC stakeholders in the areas of QI and public affairs. Noting that patients have expressed interest in leaving comments via social media,[7] the group opted to perform a pilot study to obtain patient narratives via a Facebook prompt (Facebook is a social media site used by an estimated 58% of US adults[8]). The BMC public affairs department delivered a press release to the local media describing a 3‐week period during which patients were invited to leave narrative feedback on the BMC Facebook wall. The BMC Institutional Review Board deemed that this study did not constitute human subjects research.

During March 2014 (March 10, 2014March 24, 2014), we posted (once a week) an open‐ended prompt on BMC's Facebook wall. The prompt was designed to elicit novel descriptions of patient experience that could help to drive QI. It read: We want to hear about your experiences. In the comment section below, please tell us what we do well and how we can improve your care. Because of concerns about the potential reputational risks of allowing open feedback on a public social media page, the prompt also reminded patients of the social media ground rules: there should be no mention of specific physicians, nurses, or other caregivers by name (for liability reasons); and patients should not include details about their medical history (for privacy reasons).

We collected all posts to preserve comments and used directed qualitative content analysis to examine them.[9] Two research team members[3, 10, 11] independently coded the responses. Starting with an a priori codebook that was developed during a previous study,[3] they amended the codebook through an iterative process to incorporate new concepts. After independently coding all blocks of text, the coders reviewed their coding selections and resolved discrepancies through discussion. We then performed second‐level coding, in which codes were organized into major pertinent themes. We reviewed the coded text after applying secondary codes in order to check for accuracy of coding and theme assignment as well as completeness of second‐level coding. We calculated percent agreement, defined as both raters scoring a block of text with the same code divided by total number of codes. We also calculated the Spearman correlation between the 2 reviewers. We used descriptive statistics to assess the frequency of select codes and themes (see Supporting Information, Appendix 1 and Appendix 2, in the online version of this article).[9, 12, 13]

RESULTS

Over a 3‐week study period, 47 comments were submitted by 37 respondents. This yielded 148 codable statements (Table 1). Despite limited information on respondents, we ascertained from Facebook that 32 (86%) were women and 5 (14%) were men.

From coded text, several broad themes were identified (see Table 1 for representative quotes): (1) comments about staff (17/37 respondents, 45.9%). These included positive descriptions of efficiency, caring behavior, good training, and good communication, whereas negative comments included descriptions of unfriendliness, apparent lack of caring, inattentiveness, poor training, unprofessional behavior, and poor communication; (2) comments about specific departments (22/37, 59.5%); (3) comments on technical aspects of care, including perceived errors, incorrect diagnoses, and inattention to pain control (9/37, 24.3%); and (4) comments describing the hospital physical plant, parking, and amenities (9/37, 24.3%). There were a few miscellaneous comments that did not fit into these broad themes, such as expressions of gratitude for our solicitation of narratives. Percent agreement between coders was 80% and Spearman's Rho was 0.82 (p<0.001).

A small number (n=3) of respondents repeatedly made comments over the 3‐week period, accounting for 30% (45/148) of codes. These repetitive commenters tended to dominate the Facebook conversation, at times describing the same experience more than once.

DISCUSSION

In this study evaluating the potential utility of social media as a hospital QI tool, several broad themes emerged. From these themes, we identified several areas that could be deemed as QI targets, including: training staff to be more responsive and sensitive to patients needs and concerns, improving patient and visitor parking, and reducing emergency department waiting times. However, the insight gained from solicited Facebook comments was similar to feedback gained from more traditional approaches of soliciting patient perspectives on care, such as patient experience surveys.[14]

Our findings should be viewed in the context of prior work focused on patient narratives in healthcare. Greaves et al. used sentiment analysis to describe the content of nearly 200,000 tweets (comments posted on the social networking website Twitter) sent to National Health Service (NHS) hospitals.[15] Themes were similar to those found in our study: (1) interaction with staff, (2) environment and facilities, and (3) issues of access and timeliness of service. Notably, these themes mirrored prior work examining narratives at NHS hospitals[3] and were similar to domains of commonly used surveys of patient experience.[14] The authors noted that there were issues with the signal to noise ratio (only about 10% of tweets were about quality) and the enforced brevity of Twitter (tweets must be 140 characters or less). These limitations suggest that using Twitter to identify QI targets would be difficult.

In contrast to Greaves et al., we chose to solicit feedback on our hospital's Facebook page. Facebook does not have Twitter's enforced brevity, allowing for more detailed narratives. In addition, we did not encounter the signal‐to‐noise problem, because our prompt was designed to request feedback that was relevant to recent experiences of care. However, a few respondents dominated the conversation, supporting the hypothesis that those most likely to comment may be the patients or families who have had the best or worst experiences. In the future, we will attempt to address this limitation and reduce the influence of repeat commenters by changing our prompt (eg, Please tell us about your experience, but please do not post descriptions of the same experience more than once.).

This pilot demonstrated some of the previously described benefits of online narratives.[5] First, there appears to be value in allowing patients to share their experiences and to read the experiences of others (as indicated in a few grateful patients comments). Second, soliciting online narratives offers a way for hospitals to demonstrate a commitment to transparency. Third, in contrast to closed‐ended survey questions, narrative comments help to identify why patients were satisfied or unsatisfied with their care. Although some surveys with closed‐ended questions also allow for narratives, these comments may or may not be carefully reviewed by the hospital. Using social media to solicit and respond to comments enhances existing methods for evaluating patient experience by engaging patients in a public space, which increases the likelihood that hospitals will attempt to improve care in response.

Notably, none of the identified areas for improvement could be considered novel QI targets for BMC. For example, our hospital has been very focused on training staff around patient experience, and emergency department wait times are the focus of a system‐wide improvement effort called Patient Progress.

This study has other limitations. We conducted this study over a 3‐week time period in a single center and on a single social media site whose members may not be representative of the overall patient population at BMC. Although we do not know how generalizable our findings are (in terms of identifying QI targets), we feel that we have demonstrated how using social media to collect data on patient experience is feasible and could be informative for other hospitals in other locations. It is possible that we did not allow the experiment to run long enough; a longer time or broader outreach (eg, a handout given to every discharged patient over a longer period) may be needed to allow patients adequate opportunity to comment. Of note, we did not specifically examine responses by time period, but it does seem, in post hoc analysis, that after 2 weeks of data collection we reached theoretical saturation with no new themes emerging in the third week (eg, third‐week comments included I heart your nurses. and Love Baystate but hate the ER.). More work is also needed that includes a broader range of social media platforms and more participating hospitals.

In conclusion, the opportunity to provide feedback on Facebook has the potential to engage and empower patients, and hospitals can use these online narratives to help to drive improvement efforts. Yet potential benefits must be weighed against reputational risks, a lack of representative respondents, and the paucity of novel QI targets obtained in this study.

Disclosures: Dr. Lagu is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number K01HL114745. The authors report no conflicts of interest.