Diagnotes Ease of Use Study

Client Background

Diagnotes is a preeminent provider of productivity software for the healthcare industry, improving clinician lives and increasing the quality of patient care by delivering outstanding value, continuous innovation and exceptional customer experience. They exist to make the lives of clinicians and those associated with healthcare organizations easier. Their goal is to streamline healthcare workflows, cultivate an online environment for better care team collaboration, and create intuitive healthcare-specific software.

The Diagnotes web and mobile applications provide a HIPAA compliant platform for doctors,nurses and other healthcare professionals to communicate, collaborate and share data centered around patient care. This results in an enhanced experience for healthcare professionals and patients, improved clinical outcomes, and cost savings.

Software to be Evaluated

The Diagnotes app is currently used at all IU Health facilities for healthcare professionals to securely communicate regarding patient care. The platform has taken the place of the paging system, so now the operators send all page messages to providers through Diagnotes. It is a new system that was officially rolled out in 2019 and has several updates that have changed the way it can be used.

Our team recently met with the entire Diagnotes team including their CEO, CTO, and programmers to review their app and online platform. They have created a test environment for our team to work in so those not using the app clinically will have a better understanding of the features and processes used by end users.

Purpose, Goals and Objectives of the Study

The purpose of this study is to evaluate the utilization and ease of use of the Diagnotes mobile application in order to identify potential areas for improvement and future design expansion.

Overall Objectives

• Assess the effectiveness of Diagnotes among a range of healthcare professionals to complete necessary HIPAA compliant communications
• Identify commonly used features that work well and are easy to use
• Evaluate features that are less commonly used and more difficult to navigate
• Create a usability protocol that could be replicated with future Diagnotes updates

Research Questions

1. How and why do healthcare professionals at Riley Hospital for Children use Diagnotes?
2. How and why do healthcare professionals at Riley Hospital for Children use Diagnotes Rooms?
3. What are the perceived barriers to ease of use within Diagnotes?

Conclusion

Based on our multi-modality evaluation, we found that users experienced issues both related to lack of familiarity with the features as well as technically difficult steps required to achieve their desired daily activities. The goal of the Diagnotes tools within the healthcare systems is to provide easy and quick communication between providers. This goal could be obtained through both training initiatives and simple modifications to the application design.

Recommendations

Based on a combination of the information gained during our task analysis and the think aloud protocol as well as the quantitative survey to evaluate familiarity with key features, we have compiled a short list of training and design recommendations. We appreciate that the Diagnotes products have been purchased and regulations have been applied by the IU Health system, therefore we attempted to determine how the teams could combine efforts to improve the usability and capitalize on the many features provided within the software.

Short, Interactive In-Person Training

It was evident that several of the functions of Diagnotes are not well known to all members of the teams we evaluated. During our interviews, some participants learned how to do the task by us asking them and then they tried several different steps. Several participants expressed that if there was educational training regarding the features, it would be best to be done in person since it is more interactive and there would be room for asking clarifying questions.

Several key features that could be included would be:

• Muting or removing yourself from a conversation and feedback about muted notifications
• Completing messages
• Creating or joining a room and how to communicate with a group within the room

In order to ensure a consistent message, a short and easily comprehensible video could be created and then distributed to “champions” throughout the hospital system. These champions could receive dedicated training from Diagnotes and could then be responsible for reviewing the video with groups of providers and answering any relevant questions. These champions could also be a source of continuous feedback to the Diagnotes team about how trainings are being received.

Streamline Completion of Messages

Many participants struggled with how to complete a message or found the pathway to complete a message to be cumbersome. Since the majority of participants use the Apple iOS platform, aligning the ease of deleting messages in the “Messages” app could aid incongruence with known patterns of clearing away messages.

Further Evaluation of Schedule

Based on our Phase 2 evaluation, we discovered that few users utilize the Schedule feature due to issues with finding the correct schedules and confidence in the accuracy of the listed providers. Therefore, many users are still taking extra steps outside of the Diagnotes app (searching Amion or calling the Operator) before using the app to send a message to the intended recipient.

Search Function In Rooms

The Search Functionality used to locate Rooms could provide some difficulty to the user since no options are provided. Enabling auto complete or providing a list of possible options once the user begins typing would reduce user frustration and increase their efficiency because they would not have to keep retyping the name of the Room they are trying to locate.

Top Menu Locations

The functions located on the top menu of the home page: Recent, Rooms, and Messages are some of the most used and important functions and features of the application. Having them located in a separate menu at the top of the page allows the user easy access to these functions. Adding this menu to all the other pages would increase user efficiency and add consistency to the overall design across the application.

Help Center and FAQ

Everyone will at sometime or another need help with the application. Being informed how the application works will increase efficiency and help the user perform the intended functions of the application. A survey could be conducted to confirm the current list of the FAQ is up-to-date and relevant on how the users work with the application.

Start Button Menu

The Start button gives users access to a menu of frequent and important functions typically performed with the application. A survey could be conducted to find out if these are indeed what the users are typically using the application for, or if other tasks are found to be done at a higher volume. These functions could then be included or replace what is currently available in the menu.

Consistency and Standards

Diagnotes could benefit from implementing small changes based on the usability heuristic of consistency and standards. This guideline states that users should not have to wonder what words, symbols or actions represent within an application. In short, platform conventions and consistency should be followed whenever possible. Diagnotes follows many mobile application standards and conventions very well, but some details of the app could be revisited. These areas include the main bottom navigation bar, the complete message icon and the Diagnotes icon acting as a menu button at the top left of some screens.

Bottom Navigation Bar

The bottom navigation bar gives users access to many important areas of the application. This includes a home button, schedules, directory and more. While this navigation element is present in the main tabs of the application (Recent, Room, Messages, Schedules, Directory and More) it is no longer present when a user enters a specific message or Room. To keep with the guidelines of consistency and standards the Diagnotes team may want to evaluate if it’s feasible to keep this navigation element present when inside a message or Room. As users become familiar with the application they may come to rely on the bottom navigation to move through the app. When this suddenly isn’t present it could cause confusion and user error.

Complete Message Icon

During testing, most users had either learned that the check mark icon represented the completion of a message or they intuitively identified this symbol with completion. The think aloud protocol also provided data that some users were confused by this button or didn’t naturally equate the symbol with completion. Providing text labels to this icon and others that may come across as unique could clear up confusion and provide another means of relaying user interface information.

Diagnotes Menu Icon

Consistency and Standards may also be useful in examining the use of the Diagnotes symbol to represent the menu when viewing messages and Rooms. This standard is widely used on web pages, but it may confuse users on the mobile platform. This icon could be mistaken as a non-interactive decorative element or simply as a home button as experienced on the web. A more recognizable menu button is the common hamburger icon, which consists of 3 stacked horizontal lines. Following this convention could give users clear indication that a menu is available when the icon is tapped. The use of heat maps of further task analysis sessions could give more information about how users interpret the current menu button.

Competitive Analysis

The two competitors chosen to be analyzed are OnPage and Trillian. OnPage is an incident alert management system with on-call capabilities that provides persistent notifications to ensure critical alerts are never missed. Trillian provides secure messaging that is HIPAA compliant to teams of all sizes that improves communication.

With all the features and functions available to the Diagnotes users they could become overwhelmed, especially for those who are not tech savvy. A lot of the extra functionality could go unused because the user is not aware the function exists or they do not know how to use it. Another downside could be the user does not have the time to properly learn how to use the application.

Heuristic Evaluation

The heuristic evaluation was conducted using Jakob Nielsen’s Ten Usability Heuristics and a Usability Aspect Report (UAR). A severity level was applied ranging from 0 - 4 (Not a problem to Usability Catastrophe) for each heuristic in order to assess the level at which it should be addressed. Justification was established by describing the Frequency, Impact and Persistence of the issue to determine when it should be addressed.

Phase One Evaluation

In Phase One, we performed 6 interviews with prompts to complete tasks and had them complete a survey that included demographics and a System Usability Survey.

Demographics of Phase 1 Participants

There were 6 participants who all worked at Riley Hospital for Children. There were 3 clinical faculty, 2 nurses, and one fellow. The ages of the participants ranged from 32 to 67 years. Most were female, white non-Hispanic and had worked in healthcare for more than 10 years. All used the Diagnotes app on an Apple iOS system. Regarding technical proficiency, 4 rated themselves as competent, 1 as proficient, and 1 as expert.

Task Analysis

The task analysis was completed by a total of 6 users within the IU Riley Hospital for Children. Users included male and female physicians and nurses. A summary of the results are presented followed by a table displaying the aggregated data for each task.

Task Difficulties

The results indicate that no users were able to successfully complete task 6, which asked participants to find a colleague on call on the Leukemia Services medical team. This task was also associated with the highest average number of errors. In addition to task 6, it could be beneficial to examine in more detail tasks 2, 3 and 5. While the average number of attempts and average error rates were not unusually high for these tasks 2 of the 6 participants unsuccessfully completed these items.

Task Successes

Findings also show that users were successful in completing the following tasks with a single attempt and no errors:

• Task 1: Send a message to Emily Mueller
• Task 7: Change your read receipt notifications to 25 minutes
• Task 8: Access the latest feature from the What's New Section

100% of participants were also successful in completing task 4: remove yourself from a group message, but the average number of attempts and average number of errors were higher than what was observed for task 1, 7 and 8.

Phase 2: Function Familiarity Evaluation

Based on our interview feedback, we felt it would be helpful to quantify the proportion of individuals familiar with each of the Diagnotes functions. Participants were asked to choose as many of the listed functions that they “knew how to perform.” All participants knew how to send a message to a single person. The least familiar functions included the Room options with only 7% knowing how to join a room.

We also tried to further quantify how participants were using the Diagnotes Directory and Schedules Features. We asked them how they find a colleague who is on call for their OWN service and for a DIFFERENT service. They could choose multiple options. Most providers (58.2%) chose to use Amion to find providers on their own service, while the most common tool to find providers on different services was to call the operator (41.1%).

Among the other options participants mentioned included reviewing notes or orders from clinicians to see who was caring for the patient or use Smartweb.

Phase 2: Other Feedback

Participants were given the option to add any additional feedback at the end of the survey. General themes from the comments included:

• Having a training program for how to use Diagnotes functions
  • Expressed unfamiliarity with the fact that it could do more than just send messages
  • Preference for in-person instruction rather than another email with too much information.
• There is a lack of consistency in usage between providers
  • We have to use the app on our personal phones, not all providers keep their phones with them at all times.
• Issues related to accuracy of on call listings
  • Times where individuals are listed, but not the correct person
  • Difficult to determine if you are contacting a resident, fellow or attending

Phase 2: Overall Perceived Usabilty

We calculated a mean SUS score (m) of 67.5 with a standard deviation (s) of 15.9. Using 56 degrees of freedom with 57 participants (n), we found the critical value of 1.673 for t in our t-test for a 95% (a) confidence interval. With the formula Margin of error = t 1−α * s / √n , we calculated the margin of error to be 3.5. Because the margin of error can be positive or negative to form a confidence interval, we are 95% confident that the true mean SUS Score falls between 64 to 71.1.

The sample of participants responding to the SUS questionnaire was collected and analyzed per the formulas above. Based on the confidence interval calculated above Diagnotes did have a score that overlapped with the lower limit of acceptability (70 or greater), but the means were both below 70. This indicates that there are opportunities for the Diagnotes team to identify areas for improvement.

Limitations

There are several limitations to our study that should be noted. First, for our competitive analysis, we were only able to access the free versions of the competitor’s software to compare and contrast. It is likely that the for-purchase software would have opened up more functionality that we were not able to evaluate. Also, we only interacted with participants who were part of a small subset of healthcare professionals at a single institution within the IU Health system. Therefore, their opinions may not represent the entire experience of all healthcare professionals within the IU Health system. Fortunately, we were able to gather information from a diverse set of participants within the Division of Pediatric Hematology Oncology at Riley.

Future Evaluations

We believe that our initial task analysis and talk aloud methods helped us gain enough insight in order to create a quantitative survey to assess both utilization patterns and familiarity with key features. This survey could be used across the entire IU Health system to further understand how the Diagnotes software is being utilized and which features could be better highlighted and leveraged for improved user experience and quality of communications.