Usability Analysis of Dell's Chatbot Ava
What I Did: Heuristic Evaluation, Recruiting, Moderating Interviews, co-designing Pre-test questionnaire and Testing Tasks, Running SUS (System Usability Scale) scores, Communicating with the client at Dell
Tools: Illustrator, Sketch, Validately
Organizations and corporates are recently getting interested in Artificial Intelligence. Using chatbot as assistant on a company's website is a demonstration of involving this field, while also saving the cost to hire representatives. I am especially interested to learn users' behavior when chatbot intends to help them make purchasing decisions. Can users put trust in artificial intelligence during retail experience?
pARTICIPANTS Selection Criteria
Business IT users
Has bought an IT system in the past year
Have used an automated chat feature before in a help or purchase or account management context (Optional)
Experience configuring hardware on data systems and platforms (optional)
Makes or advises on direct purchases of IT equipment (optional)
Ava is a virtual shopping assistant on Dell.com. A user can type a message or clicks provided links to start chatting with Ava. Currently Ava is designed to assist users to explore and purchase dell's products on Dell's website.
Interaction Map of Ava's Current Functions
In order to understand Ava's functionality, I examined how Ava works and co-designed an interaction map as follows.
Flows to Test
After we examined the major functionality of Ava, we decided to test the first flow: help user choose a computer, since it exemplifies how Ava recommends a product through a couple of questions.
Our study involved a competitor analysis, testing the usability of Dell’s chatbot (Ava) and HP’s chatbot (the HP Virtual Assistant). Our analysis was non-comparative competitor evaluation. hile Ava was designed to enhance shopping experience, the HP Virtual Assistant was designed to provide tech support. Thus, a one-to-one comparison between the two products is inappropriate. Instead, we evaluated the key delights and pain points as participants interacted with both chatbots, in order to gain insights on how each chatbot has been successful and how the next iteration of Ava could be improved.
Measuring scheme: Qualitative metrics such as perceived ease of use, usefulness, confidence, and degree of satisfaction, and Quantitative metrics through SUS.
1. Recruited Participants
3 males + 2 females = 5 business IT professionals (Occupations include: computing manager, business analyst, solutions architect, etc.);
2. Testing Environment
All usability sessions were held remotely, moderated through Validately. As the example shows on the right, participants shared their screens while interacting with Ava.
1 hour maximum per session
Participants answered a few pretest questions describing their most recent experience purchasing an IT product
Participants completed tasks testing Ava and the HP Virtual Assistant and were encouraged to think aloud. After completing all tasks, we debriefed participants to discuss their overall impressions of both chatbots, things they liked, things they disliked, etc.
(Note: Conditions were counterbalanced, such that 3 participants completed the Dell tasks first, while the other 2 completed the HP task first)
(1) Imagine that you are having trouble connecting your wireless printer. Using HP’s virtual assistant, find a solution to your problem.
(2) Rate the HP Virtual Assistant using the SUS
(1) Purchase a monitor for your home office
(2) Using Ava, purchase a computer for your home office
(3) Rate Ava using the System Usability Scale (SUS)
Summary of Findings
Responsiveness and Credibility
Users doubted the responsiveness and credibility of Ava (and chatbots in general). Participants felt that Ava could not fully address their needs when recommending a product.
In addition, one participant commented that he had concerns as to whether Ava was recommending the best product for his needs or the product that would yield the most profit for Dell. Thus, there is a need for Ava to demonstrate better credibility and responsiveness to users’ needs. We recommend that designers consider referring to reviews to establish credibility and making clear connections on how a product meets the needs expressed by users. Additionally, Ava currently only recommends one product after the user answers a series of questions. 4 out of 5 participants either conveyed intentions or did compare options when making purchasing decisions. We recommend future research to examine how many options should be available to users.
The position of Chatbox
Additional issues relate to difficulties interacting with Ava’s chatbox. Multiple participants commented that Ava’s chatbox obstructed their view of the web page and experienced friction switching between interactions with the chatbot and interactions with the main web page. We recommend redesigning the chatbot so that it is not in the way of the web page, and to provide affordances to make it clearer how to interact with Ava as part of the web page.
System Usability Scale Results
Q1. Would like to use frequently
Q2. Unnecessarily complex
Q3. Easy to use
Q4. Need support of a technical person to be able to use
Q5. Functions are well integrated
Q6. Too much of inconsistency
Q7. Able to learn system quickly
Q8. Cumbersome to use
Q9. Feel confident in using system
Q10.Need to learn before using system
Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average. Dell scores 66, which is slightly above HP. Both Dell and HP's chatbots are under average.