Can Zhao
dellmock.png

Dell's Chatbot

 

Usability Analysis of Dell's Chatbot Ava

dellmock.png

Overview

What I Did: Heuristic Evaluation, Recruiting, Moderating Interviews, co-designing Pre-test questionnaire and Testing Tasks, Running SUS (System Usability Scale) scores, Communicating with the client at Dell
Tools: Illustrator, Sketch, Validately

 

Team Members

Mitchell Fajardo
Can Zhao
Ray Hong
Vijay Farmah

Research Challenge

What is Ava’s current state of efficacy from a user’s perspective comparing against its competitors?
How might Ava offer a better retail experience?

Timeline

Dell Presentation for Final Presentation (3).png

Problem Setting

robot.png
 

Background

Organizations and corporates are recently getting interested in Artificial Intelligence. Using chatbot as assistant on a company's website is a demonstration of involving this field, while also saving the cost to hire representatives. I am especially interested to learn users' behavior when chatbot intends to help them make purchasing decisions. Can users put trust in artificial intelligence during retail experience? 

 
ituser.png
 

pARTICIPANTS Selection Criteria

  • Business IT users

  • Has bought an IT system in the past year

  • Have used an automated chat feature before in a help or purchase or account management context (Optional)

  • Experience configuring hardware on data systems and platforms (optional)

  • Makes or advises on direct purchases of IT equipment (optional)

 

Product

Ava is a virtual shopping assistant on Dell.com. A user can type a message or clicks provided links to start chatting with Ava. Currently Ava is designed to assist users to explore and purchase dell's products on Dell's website.

 
product.png
 

Interaction Map of Ava's Current Functions

In order to understand Ava's functionality, I examined how Ava works and co-designed an interaction map as follows.

Flows to Test

After we examined the major functionality of Ava, we decided to test the first flow: help user choose a computer, since it exemplifies how Ava recommends a product through a couple of questions.

 

Methodology

Overview

 

Our study involved a competitor analysis, testing the usability of Dell’s chatbot (Ava) and HP’s chatbot (the HP Virtual Assistant). Our analysis was non-comparative competitor evaluation. hile Ava was designed to enhance shopping experience, the HP Virtual Assistant was designed to provide tech support. Thus, a one-to-one comparison between the two products is inappropriate. Instead, we evaluated the key delights and pain points as participants interacted with both chatbots, in order to gain insights on how each chatbot has been successful and how the next iteration of Ava could be improved.

Measuring scheme: Qualitative metrics such as perceived ease of use, usefulness, confidence, and degree of satisfaction, and Quantitative metrics through SUS. 

Testing Details

 I've covered this participant's face for privacy purposes.

I've covered this participant's face for privacy purposes.

1. Recruited Participants
3 males + 2 females = 5 business IT professionals (Occupations include: computing manager, business analyst, solutions architect, etc.);


2. Testing Environment
All usability sessions were held remotely, moderated through Validately. As the example shows on the right, participants shared their screens while interacting with Ava.

3. Duration
1 hour maximum per session

 
verses.png

4. Procedure
Participants answered a few pretest questions describing their most recent experience purchasing an IT product
Participants completed tasks testing Ava and the HP Virtual Assistant and were encouraged to think aloud.  After completing all tasks, we debriefed participants to discuss their overall impressions of both chatbots, things they liked, things they disliked, etc.

(Note: Conditions were counterbalanced, such that 3 participants completed the Dell tasks first, while the other 2 completed the HP task first)

 

HP Tasks:
(1) Imagine that you are having trouble connecting your wireless printer. Using HP’s virtual assistant, find a solution to your problem.
(2) Rate the HP Virtual Assistant using the SUS
 

Dell Tasks:
(1) Purchase a monitor for your home office
(2) Using Ava, purchase a computer for your home office
(3) Rate Ava using the System Usability Scale (SUS)

 

Summary of Findings

Responsiveness and Credibility

 

Users doubted the responsiveness and credibility of Ava (and chatbots in general). Participants felt that Ava could not fully address their needs when recommending a product.

 
It’s not allowing me to put in things I think are important to me...I can answer her questions but it doesn’t let me prioritize the things I want.
— Participant
 

In addition, one participant commented that he had concerns as to whether Ava was recommending the best product for his needs or the product that would yield the most profit for Dell. Thus, there is a need for Ava to demonstrate better credibility and responsiveness to users’ needs. We recommend that designers consider referring to reviews to establish credibility and making clear connections on how a product meets the needs expressed by users. Additionally,  Ava currently only recommends one product after the user answers a series of questions. 4 out of 5 participants either conveyed intentions or did compare options when making purchasing decisions. We recommend future research to examine how many options should be available to users.

 
You (Dell) obviously are trying to sell me something. Are you really telling me the best thing for me, or is this really a machine you will make the most money on if I buy it? It would take a LOT of trust in the salesperson for you to buy what they tell you. You’d want to shop around, compare, talk to friends, take several different options for a ‘test drive’ – not just accept what they (Dell) told you without skepticism.
— Participant

 

The position of Chatbox

 

Additional issues relate to difficulties interacting with Ava’s chatbox. Multiple participants commented that Ava’s chatbox obstructed their view of the web page and experienced friction switching between interactions with the chatbot and interactions with the main web page. We recommend redesigning the chatbot so that it is not in the way of the web page, and to provide affordances to make it clearer how to interact with Ava as part of the web page.

Which one should I interact with, the website or the chatbox?
— Participant
 

System Usability Scale Results

SCALE.png
 

Q1. Would like to use frequently
Q2. Unnecessarily complex
Q3. Easy to use
Q4. Need support of a technical person to be able to use
Q5. Functions are well integrated
Q6. Too much of inconsistency
Q7. Able to learn system quickly
Q8. Cumbersome to use
Q9. Feel confident in using system
Q10.Need to learn before using system
 

 

Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average. Dell scores 66, which is slightly above HP. Both Dell and HP's chatbots are under average.