(Audio included.) Snippets from the usability tests.

Project Sponsor

As a coordinated effort between Citizens Bank and Bentley University, a team of four UX practitioners from Bentley University conducted a usability test and a heuristic evaluation for Citizens Bank.

Problem Statement

Citizens Bank had just launched a new career informational website separate from its other third-party job application portal (Taleo). To improve the online job application process, the online job application experience across the two sites was reviewed for usability issues.

Objectives

A heuristic analysis and a usability test were conducted to address the following areas communicated by Citizens Bank.

• Content Usefulness: Do candidates have enough information to apply for the right role?
• Content Findability: Is the information where a candidate/user would expect it to be?
• Usability: What are the friction points?
• Impressions: Are we making candidates overthink and starting the wrong type of relationship?

Part 1: Heuristic Evaluation

Process

• Defining the users and the user tasks that the users would likely attempt.
• Walk through the tasks step-by-step through the lens of the user
• Checking against usability heuristics
• Assigning each heuristic violation a level of severity according to Nielsen’s severity rating scale
• Listing violations according to severity
• Making recommendations for improvements

Part 2: Usability Test

Method

The team conducted remote-moderated usability tests with 8 participants: 4 tested on mobile devices and 4 on desktop/laptop devices. During the test, observers recorded participants’ comments, navigation choices, task errors, and perceived usability through standardized test questions. After completing the test, participants were compensated with a $50 Amazon gift card (provided by Citizens Bank). 

Results

SEQ (Single Ease Question) and SUPR-Q (Standardized User Experience Percentile Rank Questionnaire) scores were calculated. Despite failing to reach significance through hypothesis testing, several participants’ comments preferred the desktop interface over the mobile interface. They also wished to rate the Career Site and Taleo Site separately because they wanted to assign a lower score to the Taleo website relative to the Career website.

Sentiment Analysis

Participant quotes gathered by open-ended answers and the think-aloud process were also coded as positive, negative, and neutral according to researchers’ interpretation of tone and facial expression elicited and adjectives used. Of the coded quotes, 29.6% were positive, 25.9% were neutral, and 44.4% were negative. Each coded quote was assigned a value to visualize participant sentiments by task. A value of 1 was assigned to positive, 0 to neutral, and -1 to negative.

Coding Scheme of Qualitative Quote Data

Code	Quote
Positive	“The most user-friendly part…[Is] only two tabs got me where I needed to be…”
Neutral	“It did not ask me to create a profile.”
Negative	“I’d look for saved there, but it’s not there…The next thing I’ll try is if ‘saved job’ will take me [where I want to go]. No, it does not.”

• Task 1: Find information on 401K benefits. 
• Task 2: Search for and save a “Sr. Credit Review Manager” position.
• Task 3: Find saved jobs. 
• Task 4: Apply for the Sr. Credit Review Manager position. (It transitioned the participant from the
  Citizens Bank Career website to the Taleo website.)
• Task 5: Search for and save a senior-level job (on the Taleo site).
• Task 6: Navigate to the saved jobs view.
• Task 7: Begin the application process to a saved job.

Tasks 1 and 2 generally generated more positive and neutral comments than negative comments from participants. During tasks 5 through 7, most comments from the Taleo website were negative and neutral rather than positive.

Recommendations

1. Insufficient Error Prevention

Participants in the usability study faced various error issues throughout the application process. The form fields on the Taleo site are not optimized to prevent errors. When errors are encountered, the error messages are not always visible and effective in cueing the corrective actions needed.

• Allow for flexible formatting within information fields. For example, the educational institution and the majors are currently selected from a predefined list. Applicants with multiple majors or international institutions cannot accurately enter information. Providing a text box that allows for any text entry and multiple major fields will alleviate this.
• Provide clear and explicit instructions for error correction. If possible, place the error messages near the field where the error occurred to immediately signal to the applicant which area needs to be corrected. 

 

2. “Saved Jobs” on Citizens Bank job site

All participants tested on the mobile interface successfully saved a job on the Citizens Bank site by clicking “Save this job”. However, zero participants were successful when instructed to find the recently saved job. The site offered participants little feedback once a job was saved. The only indication was that the “Save this job” button changed to “Saved Job”, but it was not obvious enough. Participants tried various navigation pathways to find a compilation of their saved jobs:

• Include a “Saved Jobs” button on the mobile site that is easily visible to the applicant. This will meet Citizens Bank’s business objective of Content Findability. 

3. Inconsistent Features – Find Jobs on Taleo Site 

Participants were instructed to find a job once after logging into the Taleo site. They proceeded with the task in one of two ways: attempting to navigate back to the Citizens Bank job page or searching directly from the Taleo site. While the Taleo site does provide a mechanism for viewing a list of available jobs, participants encountered pain points with searching. They did not understand the various features of the page. 

• Supply applicants with a search bar to better filter jobs.
• Include autofill in the search bar to promote recognition over recall.
• Use more universal descriptions for filters and sorting.
• Display available jobs and sorting options similar to those on the Citizens Bank Career site for consistency.

4. Cross-Site Communication

Participants shared frustration that jobs saved on the Citizens Bank site are inaccessible on the Taleo site and vice versa. An applicant can search for a job, read through the search results, select a job based on the information listed, and save the job only to have it lost once they attempt to log into Taleo. Applicants who save a job to Taleo but wish to read more about the job on the Citizens Bank site likewise have to repeat the search, read, select, and save process a second time. Applicants may struggle to remember what job they are looking for across the sites and may worry about duplicate applications for the same job.

• Implement cross-site communication to provide seamless information transfer, such as with saved jobs.
• If the third-party functionality provided by Taelo does not permit such implementation, remove the option to save the job from the Citizens Bank website completely, leaving “Apply Now” as the only option, or require applicants to sign in to the Taleo website to save jobs.

 

 

Back to the Objectives

Content Usefulness

The pages were often very wordy (Figure 11). On the Citizens Bank career site, the participants commented on the length and organization of the job descriptions, with some wishing that the descriptions were bulleted or separated with lines. On the Taleo site, some participants mentioned the site presented ” a lot of unnecessary information,” often referring to the various instructions and descriptions. A simple resume upload involves at least 230 instruction words, which some participants said they would not read.

The second point related to content usefulness is the presence of large and distracting header images on the Citizens Bank career site. While a participant commented that “the image is nice as it gets the point across,” three others commented that it was too big or distracting from the content they wanted to find. This is consistent with the findings in our heuristic analysis. 

Content Findability

The usability study revealed two main deterrents regarding findability: lack of information transfer and information architecture. As aforementioned, participants expressed annoyance and confusion when saved jobs were not transferred from one site to the other. This left participants feeling a lack of integration between sites and made it clear they were two separate entities. Based on navigation choices, many participants were unsure whether the Citizens Bank or Taleo site would provide the information they sought. For example, two participants returned to the Citizens Bank career site from the Taleo site to search for jobs. 

Impressions

In the SUPR-Q section of this study, the participants were asked to rate their impressions of both websites. However, some participants noted a difference in impressions between the Taleo and Citizens Bank sites. One participant mentioned, “The Taleo site does not seem as professional.”

The SUPR-Q questionnaire included a net promoter score analysis. The participants rated the Citizens Bank online application process poorly, with a score of -25. However, a larger sample size may be needed to gauge the score better. 

The participants were asked about their impressions of Citizens Bank before and after the study. Before this study, most participants had heard of Citizens Bank primarily as a brick-and-mortar store but did not know much about the company. After completing the study of the job portal, they do not report a major difference in impressions of Citizens Bank. However, a participant added that the job portal made her feel Citizens Bank is “very businesslike, methodical and maybe a little rigid.”

Impact

Although the career site was fairly new, both sites have since been revamped after we presented our findings. However, as of April 2024, there are still 2 separate sites with some of the same cross-site issues. 

Learning

Personally, my main takeaways were regarding conducting usability tests. A small sample size of 8 did not allow for true statistical significance for the quantitative parts of the study. Citizens Bank compensated each of the 8 participants with $50 Amazon vouchers. If we had communicated earlier the need for a larger sample size to conduct a more quantitative study, we might have been able to ask for a higher budget or at least break down the $400 budget into a higher number of vouchers for more participants.

Back to Top