The hot topic right now is ChatGPT, but it comes with a few drawbacks. We compared its capabilities to a human by asking it to write a usability testing discussion guide. Who will come out better?
In our ever-growing world of digital products, user experience (UX) research is becoming even more important to help businesses enhance the design quality of their products and services – and better meet the needs of their audience.
Our UX designers have plenty of handy tech tools in our UX research arsenal, including one new AI-assisted language generation chatbot that’s taking the world by storm: ChatGPT.
But what is ChatGPT and how can it assist with UX research? Will its rising popularity really revolutionise the future of UX research? Or is it just another tech fad?
ChatGPT has been built by OpenAI and is short for Chat Generative Pre-trained Transformer — a language generator model powered by huge amounts of deep-learning algorithms for human-response text data. In other words, it’s a personal assistant chatbot that uses AI to improve the ways it string sentences together in a more conversational and human-like manner.
After its initial release just a few months back in November 2022, ChatGPT managed to reach a million-users milestone in just five days. That’s pretty impressive stuff – especially when considering that recent research shows 72% of people find chatbots frustrating and a waste of time. So what’s so different about this one? Is the craze justified?
In terms of UX, the advantages of ChatGPT are pretty clear, and rather exciting: it’s a super versatile tool with the ability to automate all sorts of UX-focused tasks — not just informed data-led user research — but also copywriting, creative inspiration, idea generation, sentiment analysis, and general UX advising.
For example, you can ask ChatGPT: “What are some potential biases in UX design?” to uncover certain blindspots your research hasn’t yet reached.
Or you could type: ‘Write 20 survey questions for UX research’ and it’ll give you a variety of closed and open-end questions you can add to a questionnaire.
You can also ask: “What are the latest behaviours in online shopping? to get plenty of insights for your research on current e-commerce trends.
Or let’s say you’re creating a new app and need some inspiration in the early stages of the design process – simply type: ‘Make a wireframe for a fitness app.’ or ‘Create an information architecture for a dating app.’ Yep, that’s right, ChatGPT can give you its own interpretation of a UX model – albeit a more basic, bare-bones type.
Pretty cool, right?
You can even get ChatGPT to help write or re-write an entire website’s worth of copy. Simply ask things like: ‘Write my button copy that will persuade newsletter sign-ups.’ or ‘Create a page title based on the best keywords for UX’.
As a super advanced dialogue designer, Chat GPT’s text-writing ability is perhaps its most valuable UX feature. It can be fine-tuned on specific datasets to generate all sorts of realistic, yet hypothetical, scenarios and dialogue for user testing – whether consumer-based text like product descriptions and customer service responses or actual website copy for pages like your FAQs or HTTP error pages.
Just imagine how many worthwhile questions you’d be able to ask in your UX survey when you combine minds with ChatGTP…or how genuinely useful a FAQs page might be when you’ve asked ChatGPT to research what your target audience tends to ask most about your product type.
There are countless ways that you can use ChatGTP to improve your user testing know-how – and it’s even more impressive when you consider how speedy ChatGPT responses are. We're talking automatic responses to complex requests in just a few seconds: certainly less time than it takes for the average human to produce substantial answers and results. The nature of ChatGTP is also really conversational, meaning you don’t have to keep re-entering your requests and it’s smart enough for you to ask it to rewrite/redraft according to amendments you want to make. Just having a foundation to work from based on the ideas it generates is helpful in itself, even if what it produces isn’t quite the perfect finished piece that you’re after.
Back to the battle…
To give a little more context, we used ChatGPT to create a discussion guide for a usability test of a fuel ordering mobile app (a discussion guide is basically a set of pre-determined questions and topics aimed to be spoken about during a user interview). The guide also needed to include a set of tasks, follow-up questions, and a debriefing.
We essentially wanted to see what kind of content ChatGPT would come up with, the quality at which it was written, and how it compares to the type of trusty human-written discussion guides that we routinely create, here at Digital of Things.
Don’t get us wrong, we were immediately impressed with its quickfire responses, but issues started to arise when we delved a little deeper.
The results? Pros and cons…
While using ChatGPT was certainly time-efficient, we did have to ask the bot to write a more personalised discussion guide and include a debriefing section (which it missed out the first time around). Not to mention consciously watching our wording to make sure the request was phrased clearly and correctly enough for it to understand, and provide a good enough outcome.
Perhaps most concerning of all though is the potential biases in the training data that can affect the generated scenarios and dialogue of a discussion guide – particularly in terms of ethics when researchers using ChatGTP run an increased risk of perpetuating stereotypes or discrimination, whether intentionally or not.
For example, if ChatGPT is acquiring its answers and responses from a data set that’s not inherently diverse, you run the risk of generating dialogue that fails to represent all users. Understanding the average user research participant as a unique individual – despite their age, race, gender, etc. will always be an essential part of user testing, and we’re not quite sure ChatGTP understands the gravity of this fully.
We also think the bot fails to understand emotional context and therefore lacks a certain level of human-like response and conversation. For example, not being able to understand sarcasm and irony. Generally, there seem to be aspects of human-to-human conversation which AI tech, like ChatGPT, doesn’t fully grasp: the nuances, critical thinking, and ethical decision-making that underpins all kinds of user research.
The use of ChatGPT also comes with a very real warning: it can easily output incorrect facts that are hard to spot because the explanation is so convincingly natural. We recommend always fact-checking with Google before believing everything it says – you might be surprised just how much of a rapport you start building with the bot after only a few uses!
It may have been hailed as the world’s most advanced chatbot, but that doesn’t mean ChatGPT comes without its flaws. Its ability to generate large amounts of data for user testing is certainly exciting – and there’s no doubt it can play a foundational role in improving the UX design of digital products and services if used correctly – plus the rapid automated responses are extremely helpful when time’s not on a user researcher’s side. How could any busy UXer not benefit from practically having a first draft built for them? But ultimately, there’s a limit to ChatGTP’s magic. We think it’s a double-edged sword.
Think of it this way: although there are revolutionary AI-assisted writing tools like Grammarly and Wordtune, that doesn’t mean actual human editors have ceased to exist. We hardly see chatbots diminishing the need for user research experts like us any time soon.
If you’re wondering whether to use ChatGPT for your own user research, we recommend considering the pros and cons of the model when planning and strategising your next UX move. And more critically, it’s of the utmost importance to remain aware of the ethical risks involved in using AI-assisted tools like ChatGPT.
However, there’s no denying its impressive features. With the ChatGPT hype showing no signs of slowing down, it will be interesting to see what direction its creators, OpenAI, turn to next: What new features will crop up? Will its AI-learning wonders widen even further? Will its conversational dialogue become even more human-like?
Whatever the future holds, there’s no doubt ChatGPT will evolve quickly and hold even more of an influential presence across all sorts of industry sectors in the coming years. But what about in the UX space?
Well, we just want you to remember, there’s nothing more human than human when it comes to understanding the depth and breadth of a user’s needs, feelings, and desires when experiencing your products.
Human, hold your hands up in triumph!
We conducted a UX audit for three major airlines. The results? Well, even the most renowned airlines have room for improvement! Discover more.
An in-depth diary study is an effective way to get up close and personal with your participants. Our go-to guide helps you reap real results. Ready?
We took time out to chat with Sudipt Shah, our CEO and co-founder, as well as Doaa Badran, one of our talented UX researchers ahead of Digital of Things launch in the Kingdom of Saudi Arabia. Hereʼs everything they had to say about what the future holds as the business branches out…