What is THIS project?
I'm a Pokemon Geek. I cannot and never will beat the allegations. I've "caught them all", I've played every main-series video game, I've written manuscripts for original/fan games, and I cried when Ash Ketchum was retired as the TV show protagonist.
But I'm also a UX Researcher... and the newest Pokemon games were, in some ways, the best in the series! And in some ways, they were the worst. While I'm opinionated about Pokemon, I believe "I am not the user" - and so, this page outlines how I would research ways to improve Pokemon Scarlet and Violet versions (shortened to S/V).
Where to start research when you have opinions about the way forward?
While my feelings for S/V have evolved over time, I want always want to counter-balance or erase them as a researcher. Fortunately, I have seen many a more passionate opinion than mine shared online. With hundreds of hours of Pokemon Scarlet/Violet content consumed between YouTube, Reddit, Instagram, Discord, and other public and closed communities, I have a sense of some joys, pains, frustrations, and hopes for Pokemon Scarlet and Violet.
Intentionally scanning publicly available discussions within Pokemon communities online is a research method I've employed in past research called "digital eavesdropping" - essentially choosing to consider myself an outsider and learn more about the "current state" by observation of digital content. I've taken copious notes on gripes in the most recent game over the year it has been available.
If I had access to guaranteed participant time/attention, I'd use more data-rich methods to confirm my findings from this method such as interviews or think-aloud play tests. I could also choose to explicitly source opinions from power-users of the games, such as video content creators and influencers, if going the "subject matter expert" route, though I chose not to.
An example of an online Pokemon community in which I used the "digital eavesdropping" research method
Research can be fun! Ogerpon agrees.
We can't test EVERY idea. How can we cut scope down?
Kano Analysis surveys are fairly long, requiring 2-3 questions per idea to complete analysis. Without being able to compensate - or incentivize - people for their time, I wanted to keep the survey short and rely upon the intrinsic motivation of Pokemon Geekery for completion.
As such, including every idea found from Digital Eavesdropping would be irresponsible and risk response drop-off. Because I don't have the game design or development skills to create prototypes of specific ideas to test based upon this prioritization survey, I decided I could test many ideas by asking questions about THEMES of ideas instead of specific ideas.
Including specific ideas as examples of the idea-theme instead of the ideas themselves allowed me to test more ideas and give more room for creativity. Upon knowing the priority of general idea-themes, I could do future theoretical concept resonance testing on more specific improvements within the top priority themes.
Kano Analysis, I Choose You!
The correct method to validate/invalidate assumptions depends upon several factors including access to participant time/attention as well as the number of ideas required to test. Kano Analysis is a great way to test a wide variety of ideas quickly as relies upon survey responses that people can submit on their own time, requires no engineering or design lift, and can bridge the gap between collecting ideas and concept testing.
I've used Kano Analysis while at Flexcar to help product managers prioritize 64 features being considered for the second half of 2023. I believe in the analysis and model as tools to de-risk design and development efforts. As someone with zero design or development power over these published AAA video games... and knowing "game-feel" is difficult to prototype at a low fidelity or in storyboards... Kano Analysis felt like the correct tool.
With unlimited time, I'd want to use a MaxDiff analysis with a qualitatively similar sample and compare results. This would help counter any biases people may have about how "realistic" getting updates to the games are.
I love this diagram of Continuous Kano Analysis, where features are plotted based upon their average Function and Dysfunction scores (ranging from -2 to +4) among all participants.
I used both this analysis and the more discrete form of Kano Analysis in past professional work.
Graphic pulled from Michal Langmajer's Understand Your Customers' Needs With The Kano Model
My recruitment post in the same digital space shown above (reddit's Pokemon community)
What are the results?
I can't say yet! I'm still collecting responses one online community at a time.
When I hit at least 30 responses, I will start analysis... and if I can't hit those totals, I will update this section to include recruitment process takeaways and "what if" next steps for the research.
Currently - First week of October, 2023 - I have 15 responses.
Want to test the survey, see what changes I made to the Kano Analysis Survey methodology to reduce selection errors, or geek out about Pokemon with me? Start by viewing my Pokemon Scarlet/Violet Improvement Survey Preview, and feel free to shoot me an email to learn more at arthurpaulmyers@gmail.com!
