A Song of Ice and Dust: Usability Tests

Research for the "Dustfaire" DLC was limited in time and resources, so I prioritized usability. I organized in-depth, moderated,internal usability tests followed and interviews, which I streamed live to our agile team. The feedback revealed significant usability issues, especially with Mission 2's excessive difficulty. This approach revolutionized our development workflow by enhancing collaborative efforts and accelerating decision-making. It also led to the implementation of regular internal playtests, which significantly improved our game design processes and the overall quality of player engagement.

The "Dustfaire" DLC pack was set to be the final Saints Row release ever, and the ambition was to deliver an unforgettable experience. Market research from Deep Silver (our overseas distributor and parent organization) revealed a strong player preference for the LARP segments of the main game, prompting the game designers to create an expansive new district with a rich narrative, distinctive dialogue style, and multiple side quests for a comprehensive seven-mission package. In sum... The ambitions, resources, and timeline for this DLC pack defiedall the laws for project management. The studio was determined to make its mark.

My primary responsibility was to strategize user research opportunities throughout development. With the release's importance and only one week allocated for this research section, efficiency and depth of feedback were paramount, especially since only three of the seven missions were in a testable state when the time arrived.

1. I prioritized in-depth, moderated usability testing with follow-up interviews. Constraints reduced us to 3 participants per mission. I designed each session to last up to an hour, maximizing the opportunity to probe and control for technical malfunctions.
2. After consulting with the design team and producers, we focused on gathering in-depth feedback for each testable mission rather than collecting shallower insights across all three. This approach meant scheduling dedicated testing days for each mission. We organized the testing sessions to occur consecutively, with all participants for Mission 1 on Monday, Mission 2 on Tuesday, and so on, maximizing our focus and the quality of feedback for each mission.
3. I live-streamed the testing sessions with the rest of the agile mission teams, which included designers, writers, narrative designers, programmers, engineers, project managers, and product managers, enabling real-time observation and decision-making in a dedicated Teams channel. This setup facilitated immediate feedback and collaborative problem-solving.
4. Before and between studies, I coached the development team on appropriate observer/moderator behaviors and questioning techniques to reduce bias and clarify priorities. Their engagement exceeded my expectations and significantly contributed to the process, and by day 2, I encouraged them to engage with participants firsthand.
5. After each study, I facilitated a synthesis/prioritization session for the stakeholders to compare notes, clarify takeaways, and organize actionable tasks in JIRA.

‍

The testing sessions and interviews uncovered valuable insights into usability and gameplay experience. Specifically, all participants struggled excessively with Mission 2, whereas Mission 3 felt too simple (despite being enjoyable).

1. The success of the live-streamed sessions and the development team's direct involvement in real-time feedback revolutionized our workflow, speeding up decision-making and fostering a more collaborative environment. Notably, the live-streaming approach enabled immediate and effective problem-solving. This method significantly improved the development team's morale and engagement with the project, even in the face of constructive criticism.
2. Following this, I initiated tutorials to train our Principal Game Designers on conducting internal playtests, turning it into a weekly practice that improved the quality of our testing materials and refined our approach to creating engaging player experiences.