Showing posts with label playtesting. Show all posts
Showing posts with label playtesting. Show all posts

Thursday, December 5, 2013

Practice 2013: Understanding Players Panel

·         Davin Pavlas (Riot Games), Morgan Kennedy (Assassin’s Creed), Naomi Clark (GameLab, FreshPlanet, RebelMonkey)
(by Davin Pavlas)
·         Research at Riot Games, Senior User Researcher
·         Research as player focus, being player focused requires understanding players (we’re players too, but many kinds of players play our games). Research equips us to understand wants, needs, behaviors.
·         League of Legends – MOBA game with lots of players, competitive 5v5
·         Research definition = propose a model of the world and test that data without bias.
·         Actual definition = test design assumptions to ensure player experiences are positive and intentional.
·         Data-informed approach for design = informed but not driven (use knowledge and data alongside intuition, work with goals and assumptions, intended experiences), players are collaborators and not just data, research isn’t a gate for a box to check (integrated into the design process in an iterative fashion)
·         Interdisciplinary inquiry = telemetric data, modeling, surveys, community management
·         Case Study #1
o   Lucian the Demon Hunter with two guns (Equilibrium style). Sleek, speedy, serious ranged damage character with finesse gameplay.
o   Research flow = concept testing, 3 lab studies, release, post-launch
o   Concept test = simple read of concept art, survey & interview (fantasy, mood, personality, role, fit into the fiction), high appeal scores, good theme conveyance. People thought his two different colored guns meant something.
o   Lab test = moderated one-on-one with players, think aloud protocol, streamed and recorded for designers and team
o   Lab study #1 = felt like stiff movements, not what was promised by concept, movements felt like a turret, abilities didn’t fit player model of the concept art, visual theme compelling but his gameplay was problematic. Solution = remove stuff movements and confusing abilities.
o   Lab study #2 = aesthetics still a mixed bag, high base speed but players feel he’s still slow, ultimate was cool but frustrating to use.
o   Why does he feel slow?  Animations
o   Lab study #3 = he finally felt fast,  gameplay and aesthetics fit the concept, mix of animation and frame speed changes make Lucian feel fast, ultimate was easier to use.
o   Post-launch = fiction quality, theme quality, and default aesthetics were considered high. Visual quality was okay and gameplay quality was low.
o   Summary = difficult champion to get right, theme was very compelling but its mechanics needs to be matched with experience design
·         Case Study #2
o   Champion select screen is where many intense interactions happen. There’s very little time and little information to help make players make these tough decisions.
o   Champion select is not an effective setting for collaboration. Solution = remake champion select to be more collaborative and have intent-based queuing that does not set the meta-game.
o   Externalize all the assumptions and test them.
o   Research flow = concept testing, global testing, prototype testing, lab studies
o   Initial survey = random sample of 1000 NA/EU players, player preferences (each role has at least 15% favorite rate), feature disposition, composition reactions
o   Global testing = initial survey revealed lack of role terminology consensus (fighter, carry, tank)
o   Prototype testing = an idea with merit still needs to be executed correctly. Results = positive disposition, Captain vs. Solo experience, usability flow
o   Onboarding study = test with brand new players, Team Builder vs. Champion Select
o   Public beta testing = more testing with the player base
·         Summary = system and character testing are similar (test assumptions, validate and ensure intended experiences are being made)
(by Morgan Kennedy)
·         Understanding Players (Assassin’s Creed 4 Silhouette and Behavior Differentiation)
·         Was a manager at Gamestop before working at Ubisoft. Worked on Assassin’s Creed series, Ghost Recon, Michael Jackson Experience, Your Shape, and Just Dance 3.
·         Affordances in AC3 = a railing means that the player can free-run there.
·         Mental models = the idea starts in the players head, player learns things about your game each time they play them
·         Ship identification = players had problems telling the different ships apart, can usually tell the gunboat (smallest ship) and the man of war (biggest ship), but unable to tell the difference between the three middle-sized ships (Schooner, Frigate, Brig).
·         Perceived differences = small ships were faster and tended to flee, big ships were slower and more aggressive, the gunboat was the exception
·         Players identify ships = differences in behavior (speed, aggressive, size)
·         Ship archetypes should be optimized around familiar affordances (man of war kept its large size, made the medium long-ranged ship less aggressive so it was more differentiated with the other medium ship that was short-ranged)
(by Naomi Clark)
·         How Indies Playtest, enlisted over 20 indie developers about their playtesting process
·         halfrobot.com/indiesplaytest.pdf
·         “Playtest before you think you are ready. Is it too early for you to playtest? If the answer is yes, then playtest anyway.” – Eric Zimmerman
·         “Don’t playtest very early one.” – Dan Cook
·         Methodical time-slice testing = know your questions, know what your build is suppose to deliver, hit your targets
·         Doug Wilson doesn’t even like the word playtesting. Conventional playtesting was good for small UI tweaks, but not mechanics design. Get key suggestions from influential people.
·         Techniques = don’t talk while players are playing, ask players to think out loud, listen to what players say about problems they had and not the solutions they come up with, construct a heat map, do a more targeted demo (depending on the audience, tell them what to do instead of letting them go through the coded tutorial)
·         “Playtesting is for masochists.” You need thick skin, it’ll upset your expectations, let a day or two to let the ideas swim around in your head without thinking about them too consciously
·         Eric Zimmerman’s under-the-desk creative meditation
·         It’s very bad to show your game to other developers, they will tell you how to make the game instead, other devs can help you construct your own weak points, kid testing is great because they are honest and blunt, don’t necessarily listen to everything kids are saying
·         Maybe games aren’t meant to be played (lose/lose), or will never be played (Jason Rohrer’s A Game for Someone), or refused to be played (Train)
QUESTIONS AND ANSWERS
·         Lucian’s iterative design took about 3 weeks per cycle. Designer’s intention of the character can sometimes be quite different from player’s perception of the character.
·         Playtesting at Ubisoft is a different service, so testers are not considered game designers. They just provide a mirror of how players are playing the game. The Assassin’s Creed 4: Black Flag team is about 300 people, with 20 people being game designers. The testers write about 80 page book report every two weeks with lots of empirical data (telemetry data, player quotations and feedback).
·         “Data is fiction. Data cannot tell you anything. Only you checking your own assumptions and intuition will tell you something.” – Davin
·         The community from the internet is not representative of your full community. Everybody on the team engages with the community. You need to learn how to be qualitative with that data.
·         Adam Saltsman’s Grave has complete publicity and openness to the community.
·         Be wary of changing your entire design, which can happen if you listen to too much feedback.
·         Bad research is bad research, and it can come from anywhere.
·         The commonality between all designers is that playtesting is extremely useful for seeing how well your UI works. This is something that is not shared with films because films don’t need to test the viewer’s usage of the product.
·         There is more than just raw playability testing. Testing is useful for testing emotional resonance and connection that players have with the space and characters.
·         In Assassin’s Creed 4, players complained that they hated fog, but sailors hate fog.
·         Set is a game that Davin loves and cannot play because he is red-green color-blind. That is a game that wasn’t playtested enough.
·         It’s not about whether a course of action is correct, it’s about your goals and being able to meet them. Lucian was meant to be a very high skill ceiling char that is fast, and the playtesting and design choices should work towards that.
·         Tight-loop in game design when professional players become game designers of the game. There’s a natural progression at least in the triple A space for players to become its designers. You don’t need to be an expert to give data and insight, but you do if you want to be an effective collaborator.
·         David Sirlin’s games have a very developer-y like feeling to them.
·         People know they are compensated to test, but for League of Legends andAssassin’s Creed, the playtesters are so passionate and excited to test that the games that they give biased information.
·         What kind of affordances can you create with Team Builder to not dictate strategy? Team Builder decoupled roles and position, it focused on how players expressed themselves, not on how the character is meant to be played.
·         Riot uses a lot of designer intuition to make decisions so they come into failures a lot fewer times. Failures are usually not of testing and researching, but of interpretation of the data.
·         There are people who are very excited to use data to dictate decisions. We need stewards to guide the data and interpret them correctly.
·         Self-selection bias – compare to a random sample.
·         In Assassin’s Creed 4’s fiction, you are actually a game developer that’s making the simulation that you experience in the Caribbean. At the end of each mission, you will rate each level from 1 to 5, and it will actually be sent to Ubisoft. It’s gamified gamification.
·         Riot uses modeling to look at impact, temper risks, and find parts to sweat out and be agile about. You want to be able to knee-jerk and develop longitudinally.
·         Was Dark Souls playtested at all? It’s a beautiful disaster that makes you feel alone, afraid, and frustrated… so it clearly has been tested and achieved its goals. It has a very good tutorial too.
·         Riot has a whole team devoted to improving the community and encouraging positive behavior. Team Builder was made by that team. In general, the online player base is not very negative and the negative players that you see in games are not from trolls. Negative behaviors usually come from someone having a bad incident. This makes it easier for the developers because that’s a point where they can intervene and try to change for the better.
·         Testers are injected midway into the process and collaboratively co-design. It’s not a waterfall process.
·         Recommendations = Katherine Ibister’s book, Jakob Nielson’s Usability,Pedagogy of the Opressed

Tuesday, September 10, 2013

Casual Connect 2013: Incorporating Player Feedback in the Game Design Process

by Eric Cantini, CEO, Sunstorm Games
·         Abstract: This talk covers how to best run playtests with kids and offer other forms of collecting feedback from players.
·         Sunstorm Games was founded in 2009 and has published over 80 titles like FairFood Maker and My School Dance. They have 40 employees and focus on games for 11-13 year old girls.
·         There are two main strategies for collecting player feedback: analytical data and surveys. Get gameplay analytics via Kontagent or other similar platforms, and feedback through usability studies.
·         Sunstorm has one resource dedicated to full-time analytic implementation. QA are also required to become familiar with sophisticated reporting. Have a platform able to cleanse data, protect against fraud, and do occasional de-duplication.
·         Market performers are not universal. Just because an ad works overseas does not mean it works in the US, so test for each market.
·         Usability studies are one-on-one lab sessions with an observer watching and webcams pointed at the player’s face and the player’s hands. Girls tend to blame themselves for losing and don’t say that the game is too hard or if the game is broken. It’s important for the observer to ask the right questions. Don’t ask if the game is fun, since kids will always try to give you the pleasing answer. Also, having a female observer works more effectively, since girls are more comfortable with other females and boys are used to having a female teacher figure.
·         Do QA via crowdsourced testing such as uTesting. There are often a lot of QA feedback about bugs and crashes, and a few usability and gameplay mechanics feedback.
·         Run focus groups with product plan reviews for ideation and validation, and game design discussions.
·         Create player game design teams, who meet for a couple of sessions over the course of a few months. Start with a kickoff session, run several round table discussions, and end with a launch party.
·         Create a virtual player panel of 1000+ players who you can e-mail game design questions to and receive feedback. This is great to find favorite art styles, best name for the game, popular clothing, etc.

Sunday, September 8, 2013

Casual Connect 2013: From the Mouths of Babes: Re-learning How to Design a Game By Watching Kids Play

by Kelly Fennig, Producer, Slant Six Games
·         Summary: When making a game for kids, test with kids often. Don’t make assumptions and don’t listen to the parent’s advice. Kids like edgy characters, slightly older characters, game mechanics that involve tapping on moving objects, and watching AI play.
·         Kelly worked on Max’s Pirate Planet – A Board Game Adventure, a game similar to Mario Party for children aged 5-8. The company consulted Dr. David Barner who provided help with child psychology.
·         Craft questions carefully. With characters, they made a lot of designs and asked kids, “Who would you want to play with at recess? Who has the best toys? Who would be your best friend?” Don’t ask negative questions (ie. who doesn’t…) and never follow up questions with “Why?” because kids usually cannot articulate their reasons.
·         Create characters that are 2-3 years older than the kids you’re targeting. This gives them a relatable older sibling that they can aspire to become soon. Don’t make the characters too young because kids are turned off by “baby games.”
·         Kids like edgier characters. The most popular character designs were those with mean eyebrows, tears on clothing, and “badass” attitude. Parents will say that these characters are too mean for their children, but studies prove that parents don’t know their own kids. Edgy characters appeal to both boys and girls. It’s the Angry Birds phenomenon; the birds are angry for a reason.
·         How something is said, or not said, is more important than the words that are said. Kids may say things just to please the person asking, but they are completely honest with their body language. Don’t ask if they like something or if they think it’s fun; it’s more important to watch the way they play.
·         With mini-games, they used intelligent design, where form follows function. Instead of prototyping, they tested using other games, specifically The Crazy Test on iPad. They only verbally explained the goals and allowed the kids to continue playing until they themselves choose to stop. They watched body language to determine the games that were the most intuitive and popular.
·         The top 8 ranked mechanics among kids and adults became mini-games. Tapping and dragging multiple moving objects going towards a center target became the tentacle swiping game, tapping on numerous random moving objects on the screen became the bat whacking game, tapping on multiple predictably moving objects became the spider smacking game, dragging through a randomly generated maze became the whaling sailing game, tap and flick on a moving object became the final end game, random lottery became a pick-a-card game, swiping in a random direction very quickly in a time-limited fashion became the sword duels, and whack-a-mole became the cannon duels.
·         A big decision was whether to allow players to watch the AI perform their turns. Adults claimed that kids would be bored waiting for the AI to play and requested an option to skip the animations. In reality, kids are completely engaged with the AI. Their eyes were glued to the screen, watching what their opponents were doing. Adults were the impatient players.
·         Don’t put in negative turns (where nothing happens) or the players would get frustrated. Players were given options for every turn so they always had something to do. There was no negative turns unless by choice. They can draw a card, teleport, look for treasure, or duel another player.
·         There are distinct play patterns not based on culture or gender; it was solely based on age. The 5 year olds were explorers who were just happy to be playing and moved around aimlessly. The 6 year olds were the treasure seekers and would much rather have someone else play the mini-games for them. The 7 year olds were the well-rounded players, who would help out younger players and would sometimes duel, but much rather keep their treasures and cards. The 8 year olds were duelers, but they weren’t very confrontational and would not duel someone who doesn’t want to duel back. The 9 year olds were the tactical players who took advantage of every situation that were there.
·         Lessons – shape early and shape often, over 60% of 8 year olds don’t read even if they know how to read, minimize text to short actionable sentences, don’t make games for younger audiences because kids don’t want to play “baby games,” ignore the reviewers because they’ll throw out a lot of gameplay assumptions that are simple wrong when tested