​
https://preview.redd.it/65byake3d74b1.png?width=1920&format=png&auto=webp&s=d3e02a536525b817c7727a6fed72a2492c8142e0

Source: Playtesting: Avoiding Evil Data
Speaker: Adriaan de Jongh
This article offers a comprehensive analysis of the source video. While it closely follows the video's content, it also includes my own interpretations and expansions on the concepts presented, providing a deeper exploration into the art of game design.
​
​
Definition of Evil Data

1. What is Evil Data

In the domain of game design, playtesting is commonly considered a fundamental aspect. This crucial stage of the development process often encapsulates the game's climax, offering the first glimpse into the success or failure of its design. Playtesting can reveal if initial pitfalls transform into unforeseen advantages or if quick fixes metamorphose into unforeseen challenges. However, for numerous independent game designers, playtesting is viewed as a necessity tinged with apprehension—an inevitability that brings a confrontational moment of truth.
The reception of critique on a piece that one is deeply invested in is not typically the major challenge here. This hurdle, while potentially daunting at first, is surmountable with time and experience. The more significant issue at hand is the prevalent use of traditional playtesting methods by independent game designers. These conventional methods, such as sourcing playtesters, executing playtests, gathering feedback, and deploying analytics, often result in ambiguous and confounding data—often contradictory and challenging to interpret. This type of data is characterized herein as "evil data," a term that signifies its deleterious implications on the game design process.
Evil data not only complicates the process of game development but also diminishes the appeal of playtesting. It is, therefore, the objective of this discourse to examine the avoidance of evil data based on insights gathered from the development of seven different games over several years. This analysis will challenge some of the traditional wisdom surrounding playtesting and offer alternative perspectives. It intends to facilitate a more valuable and organized playtesting process, thereby mitigating the impact of evil data.
The discussion progresses in a structured format, beginning with the explanation of what constitutes evil data, its origins, how it can be circumvented, and how to conduct more value-oriented playtests. It is imperative to distinguish at this juncture that QA and playtesting, though seemingly related, are fundamentally different aspects of game development. While QA emphasizes the functionality of the game, playtesting concerns itself with the quality of the experience.
The concept of evil data is elucidated through an anecdote from a playtest of Hidden Folks. A confusion over the interaction mechanism of an in-game object highlighted the consequences of evil data. Misinterpretation of playtester feedback led to the implementation of a solution that detracted from the game's appeal, instead of enhancing it. This case underscores how misleading data can lead to misguided decision-making, thus making the game worse.
Evil data is a by-product of the playtesting phase that can be misleading, unclear, and distracting. The ramifications of evil data can range from erroneous decision-making, degradation of game quality, and extended development periods. Understanding and circumventing evil data are therefore integral to creating a successful game and optimizing the playtesting process.
​

2. Eight Factors Result in Evil Data

During playtesting of Fingal, John and Brenda Romero criticized the sexually suggestive gestures in the game at a public event
2.1 Physical Location
Evil data in playtesting can stem from multiple factors, with physical location being a significant one. The setting of the playtest profoundly influences the data obtained. For instance, during a playtesting session for the game Fingal, renowned designers John and Brenda Romero provided critical feedback at a public event, which might have differed significantly in a private, relaxed setting at home.
Similarly, games like Hidden Folks and Bounden recorded differing player behavior based on the context of play. Event-based playtests often resulted in quicker and less thorough play, which influenced the game design negatively. In the case of Bounden, this resulted in repetitive gameplay as the design was overly influenced by the first few minutes of play.
To mitigate the introduction of evil data due to the physical location, it's crucial to align the playtesting environment with the game's intended setting, be it a home, a party for local multiplayer games, or other suitable locations. Hence, physical location plays a critical role in achieving accurate, useful playtesting feedback.
​
​
Different types of players have different understanding of the dialog boxes in the game
2.2 Tester Diversity
The diverse nature of potential players necessitates an expansive testing demographic in game design. During the early playtesting of "Hidden Folks", distinct reactions based on gaming background underscored the importance of this diversity. Self-identified gamers and non-gamers perceived additional elements in the game, such as dialog boxes, differently. This disparity in interpretation extended to other elements and demographics, such as age and gaming habits.
To mitigate this 'evil data', playtests should not be limited to a specific group. Everyone, irrespective of their gaming background or demographics, can offer valuable insights. Expanding the testing group to include individuals from different backgrounds, gaming experience, age groups, and cultural backgrounds enhances the understanding of how diverse players interact with and perceive the game. This diversified feedback provides a more comprehensive and accurate representation of potential player responses, leading to improved game design and development.
​
​
Don't explain anything
2.3 Your Introduction
The third aspect pertains to your game introduction. A common error among independent game developers is a full game briefing, encompassing story, controls, and objectives before allowing testers to engage with the game. This approach is detrimental as it potentially overlooks vital stumbling blocks that players may encounter without such instructions.
In order to circumvent this error, it's beneficial to eschew exhaustive introductions. Allow testers to independently navigate the game, thus mimicking the actual player experience. Such a strategy provides valuable insights on user-interface, control, and game introductions.
However, disclosing that the game is still under development could alleviate tester stress and encourage honest feedback. The aim is to observe where testers encounter difficulties without any pre-conceived instructions, thereby ensuring authentic data collection.
​
2.4 Surface Problems
The fourth point is 'surface problems'. These are challenges within the game design that are noticeable and yet frequently overlooked due to their persistent nature. An example of a surface problem might be the variability in difficulty across different player types or the excessive focus on one detail, such as a character's jump, neglecting broader concerns like the number of controllers required for multiplayer engagement.
These problems, whilst seemingly inconsequential, can lead to negative player experiences. Over time, repeated exposure to these issues can cause a sort of blindness among developers, thus neglecting crucial facets of the gaming experience.
To circumvent the rise of evil data from these issues, developers should aim for holistic player experience evaluation. Avoid testing with a narrow focus, subdivided playtests, or a specific hypothesis. Implement frequent playtesting to generate comprehensive data, enabling the identification and mitigation of overlooked surface problems.
​
​
Overlap of game patterns can be quickly fixed during testing
2.5 Alterations Between Playtests
The sixth point involves a somewhat contentious practice of altering game builds between playtests. Issues can occur during playtests that affect players' experiences, for instance, the game's layering problems or an unintended default setting screen. These small glitches, although quick to fix, can impact the player's engagement and the extracted data, causing players to potentially abandon the game earlier.
Contrary to conventional advice among game developers, this issue can be mitigated by making changes to the game build during playtesting sessions. Shifting targets or disabling certain features between tests can greatly enhance the value of these sessions. This practice of adapting and amending can potentially lead to a more efficient resolution of these impediments.
​
​
Oversized UI blocking the screen
2.6 Feedback
The seventh consideration pertains to the interpretation of testers' feedback. Testers often express their sentiments regarding aspects they appreciate, those they do not, and elements they fail to comprehend. However, the game designer's task involves sieving through the assorted remarks to isolate the fundamental issue. Testers may articulate perceived problems that are misrepresentations of the core problem.
For instance, an issue with the UI in the initial version of 'Hidden Folks' illustrated this conundrum. Many testers suggested the targets should not constantly remain on the screen, leading to a modification that allowed the UI panel to be hidden. While no further complaints were raised, hardly any testers used this feature. Eventually, it became evident that the issue was not the constant presence of targets but the UI panel's substantial size obscuring the screen.
This example accentuates the importance of discerning the underlying problem from the provided feedback. This feedback filtering process is a crucial skill for effective playtesting. Thus, designers should delve into the 'why' behind testers' comments to better understand the concerns raised.
​
​
Traditional ratings can be insufficiently instructive
2.7 Surveys
The seventh facet pertains to surveys, quintessential sources of secondhand data. Traditional ratings can be insufficiently instructive, not providing the clarity needed to address specific game issues. A one-star review, for instance, provides no substantive guidance on how to improve level-3 gameplay, leaving game designers to hypothesize about potential solutions.
Furthermore, open-ended survey questions often result in problem-solving suggestions that may not reveal the genuine root of the problem. An example includes players who struggled with level-3 not because of difficulty in finding 'John,' but due to the unrecognized ability to drag a car that would reveal 'John.' No suggestion in the feedback directly hinted at this core issue.
Moreover, player perception of the game often evolves during gameplay, resulting in correspondence that reflects the emotions and frustrations experienced towards the end of play sessions. Thus, to mitigate the risk of unhelpful secondhand data from surveys and questionnaires, other more labor-intensive yet reliable methods of gathering player feedback should be employed.
​
​
Quantitative analysis cannot explain the reasons behind the data
2.8 Online Playtesting
The eighth dimension of our discussion pertains to online playtesting, a process often rife with evil data, which implies misleading or unhelpful information that may lead to misguided game design decisions. Developers often solicit feedback from colleagues, friends, or beta testers. However, the interpretive ambiguity of the resultant feedback can distort the understanding of game design challenges.
There are various strategies for eliciting useful data from online testers, yet each harbors its own pitfalls. Direct questioning is the most straightforward approach; however, it tends to yield the most unhelpful, secondhand data. The feedback derived from this method is often retrospective and solution-oriented, lacking specificity and failing to illuminate the underlying problems. Hence, we advocate a shift away from this method due to its generally lackluster quality.
Game analytics is another option, although it also presents hurdles. For instance, to leverage analytics effectively, developers must first hypothesize what they're attempting to understand. It can reveal quantitative measures such as the time taken to find a target, but it fails to elucidate the reason behind the measure. Therefore, designers are still left guessing - a practice synonymous with evil data.
Another prevalent analytic technique is funnel analysis, tracking player progression from level to level. While it can showcase where players drop off, it, again, does not explain why, leading us back into the realm of guesswork.
A more spatial analytic approach is heat maps, offering a more granular view of player interactions. However, they still fall short of observing actual gameplay. Gameplay recordings, on the other hand, offer a closer approximation to a first-hand experience. While such recordings still contain some evil data, they allow developers to see what the players do and hear their concurrent comments.
Despite the potential pitfalls, gameplay recordings can be a valuable tool for extracting information about level flow, timing, and other specific gameplay elements. However, the requirement for technical setup may deter some potential testers, particularly those less tech-savvy. It also necessitates substantial investment in time to review the footage thoroughly.
In conclusion, each method of gathering online playtesting data harbors potential for generating evil data. Recognizing and navigating these potential pitfalls is a critical skill for developers seeking to understand and improve their game design effectively.
​

3. Organizing Playtests with Minimal Effort

3.1 Playtesting Overhead and Strategies to Minimize It
Despite its importance, playtesting can be a time-consuming and resource-intensive process. It involves recruiting testers, setting up testing sessions, collecting and analyzing data, and implementing changes based on feedback. This overhead can be a significant burden, particularly for small development teams or those with tight schedules or budgets.
However, there are strategies to minimize the overhead associated with playtesting. One approach is to streamline the recruitment process. This could involve maintaining a pool of potential testers who can be called upon as needed, rather than recruiting new testers for each session. Another strategy is to automate data collection and analysis where possible. For example, developers could use software tools to track player behavior during testing sessions and generate reports.
Furthermore, developers can prioritize playtesting efforts based on the stage of development and the nature of the game. Early in development, it may be more important to test core mechanics and gameplay loops, while later stages might focus more on content, balance, and polish. By focusing playtesting efforts where they are most needed, developers can make the most of their resources.
​
​
Use of Email Templates
3.2 The Use of Email Templates and Reminders
Communication is a crucial aspect of playtesting. Developers need to communicate with testers to arrange testing sessions, provide instructions, and gather feedback. This communication can be time-consuming, particularly when dealing with large numbers of testers.
One way to streamline this process is to use email templates. These can be used to send standardized communications to testers, saving time and ensuring that all necessary information is included. For example, an invitation to a testing session might include details of the time and location, instructions for accessing the game, and information on what the developers are looking to test.
In addition to email templates, reminders can be used to ensure that testers are aware of upcoming testing sessions and any actions they need to take. These reminders can be automated, reducing the administrative burden on developers and helping to ensure that testing sessions run smoothly.
​
3.3 Alternative Playtesting Structures
While traditional playtesting sessions, where testers play the game under the supervision of the developers, are valuable, there are alternative structures that can provide different insights and reduce overhead.
3.3.1 Remote Playtesting
Testers play the game in their own time and provide feedback via email or an online form. This approach can be less resource-intensive than traditional sessions, as it does not require a physical location or the presence of developers. It also allows for more flexible scheduling, which can be beneficial when dealing with testers in different time zones or with varying availability.
3.3.2 Group Playtesting
Multipletesters play the game simultaneously and discuss their experiences. This can provide insights into the social dynamics of the game, reveal how players learn from each other, and generate a wider range of feedback. It can also be a more efficient use of resources, as feedback from multiple testers can be gathered in a single session.
3.3.3 Longitudinal Playtesting
The same testers play the game over an extended period. This can reveal how the player's experience evolves over time, how players engage with long-term goals or progression systems, and how retention and replayability factors play out. While this approach requires a longer-term commitment from testers, it can provide deep insights that are not possible to obtain from single-session playtests.
​
​
Highlights of the talk

4. Conclusion

The insights gleaned from effective playtesting can be a game-changer in the development process. They provide a unique perspective on the player's experience, revealing how players interact with the game, what they enjoy, and what frustrates them. This understanding can inform design decisions, ensuring that the game delivers a satisfying and engaging experience. Therefore, it is highly encouraged for game developers to apply these insights in their work. By avoiding evil data, organizing playtests with minimal effort, and leveraging the strategies discussed, developers can enhance the effectiveness of their playtesting process. This not only reduces the effort required but also leads to a better player experience, ultimately contributing to the success of the game.
​
I will continue to delve into the fascinating world of game design in future articles. If you find this topic intriguing, please follow me for more insights. I also welcome your thoughts and suggestions on game design. Let's explore and learn together :)
My discord server: discord.gg/cXTKubD7Zn

​
Most Heart-Warming Romantic Comedy Movie Recommendation by the Recz App
Rom-coms, or Romantic comedies, have always garnered the hearts of the viewers. The meet-cute, the heart-warming proposals, the heartstring-tugging monologues. Rom-com movies know how to reach the hearts of the viewers. In this blog, we will be taking a look at some of the best romantic comedies of the decade. Read on to learn more about some of the best rom-com that will melt your heart away.
Best Rom-com Movies
1. Mr & Mrs Smith
Mr. & Mrs. Smith is an action romance comedy flick. It can be considered a great romantic movie recommendation. It is a movie that features Brad Pitt and Angelina Jolie in the lead role. Mr. & Mrs. Smith is a story about two individuals who are married to each other. Their normal life is disrupted when they realize that they work for competing assassination agencies. These agencies then ask the couple to assassinate each other. This is when all hell breaks loose.
This action romance comedy recommended by our review app was a hit at the box office. Reportedly, it also helped Pitt and Jolie with their personal relationship. Mr. & Mrs. Smith was released on June 2005. This rom-com received mixed reviews from movie critics. According to data, this movie grossed $487 million worldwide. Mr. & Mrs. Smith also featured Adam Brody, Kerry Washington, Stephanie March, and others in supporting roles.
2. The Lost City
The Lost City is an adventure romantic comedy. It features Sandra Bullock and Channing Tatum in the lead role. It features Daniel Radcliffe as the antagonist. The Lost City also featured Brad Pitt in a small cameo. The film follows the story of a middle-aged woman who is a novelist and her cover model. They are kidnapped by Daniel Radcliff’s character, who wants them to find a lost treasure hidden in a burial chamber.
The Lost City garnered mixed reviews from the critics and the audience alike. The Lost City is an adventure comedy directed by Adam Nee and Aaron Nee. The story of the film is provided by Seth Gordon. It is also co-produced by Gordon and Sandra Bullock with Liza Chasin. In the film, Bullock plays the role of Loretta Sage, whereas Tatum plays the role of Alan Caprison. This film also features Oscar Nunez, Patti Harrison, and Bowen Yang in supporting roles.
3. Crazy, Stupid, Love
Crazy, Stupid, Love is yet another good movie recommendation provided by the Recz review app. It features an ensemble of popular actors. It features the life of a middle-aged man whose life takes a dramatic turn when his wife asks him for a divorce. With help from a stranger, he seeks to rediscover his manhood. Crazy, Stupid, Love features Steve Carell, Ryan Gosling, Julianne Moore, and Emma Stone in the lead roles. In the film, Gosling’s character helps Carell’s character to rediscover his manhood. Crazy, Stupid, Love is one of the best heartwarming romantic comedies. It received a rating of 7.4 out of 10 on IMDb. You can watch Crazy, Stupid Love on Amazon Prime Video.
4. Music and Lyrics
Music and Lyrics is one of the best romcoms in this list. It features the legendary actor Huge Grant and Drew Barrymore in the lead role. The film also features Brad Garret, Kristen Johnston, Haley Bennet, and others in supporting roles. Music and Lyrics focuses on a relationship between a music pop star and an aspiring writer. The film depicts the struggle of these two as they write songs for a pop idol.
Music and Lyrics received 6.5 out of 10 on IMDb. This romantic comedy flick is written and directed by Marc Lawrence. Music of this movie is provided by Adam Schlesinger. This flick was distributed by Warner Bros. Picture. It received $145 million at Box Office.
5. Silver Linings Playbook
Silver Linings Playbook is a romantic comedy drama. It features Bradley Cooper and Jennifer Lawrance in the lead role. Ths romcom also features actors like Robert De Niro, Anupam Kher, Chris Tucker and Jacki Weaver in supporting roles. Silver Linings Playbook features the love story of a man with bipolar disorder who is freshly released from a psychiatric hospital. He moves back with his parents and meets a young widow Tiffany Maxwell, played by Lawrence.
Silver Linings Playbook was premiered at the Toronto International Film Festival back in 2012. It was then released in the U.S. in November 2012. It received a rating of 7.7 out of 10 on IMDb. This film is a good movie recommendation provided by the Recz review app. You can watch Silver Linings Playbook on Prime Video and Youtube Premium.