An (Ice) Dev's response to the 2022 Gain Jam

Having participated in the 2021 and 2022 Gain Jams, I have a number of issues I hope to address with how the Jams have been run. I hope to outline problems I have noticed with the jam this year and propose some solutions to make the Jam more inviting as a friendly competition for developers to flex their skills and focus on making the best games they can under the criteria given.

In my response, I will reference Grot’s three major goals for the Jam:

  • Challenge devs to improve their skills;
  • Encourage development of rapid prototyping skills & try new concepts;
  • Try to encourage new devs to at least try their hands at game dev.

ISSUES

WILD CARDS

Wild card, while an interesting addition, failed to either meet the three main goal or address the problems of last year. While the idea of a simple theme and optional, more complex wildcards are good in theory, any serious competitor finds themselves required to attempt them all due to how the games are scored. If not already inconvenient, judges would on occasion miss wildcard additions entirely, something that could be fixed by simply attaching a form to the corresponding post.

While there is certainly fair challenge in implementing all three wild cards, all three cards leaning in a mechanical direction pulled too hard towards certain ideas. With the cards being mechanical in nature meaning that using it in a narrative sense would be potentially invalid. The basic pass/fail was presented more clearly, but it was unclear what would result in full marks or not.

For the goal of getting participants to try new concepts, it certainly paints clear goals of things to try, but announcing the wildcards weeks in advance seems to be actively opposed to the goal of rapid prototyping. To me, the excitement of a game jam is making an idea that did not exist before the timer started and having as complete and realized an product as possible by the time it’s over. I think it’s telling that Minions was such a prevalent idea in many games this year, with every subsequent wildcard falling off in usage. Rapid prototyping is directly undercut by having weeks of advance time to develop concepts before the competition.

As for encouraging new developers, it isn’t hard to imagine that having these additional rulings, some of which are only clarified by scouring through forum posts or participating in the discord, would present a noticeable barrier.

ADMIN RESPONSE

When asking questions on the forum and discord regarding the scoring of wildcards, as well as the distinction of fetish elements and their anomalous splitting apart and different scores in the second round, there were a series of problematic responses.

Often, questions regarding the second round were answered by judges who had only been in charge of the first round. Though certain scoring areas were inevitably measured differently by every judge, a lack of cohesion contributed towards inequalities that ultimately cost points.

The role of Alex as community manager is perplexing if, when answering questions, everything is overridden by Grot regardless. This indicates a lack of communication within the admin team.

SCORING

Since last year, the problematic scoring system was functionally unchanged. Every score was dropped by a decimal point and wild cards were introduced, meaning that the actual weight of each category was unaltered. While the wildcards in theory would offset lower scores, anyone who wanted to maximize their scores is required to go for them.

Judge’s Preference has been a mark of contention. The wiki currently defines this category as “Points that can be awarded based on a judge’s discretion.” which means any judge is free to use this however they want, giving them more freedom than the more defined definitions of other categories. Having fresh guest judges helps prevent the scoring from being a checklist and it’s unreasonable to think there’s a truly objective way to evaluate whether one game is better than the other.

With a guest judge’s atypical scoring approach for the majority of the games, this particular issue has become a major problem for participants. There was a non-zero amount number of game rankings that could have been altered by this category being filled in rather than ignored. While the Preference category is allowed to mean whatever it wants to the individual judge, that does not mean it doesn’t matter to the participant. A Preference score that is significantly above or below the average of their other scores, should signify that there are elements of the game that cannot be cleanly defined on the score sheet. When a judge hands out scores of zero, it says that they find nothing of value in the game. Numerous people were sullen by the thought that a judge disliked their game enough to give a zero. Giving a game a ten simply to boost it’s score makes the other points feel meaningless in comparison when they’re also worth the same amount. A game that already scored well should have no need for this. While many of the games in the top five did receive zeros in Preference, it’s about the feedback this gives, not the placement that results.

PROPOSED SOLUTIONS

WILD CARDS

I would recommend looking at how other jam integrate them into their competitions. Make all the cards mysteries up until the day the jam starts and require a form to be filled out to declare what wild cards are used and how they are used. Removing the ambiguity of inconsistently placed in-game cards and clarifying the intentions will help judges understand how the wild card is being used. If the community card poll remains, the results should be hidden.

Furthermore, wildcards should be expanded to more areas than just mechanics. Settings, themes and other more creative directives would help prevent the overlap many cards have and the strain it places on a participant.

One final suggestion would be the removal of the admin card. If the administrative team wants to have a more challenging theme for the jam, then they should make it so rather than have an awkward double dip into the dictation. Having to include an expansion fetish element to the game is already a fourth theme in it’s own right. Main theme, fetish, and two wildcards feels more than enough to provide challenge to participants. If not, reducing the maximum bonus score to 10 instead of fifteen means a card can be safely ignored, or effort can be divided among them with less stress about how perfectly they are done compared to the main game.

ADMIN RESPONSE

The continued development of the Weight Gaming wiki will help alleviate these he said/she said scenarios that occur between admins and judges when scoring is questioned. In addition, I suggest appointing a member of the administrative team to be a jam director, someone who has a clear say on the rules and organization that both the other admins and participants can build their understanding off of. What's most important is that this person is NOT a judge, and serves purely to facilitate the jam.

This would also remove the need for two round judging. While many object to the time it takes, the bigger complaint is that it is not fair for a pair of people to have a far more important say over the efforts of others. The opinions of a judge who played every single game and evaluated them are worth magnitudes more than a judge who plays a mere fraction of them.

SCORING

The mechanics section refers to the base aspects of the game function, not how engaging the actual gameplay is. A game that focuses on being a gameplay experience first and foremost can be at a deficit. In addition to mechanics being a measure of how well a game functions, game play should be added as a category for how well the game feels. While this may sound confusing, allow me to provide a pair of examples to help clarify;

A world simulation game that has an elaborate set of environmental interactions that the player can trigger might be high on mechanics, but if it’s dull to watch these interactions play out, then it would be low on gameplay.

Likewise, a single button rhythm game based around following a call and response pattern would be rather low on the mechanics, but if the rhythms are interesting, if well timed pressed result in interesting scenarios, if failure to keep time results in negative feed back, then that would be high on gameplay.

If gameplay is introduced as a category, my two recommendations would be reducing mechanics to 5 points while gameplay is 10, and include a “Best in Gameplay” award.

Renaming the writing category to narrative would allow for less strict evaluations of what is literally written word vs what is story. Narrative is the emotions conveyed by the medium, which in videos games can be done with literal story told through words, voices, and environments, but also gameplay. This can allow more arcade style games to shine in scoring. A puzzle game that gets increasingly chaotic with it’s mechanics and visuals can still convey a narrative more engaging than, say, an rpg with dull story beats and uninteresting characters.

Finally, Judge’s Preference needs to be altered. This category is valuable for allowing a judge to give feedback on elements that aren’t merely tied to scores or represent a synergy between different factors. A clearer definition of “Judge’s personal response to the game” would help, and this score should be reduced by 50%. There is no reason a game should be docked a full category’s worth of points simply due to personal preference with no accountability.

PARTING THOUGHTS

The Gain Jam is a meaningful event for me; it’s where I got my start in game development last year. Last year, I voiced confusion about the winner of Best Writing not being publicly announced. My concerns were brushed aside and later I was given a half-hearted apology. This year, I asked why the second round of judging had multiple fetish categories each worth double the normal amount. My concerns were brushed aside and later I was given a half-hearted apology.

My greatest desire in commenting on these and various issues is to have problems addressed before they result in errors. Seeing the confusing changes made to the jam this year makes me feel like the wrong lessons have been learned from the 2021 Jam, changes made that seem deaf to the desires of participants entering the jams. The developer response survey is a fantastic step in the right direction but with it’s limited format and leading answers, I fear will not allow the full extent of participant frustration to be expressed and addressed.

Forty-four entries times by two weeks equals nearly two years worth of development time across dozens of people and their teams and that effort need to be respected. I know I and many others took time off of work and other commitments to try and make the best games we could, but it’s with a heavy heart that I have to say that without meaningful change it is simply not worth my time to participate next year.

33 Likes

Damn look how cool and sexy Ice is for writing this gain jam essay

1 Like

This all seems like valid, well-thought-out feedback. I definately agree about the wildcards, knowing what they are only when the competition started may have created much different games than what we got. I’m sure that some people may have already had an idea in their minds of what game they wanted to make long before the theme was announced because of the wildcards alone.

5 Likes

As a whole I feel like I agree with your post. If I were to add one thing, perhaps cap wild card participation to one card. I.e. propose multiple wild cards (according to the criteria above) but only allow for one to be used. By allowing all three to be used and granting points for all three it seems to greatly filter down what sort of game will be made and hamper creativity. However, if you were able to choose one wild card to approach wholeheartedly among several options, I think that it would encourage people to think outside of the box and have more options in choosing which options they actually wanted to pursue.

As far as the judges preference bit, I agree it shouldn’t be a part of the final score. I like the idea in concept, but in practice it turns into something like 30 points which are assigned effectively at random unless you play to the judges’ tastes, which I refuse to do for a passion project.

5 Likes

I was gonna do one of these as well, but you did it better then I could have done it.

I think having wild cards was a nice idea but making them provide points was a bit of a misstep in my opinion. It might be worth looking to larger game jams to see how they run things.

I think the current scoring system is vulnerable to a snowballing problem. Entries that cash in on a wildcard or happen to align with judges’ tastes are already going to receive praise. So, rewarding that with points means that attention is effectively funnelled into a small group which can feel a little disheartening to anyone outside of it.

The Epic Games jam this summer used a system kinda similar to the wildcards called “Diversifiers” they didn’t have a scoring value, but were instead auxiliary awards that encouraged devs to branch out. Stuff like “Best audio”, “Best realistic graphics”, “Best stylised graphics” to name a few. They were revealed alongside the jam’s theme and as far as I understand there was still an overall winner. But by having smaller awards that didn’t directly feed into the main race, the jam rewarded devs for going off the beaten path and gave a wider variety of devs their moment in the limelight. Maybe have ones for best narrative, best use of mechanics, best art, sound etc.

The same could be applied to judge preference I think. Having an honourable mention for each judge, or something to that effect that doesn’t have a numerical value would prevent entries from being snubbed by mismatched tastes while still allowing judges to express their preferences.

I think the linearly competitive structure kinda stands at odds with the objective of the jam, which is a shame because I think game jams are a brilliant thing.

7 Likes

I agree with most of the points this post makes, and I hope the admins take note of it for next year.

Maybe for the wildcards, instead of awarding points the reward could be a small cash prize for whoever implemented the card most effectively/creatively.

Two things.
1 - Where would that cash prize come from?
2 - How would that fix the issue people have with them?

I personally would make all Cards Bonus points only, and turn Story/Judge Preference into Bonus points as well.

If you don’t know how Bonus Points work for the jam they are points that can fill in for points you didn’t get somewhere else. (they probably should be used for tie breakers [but not Judge Preference])

The reason for story to be turned into Bonus points is this is a Game Jam and not a Story Jam, Gameplay doesn’t need a story and as books show Stories don’t need gameplay.

If someone wanted to make a Visual Novel for the Jam Story and Gameplay could swap primary and bonus point places for their scoring.

1 Like