Hey folks! I’ll try not to repeat myself too much from the initial Discord post that warranted this thread. (Link to the post here) but to briefly explain the proposal for those without Discord:
Rather than forcing judges to judge an increasingly-untenable numbers of gain jam entries, to have the community at large rank the entries in a series of polls broken down by cateogry (Graphics, Sound, Writing, Gameplay, Innovation, Gain Jam Theme, Kink). Points are awarded based on a game’s ranking in each category and the points tallied together over all categories to determine an overall “Community Score”. Entries that either meet a score threshold or simply the top scoring games move on to the Finals where they can be assessed in the usual vein by the judging panel.
The polling for each category would be kept simple - a random pairing of game entries and the user given the simple question “Which do you prefer for [poll category]: game A or game B?” This process is repeated with a different random pairing each time, for as long as the user continues picking winners from each match-up. Wins and losses for each game are tracked and the games ranked accordingly within the category (this video from Tom Scott does a great job of demonstrating the format, and was an inspiration for this idea.)
Ultimately the idea is to 1) lighten the burden on judges and 2) foster more community engagement. The former point I feel will need to be addressed at some point if the community, and subsequently the Gain Jam itself, continues to grow. The 40+ entries this year were largely unprecedented and I cannot see this number dropping next Gain Jam - nor, arguably, should it.
@Juxtaterrestrial raised some good points in the Discord thread that I will attempt to address here shortly but I do invite all to comment on and discuss this proposal!
(edit: also wasn’t sure if this belonged in the “Gain Jam” section or under “General Discussion”, please move the thread as necessary, mods!)
having a community score often devolve into a popularity contest rather than what game is most popular/best at larger scales. since this is supposed to help with scaling I don’t think it’s the best way to do so
the vote assumes that everyone that votes has played every game, but that probably won’t happen.
so I believe that it will give people that know how to “sell” their game an advantage and people making more niche things have an disadvantage.
when people have to vote they are incentivized not to pick based on the category but rather what game is their favorite. even if they know that a is better than b at x they would vote b so it is more likely to win
it might also make devs feel like they need to make their game a certain way if they want to have a chance at winning or getting detailed feedback from the judges
You raise points that I absolutely think need to be addressed and I hate to see folks overwhelmed while proctoring this contest.
Rather than forcing judges to judge an increasingly-untenable numbers of gain jam entries
As soon as I saw the games come in, I had the same thought. I think there’s a potentially huge benefit to community judges, but the advantages are more difficult to realize online. But, I know IF Comp does it. IFComp - Best practices for judges
having a community score often devolve into a popularity contest rather than what game is most popular/best at larger scales. since this is supposed to help with scaling I don’t think it’s the best way to do so
And I just want to recognize that by giving people, including creators, voting power in a contest with a prize pool, it does create a possibility and incentive for exploitation. This could mean fake accounts, could mean unfaithfully scoring games to affect the overall score. Whatever our solution is - and as much as I hope those lines don’t get crossed - I think it’s important to create impediments to cheating and misconduct.
Yep, a fair question and one that was touched upon in the Discord thread as well so let’s touch upon biases.
The voting should be kept as wide open as possible to encourage as large a sample size as possible. As to gaming or abusing the poll, the key here is in the random generation of the pairings: to put some numbers into context, with 40 games in the contest that is a total of 780 possible pairings of games. If I, a scurrilous individual, was dedicated to pushing a certain game up or down irrespective of category then there is a P(0.05) roll of the dice that I’d even see a specific game in a given match-up (39 pairs / 780 possible pairs = 0.05).
As the number of games in the Gain Jam increases, the % chance of spotting a specific game decreases: 50 games goes down to P(0.04) (49/1225); a 100 games (at which point could a judge be even able to accurately judge 100 games in a timely fashion) goes down to a 2% chance of seeing your chosen game come up (99/4950). And that’s just for one category. There are seven polls!
Put it this way, the efforts of one super user with an axe to grind would be to spread votes widely over a majority of entries rather than target an individual game (random chance permitting); if a hypothetical group was orchestrated enough to purposefully raise or lower a game’s rankings… then maaaybe that should be pretty telling as to its quality, no? I’d be pretty chuffed as a developer if dozens of people decided to swing a vote in my favour, as it meant I had made a killer game >w<
The one way I could foresee this being genuinely abused is with botting/scripting exploitation, which I had mentioned in the thread. Means to prevent bot/script abuse such as imposing limits on voting frequency could work. Along with other bot-detection measures to expunge suspect votes.
As for punishing specifics interests (a point raised by @Juxtaterrestrial) keeping the question as simple as possible (which do you prefer: A/B?) mitigates bias over a large enough sample but still allows for individuals to make a personal choice. If an inflation-based game happened to have a higher win-% compared to a stuffing-based game in the Kink category, after all statistical weighting, then that would seem pretty demonstrable as to which the community by and large preferred. I see that as working as intended.
I can see issues with allowing the community to decide. For one we don’t all share the same fetishes. Like some folk here are furry’s and some aren’t, some like male wg and others are female only. So in having a community poll people will vote for whatever game suits there fetish, not necessarily on the quality of the game.
But therein lies the beauty of a community-wide poll: it would accommodate for all interests since anyone and everyone in the community can contribute. Since all tastes are accounted for by the nebulous entity known as the “community super judge” I would reckon that games with niche interests would sit near the middle of the pack in the hotly-fought “Kink” category since they could still be supported by those who champion said interests. For every voter who votes down on principle on a given kink, there will be the gamut of fair reviewers and ardent supporters, so I see it balancing out. If it’s a kink done well in a game, that will be recognised by fair voters; everyone can universally agree on a game being flat out unsexy- niche interests or not!
And let’s not forget that this is but one poll among many - a game with a niche interest can still acquire points in other fields if the game is good enough!
If an inflation-based game happened to have a higher win-% compared to a stuffing-based game in the Kink category, after all statistical weighting, then that would seem pretty demonstrable as to which the community by and large preferred. I see that as working as intended.
I see that as a deviation from the original scoring rubric, if I’m correct here. The “Kink” category is most comparable to the “Use of fetish” category in the 2018 Fat Fortnight contest (since I have its rubric handy right now). There were no submissions where the judges weighted games whose kinks they preferred more heavily, right? That’s not meant to be a critique so much as it is a recognition that the rules seem to have changed a bit.
So, in my opinion, if niche interests ever had as fair a shot of winning as popular kinks, I still wonder if using a voting system instead of a dedicated judge board increases the chances they don’t win.
For every voter who votes down on principle on a given kink, there will be the gamut of fair reviewers and ardent supporters, so I see it balancing out. If it’s a kink done well in a game, that will be recognised by fair voters; everyone can universally agree on a game being flat out unsexy- niche interests or not!
You might have too much faith in me While I hope that I - representing a voter that tries to be impartial and thoroughly tests each submission - could see past that, I wonder if majority doesn’t still rule here.
Might that still deter creators from making games about their niche for the Gain Jam? If so, is there another statistical way we could stratify the Kink category differently, so it doesn’t affect overall score, maybe?
With all that in mind, I actually am more convinced that something like this community polling system would be effective to implement even if it doesn’t substitute the judges’ score. Like how Rotten Tomatoes does its Tomatometer versus the Audience Score. It would at least be an interesting trial run.
I second that a community pick will just become a popularity contest, and I don’t think that’s a good metric.
There’s a couple games in the Jam that I personally think are incredible, but have hardly any posts on the forum because the dev isn’t an artist and the art isn’t great, while a few really popular games can barely even be called a “game”, but are popular because the dev’s a big artist and the picture you stare at while you’re playing the equivalent of tic-tac-toe looks nice.
while a few really popular games can barely even be called a “game”
Although I understand where you’re coming from, I would actually prefer that just as many of these games of smaller scope and good art be made as the others. It’s not always the depth of the gameplay that makes it a fun kink game, although there are definitely submissions where the depth is the allure.
I think you underestimate how popular some games/creators are compared to others. the range between the most downloaded games and the least are about 2.5k vs 300. I don’t think the ones with just hundreds stand a chance against the ones with thousands even if the less popular one is better than the popular in every way. you are not gonna vote for a game you didn’t play, so games below the average amount of downloads are going to score worse
as an example, there are two games both with about 25 hearts but A has 1.1k-ish downloads and B has only 350-ish. even if B is better, A is gonna get a higher score. this might not be a problem now because it’s a small community and a lot/most games would make it to the judges. but when only say 25% of the games will meet the judges and a popular game gets around 25k downloads the problems in the system will be worse. also I think everyone has seen that a bigger vote has a tendency towards “dishonest” and “low information” votes.
I think the hearts work better because you don’t have to pick one over an other, so you don’t get votes just because people know about your game and not about the other. if there was a way to ensure that everyone that votes has played everything it could work but I don’t think we can make that system.
when I said niche I was thinking in terms of gameplay (which might be proof of a bias against stuff that puts gameplay over fetish, though it might be warranted). I remember a while back I did a survey asking what people liked and whatnot and when it came to genres, RPGs and visual novels was more popular than action and puzzle games for example. so making a game in a certain genre will give you an advantage over other genres
Yeah, honestly you’re not wrong. Two biggest hurdles that are hardest to refute are:
Avoiding “star power” from unduly influencing people’s choices.
The feasibility of cobbling this together- both the means and the will behind it.
Whilst I would still argue that Issue #1 is somewhat tempered by the sheer weight of pairings, it is entirely plausible that games will consistently win match-ups when they arise based on the creator’s name alone. If this was simple popularity contest that would be fine, however it’s slightly more complicated than that, given the judging categories.
Possible workarounds I’ve thought of all end up exacerbating Issue #2, potentially bloating a system that doesn’t even exist yet. Whilst I have no doubt that amongst a talented community pool of coders and developers such a project could be achievable (and partly the reason why I had included the video was to provide a tangible demonstration of someone achieving this) there has to be the will and motivation to create the system in the first place.
The current hearts system is intriguing and could be used to weight popular games accordingly; as to ensuring whether a secured heart = a successful game test, a slightly goofy idea is to give registered contestants a 4-digit code at the start of the Jam, to be incorporated into the game and sought by a player so as to unlock a “yes, I played this game” heart.
If this idea has legs at all, and the means to create and host the polling sites is feasible, I would propose trialling it with the current Gain Jam entries after the competition to see how the voting could break down and to verify the extent of some of these concerns.
I can see value in a ‘Peoples’ Choice Award’ - either as a singular prize, or one split across a select set of categories, but I don’t think it’s a substitute for an experienced panel of judges. I don’t think we can expect EVERY user on the site to review every game and therefore be in an informed position to vote on all of them. It would be too open to arguments and disagreements in the community over what games are ‘better’ and such. We’ve already seen people make posts essentially complaining about certain genres/engines/design choices and that can of worms doesn’t need to be opened, lol.
I think a lot of the issues with the Jam voting right now are related to the ergonomics of this forum platform and the volume of submissions rather than who exactly is voting. Having a myriad torrent of polls is itself a fair amount of time and effort after all. But it’s cool that you’re interested in making the Jams better.
I do agree with the popularity problem is a real problem, but that aside i don’t think most people can keep personal bias out of judging. Developer popularity is just one form of bias, what about others. For instance i really have a distaste for rpgmaker games, not that their bad games by any means, however i know i would subconsciously rate even a really good rpgmaker game below some other games just because of this. Part of the role of judges is to be impartial! I suspect more of us than not are incapable of leaving bias out of it. Also, when it comes to judging i can say that my favorite game and the games i think is most likely to win are not the same. You would be relying on people not to bump the performance or look favorably on their personal favorite even if its not the best, according to the categories and rubric. This brings in the problem of the most liked game, not being the actual best game, getting better scores than they should.
Edit: To prevent bots and the such how about only active weight gaming users get to vote and only users of a certain age and account maturity, like the account has to have made at least 2 comments or posts and given at least 5 likes, a long with be at least 3 months old before game jam starts, to be able to participate in judging, not hard to achieve.
Also how do you check that someone has actually played all the games or even both that are getting voted on in the suggested system?
Well, I don’t totally know about it being a popularity contest. Because it’s such a niche website, if you look around there really aren’t any developers that absolutely dominate the game development here. If you look at last game jam, and I won’t name names, but a popular developer made a game jam game, and people were ultimately honest in their feedback that they didn’t think it was up to the standards of their last game. If you take roops for example, he was a developer who wasn’t exactly one of the top ones on this site until he made a game for the game jam, and many many people have agreed that that game is awesome. Point is, though popularity could maybe play a small small piece of the voting, ultimately if its a good game, its a good game.
While I hope that I - representing a voter that tries to be impartial and thoroughly tests each submission - could see past that, I wonder if majority doesn’t still rule here.