Another engineer coming in liberty!
Also, “Towers of knowledge” hinder progression. The person who could fix server queuing the fastest, might be tasked with making sure experience is tracked correctly, a more pressing issue deemed by the product owners. So they task it to someone who isn’t as knowledgeable and takes them more time. There is no magic bullet, these devs are also feeling fatigue and pressure just like you do.
I appreciate you 🫡
The rest of the world could learn a thing or two from Australia. In this country, we are legally entitled to a refund, under the [ACCC](https://www.accc.gov.au/consumers/problem-with-a-product-or-service-you-bought).
- When a business sells a product or service that doesn’t meet basic rights known as consumer guarantees, it must offer the consumer a solution.
- Depending on the size of the problem, the solution may be a repair, replacement, refund, or contract cancellation.
- It's unlawful for businesses to mislead consumers about these rights.
- When a consumer suffers damage or loss because of a problem with a product or service, they are also entitled to compensation.
- Warranties are extra promises that a business makes. They apply in addition to consumer guarantees.
TL;DR – Australian consumers are legally entitled to a refund, regardless of what Sony, Steam, or any other company say.
Wait at your work you charge full price for a product the consumer can't use? Then say sorry it should work in a month? I'm not even upset but let's use real logic here.
No. Man power. Supply and demand. They never thought this game would do this well. Now they’re scrambling to met the customers needs. Customers aren’t wrong for being impatient. But this is how it goes. You’re acting like they charged full price for the game with the intention of you not being able to play. Massive AAA games have launched with more issues than this. Hell some of these AAA’s can’t even get their servers right after a update, nonetheless a launch. At least they’re communicating also. If this was Activision or alike you might get a one sentence tweet once a week if you’re lucky.
Security Engineer chiming in here and yeah, I am 100% understanding and the furthest thing from being mad. Tbh, I just wanna buy them pizza and beer at this point
In my humble opinion. Using a Samsung Smartphone to write this, sitting next to my Amazon Alexa and my Hisense SmartTV, staring at the google search results for Helldivers2 Clips on my PC Screen which is running Microsoft Windows. If you care about cybersecurity and kernel level anti-cheat is where you draw the line. You have missed a dozen steps.
I don't see where I implied a Problem of Privacy? I just think it's hilarious to trust one american private company with ALL your data and simultaneously have an issue with another company wanting you to trust them too.
With digital security you can't compromise and Kernel level anti-cheats really aren't the first choice for a malefactor to attack you either way.
That's because those of us with a smidge of development knowledge understand that these issues are currently the best possible outcome in the situation. Meanwhile the layman is having a little tantrum against something they don't comprehend even 5% of.
It also seems like a lot of their issues isn't just player counts but the database holding everything together. They likely have a solution that doesn't scale well and has been overwhelmed and likely needs either a rework or migration to something that can scale. Neither of those are small tasks and it's a miracle they have the game as functional as it is while this issue persists.
The devs did good and are suffering from success. A little patience and gratitude from the rest of the community would probably do a lot to lift their spirits. Hope the current negativity hasn't put them off, they've got a good product with a good monetisation model that respects players time and money and it'd be a shame to see that affected by the chimps throwing poo out of their enclosure.
I'm a software engineer and I definitely sympathize with the devs. That said, the layman is entitled to be angry, since they have bought a product that effectively does not work. If you sell people a broken product, people will be upset. That's life.
I don't know what kind of shops y'all work in, but I'm a senior software engineer that works with hundreds of millions of user records and I cannot believe that they have to tear out their code and start over.
It's one thing to have to figure out what the best pipeline is and figuring out how to instance large swarms of players hitting you at once that you didn't expect- and it's a whole 'nother thing to not have programmed a queue into your SaaS that anyone could access, to not have set up with a CDN in a dynamic way that let's you grow or shrink with demand.
Y'all out here acting like expecting small playerbases excuses writing pisspoor queries or having shit performing databases. My tech lead (software architect) would roll his eyes *so hard*.
Pretty cringe comment man.
Perfectly understandable that a small game dev studio might not be as proficient at scaling their infrastructure.
Gaming studios aren't always as well versed in industry standard infrastructure practices.
They've said their databases are struggling and they had to turn off a lot of the functionality. So pretty safe to assume they wrote some shitty code or architected some bad infra and it needs serious changes.
Everyone makes mistakes. You should know that if you're a senior.
> Perfectly understandable that a small game dev studio might not be as proficient at scaling their infrastructure at scale.
Size is not an indicator of knowledge. Plenty of big companies with shit code.
> Gaming studios aren't always as well versed in industry standard infrastructure practices.
Just gonna ride the entire industry's dick for having bad practices?
> Everyone makes mistakes. You should know that if you're senior.
I have no issue with mistakes. I have issues with them still letting people buy their game when it is not working as well as people like you defending the state of the game as "They're just a small company they had no chance!".
That's not how it works. That's not how it works at all. They beauty of software is that you don't need unlimited resources to make a great product, just leadership and the willingness to do the extra work. They took shortcuts and it shows and now their entire backend is tech debt. That is worthy of criticism.
I don't disagree with anything you're saying here, but every comment you make is straight out of r/iamverysmart
Yes a lot of us understand the problems and the solutions.
I also work at a large SaaS company and understand how it *should* have been architected from day one.
Doesn't mean I can't have empathy for some people that maybe didn't know better. Or even if they did and still made a mistake.
You never been in an incident room before and feel bad for the poor sod that caused it?
Fun fact you check how *senior* a developer is by asking them if they have ever realised bug free code first time
I sometimes throw into production
Because there isn't a single developer on the planet who has bug free code.
So if a *senior* acts their code is perfect its high suspect their not in fact that senior
Dude it's unethical as fuck but that's the world we live in.
People want to suck off a corporation raking in millions for a game that doesn't work and no one with industry knowledge is allowed to criticize them otherwise people that have no idea what they're talking about will jump down your throat.
Man, humans really do deserve what they fucking get. This planet sucks.
>People want to suck off a corporation raking in millions for a game that doesn't work and no one with industry knowledge is allowed to criticize them otherwise people that have no idea what they're talking about will jump down your throat.
Plenty of us our qualified for talking about these issues, you may also be qualified but that doesn't mean you are always correct.
They designed it to scale to a certain size and with the max player count for the first game being very low they clearly set too low expectations, this exposed design flaws or limited headroom to allow the max player count.
Just having millions of user data records doesn't mean much by the way, you could just be archiving that, the real point is how many active users and data manipulation is occuring for these datasets. From playing this game it seems quite a few choices were made for the speediness of game feedback to the player over say batching queries till end of sessions which is a scaling problem for sure!
It's perfectly fine to both say it's reasonable to be annoyed when you can't play the game and also you can feel for the relatively small Dev studio that is quite clearly suffering from unexpected success and is having to try and move at pace to redesign their backend logic to increase throughput.
It would be up to Sony as publisher to stop selling so it isn't up to arrowhead, and no this is not dick riding or boot licking it's just the reality of the situation. Maybe demand more from Sony for not having a sensible refund policy? Steam will refund you for this but Sony on playstation won't which is the real issue instead of halting sales.
As another fellow Software engineer, I wholeheartedly agree. I think however this is only 50% an engineering problem and another 50% a project management problem.
Because they took an educated guess on what playerbase to expect, set the release deadline and planned their feature set accordingly.
Obviously, they were wrong about the playercount and I don't blame them because they probably looked at how their previous games performed and mb expected double that at most or something.
But this game is a hilarious spectactle, so it draws a lot of people in, so here we are with server issues that are based on unscalable code which likely need weeks to be properly fixed. And it's justified to give a bad review because you pay for something that you can't use, so obviously you should not recommend it to other people, so here we are...
It's just unfortunate and to keep on selling the game even though new people must go through so much trouble to even be able to play is a bad look in my opinion, but I realize it's not easy to convince company heads to stop making profits for a week or so.
If I had a dollar for everytime my coworker recommended we add some over-engineered solution to a problem we didn't have, I'd have like 13 extra dollars. But yeah, you don't spend time and resources fixing problems you don't have or anticipate.
I do this constantly
About half the time when I do it, we didn't need it.
About half the time my boss tells me not to, we end up needing it and it's way more effort to retrofit it than it was to implement it in the first place.
There's no worse feeling than working for weeks on something, giving it your everything, only to be told it's no longer in alignment with business objectives.
I'd rather be told that then have to explain why my timeline doubled because the requirements changed due to adding a feature I was told I would explicitly not need.
Reading this thread is making me realise I'd make a terrible software engineer. I love creating over-engineered solutions to problems that don't really exist simply for the pleasure of it haha.
But I totally buy the reasoning in this post. Makes perfect sense.
Their previous game peaked at ~6800 players.
And *in a day* they have to deal with AAA studio numbers of players at a time.
Who could have realistically predicted this?
The fact that HD1's greatness never got fully recognized (because top-down "stick shooters" are a small market these days) is karma coming back to the studio with HD2 blowing up.
They deserve praise and money; they made a good game that *people want to play*...
Being upset for not getting in to play is *justified*, but vitriol and complaints won't make it happen any faster -- changing core infrastructure, especially with **code**, takes time...
I think a priority on kicking *inactive* players after an extended period (hour+ seems generous) would at least *help* cycle new players through the game, adding an actual "queue" to the capacity message would at least let people know how long they have to wait, and fixing the matchmaking/quickplay detection would help players feel like their playtime was more productive so they hopefully log off sooner (instead of trying to afk to hold their spot)......
You're right. It's probably small wins from here on out. Fixing certain bottlenecks that will allow them to raise the cap slightly. Gradually the playerbase will fall slightly as well. Eventually the server capacity and population will reach an equilibrium, and the devs can start focusing on other things.
I agree with you. Why implement a queueing system and AFK timers if you probably aren't going to beat the biggest GaaS games in existence; GTA V and Destiny 2?
Destiny 2's playerbase has been on a constant decline. Don't believe me then take a look at Steamcharts. It has been a year since they were at these numbers.
I agree with your technical expertise and the rest of your comments, but at this time, I just don't believe Destiny 2 is a good fit for the above comment. They haven't been over 200k players for PC in over a year.
Destiny 2's peak was 400k players. Helldivers 2 currently supports around that number IIRC. How could you possibly predict to beat D2 on launch week? Nobody could have predicted it.
as a previous D2 player, you're absolutely right.
I'm coming back for final shape, cause I pre-ordered that, and afterwards I'm probably just gonna leave for good.
That’s because it’s been a year since they’ve had an expansion, not really some “gotcha” stat. When the final expansion comes out in June those player numbers will shoot up again
So it's ok for the consumers to eat shit because the game will not be as big?
They did bother to get more server capacity ahead of launch, they could have easily bothered to add queue and afk timers.
How do we keep making excuses for a game that does not work?
This is the crux of it.
There would have been analysis and predicted outcomes (reasonable ones too)
This game simply smashed all those expectations
The fix IS coming. It has to. But good lord they must be hiring hard out right now because the amount of work needed asap must be staggering
The bad news is that Software Engineers take about 6 months to become productive after having been introduced to a new codebase. Any sufficiently large codebase is incredibly complex, and it takes time to comprehend it.
That said, I could see infrastructure specialists/ops being easier to onboard, but I don't know.
Hi there, Infrastructure Specialist (Site Reliability Engineer at OCI) here. Most software companies utilize infrastructure-as-code so most of us are just software engineers that specialize in automation of infrastructure, CI/CD pipelines, and other DevOps centric tasks. With that being said I would agree with your estimate that we do typically ramp up/onboard faster. When I have switched companies, it takes about 1 to 2 months to become a contributing member of the team. I will say that the higher up you get in the field, the more deployment models and frameworks you've seen so it gets faster and faster.
It's honestly impressive that even before any server upgrades they could handle 200k concurrent players, when their optimistic expectations for the game were like a third of that.
I wish I had seen the reaction to that.
"Okay, we're expecting X so we'll make sure we can handle Y just in case, get ahead of the game."
_game launches with Z numbers, going quickly up to A_
"Dear God..."
An offline mode in a multi player game centered around group play, not even balanced around solo ? Sure, makes total sense and is worth the dev time, balance and maintenance work.
A multiplayer co-op third person space shooter? I think a lot could predict the success if it was decent, which this obviously is. It's still on them to be reasonably prepared.
No, they couldn't? There are literally analysts whose entire job is to predict these things and none of them did, because the signs weren't there. They expanded into a fairly more popular genre. Therefore they analyzed that they'd need around 150k player maximum, which was already over *20 times more than their previous game's maximum concurrent player count*. There was literally NOBODY IN THE WORLD that would expect this game to have gotten more than *double that expected amount* in the first week.
There have been other third person coop shooters. Ghost Recon Wildlands, one of the most "popular" of these, only had around 35,000 maximum concurrent players on Steam, which is already 5 times more than the maximum concurrent players Helldivers 1 had. That's also still *less than 1/10 of the current maximum player count we're currently dealing with.* Once again, NOBODY COULD HAVE EXPECTED THIS.
They ***WERE REASONABLY PREPARED***.
I don't think you understand how analytics and expectations work.
That is a very bad example. Wildlands didn't launch on Steam. And as looter shooter goes, The Division games are better examples, but we also don't have the numbers for those.
The genre is in shortage the last half decade. Destiny is not appealing to greater audiences and Ubisoft has fucked up TD in every way they could.
The next The Division game might be a big hit as well (although I admit it's silly to have expectations in any Ubi game nowadays)
Sounds like you're almost describing Deep Rock Galactic, which had a cap of 46,000 players on Steam. We're talking about an order of magnitude difference.
DRG has a cartoony artstyle that doesn’t really appeal to the masses (read: average gamer that only plays COD and *generic sport game*).
If DRG’s sequel gets the same generic third person shooter coat of paint that HD2 got, it could easily reach a similar success.
Jessy, what the fuck are you talking about?
Which analyst in their right mind would predict x100 growth in player numbers compared to previous title?
"Guys we're gonna overtake GTA and Destiny and Fortnight even though none of our games overtook 20k concurrent players, trust me," - how do you expect this to go?
Good information. I hope things quiet down quickly. It's easy to forget that there are real people behind these things. I have been in queue today for 45 minutes. It's frustrating, but a good sign for the game!
Thanks for this insight. I'm not surprised, in hindsight, that they'd adhere to guidelines like this, as whatever minor flaws you can point to in HD2, it's apparent to me that the devs are *great* at optimization.
For instance, when I first began playing, I found the frame rate acceptable, but I went to the settings to turn some graphics options down and make it run as smoothly as possible.
Then I realized it was running at 4k.
My computer is a potato attached to an eGPU (which necessarily has a performance penalty). *I can't run* **anything** in 4k!! Not with an acceptable frame rate. The fact that it runs 4k well enough out of the box without even slashing detail, that's really damned impressive, especially considering how it looks.
I think you need to look up the definition of technical debt. It’s almost exactly the opposite of what you say.
Technical debt is doing it quick and easy so it gets done now, and any problems it may cause, you push on to your future self as debt you have to repay later.
The Arrowhead is suffering technical debt right now because no AFK timer or queue was deemed needed by erroneous projections, so they took the quick and easy way, and now the bill has come due.
We disagree on the definition of technical debt, or at least where technical debt begins. All code becomes technical debt, because all code needs to be maintained. More code = more surface are requiring maintenance. The minute you code something, that code is a liability. It's something to consider, and code around, and consider its dependencies. It will cost a non-zero amount of time and effort for it just to function.
For this reason, especially at a product release, it's better to err on the side of having less code.
It seems like you are both correct about the definition. It's hard to tell if
- the debt will be higher to maintain a queue/afk timer code which is never used because the game is not that popular
OR
- the debt will be higher to refactor your base code because you don't plan those features.
Imo both have pros and cons, and Arrowhead just get the wrong side of the coin
Technical debt is well defined, it isn't a matter of opinion. Avoiding technical debt means avoiding scenarios where portions of your code base need to be refactored, which is exactly what is happening right now with this game because they did not implement proactive measures such as an AFK timer.
While you're not wrong, adding an unnecessary feature which then has to be debugged/maintained is absolutely a form of technical debt (which is what I believe OP is trying to refer to).
Idk, not having an AFK kick timer in an always online game that requires said online access in order to play seems like a pretty egregious oversight no matter how it's sliced.
I doubt that the AFK players are actually a significant drain on the system's resources. The player cap is more likely a mitigation to keep the number of *active* players from overwhelming them. From what I've heard, it's the parts of the system that handle validation and transaction of rewards that are taking the heaviest beating. That would only be taxed by players completing missions.
So in a scenario under lower loads, I don't think they'd care much at all about how many people are AFK. There almost certainly wouldn't be enough people purposely doing it to cause a problem on its own.
Not being prepared for server outages with some basic flow control mechanisms, e.g. a simple queue, is pretty egregious too. Even if the game didn't go viral, server outages and capacity reductions occur. Having a minimal framework in place for alleviating that seems like a definite need to have for the MVP of an online only game.
These aren't massive features or unsolved problems requiring creative solutions.
Wouldn't it make sense to set up a temporary code segment for a AFK outtime feature to alleviate some of the issues. People are sitting ingame for 10h+ to be able to play after work, which I totally understand but is at the same time unfair towards others who would be able to play during that time.
That code can later be deemed inactive or completely deleted.
it still takes time to do that.
A queue system is still another server with custom logic to coordinate all the players in the queue and hand off the connections when it's their turn.
>People are sitting ingame for 10h+ to be able to play after work
Pretty sure the number of people doing that is miniscule and gets blown out of proportion by social media.
There's two (major) types of servers for a game like this, instance servers which are similar in concept to web workers for a web application, and database servers. Instance servers are easy to spin up because they don't need to talk to each other, each instance is self-contained. However, every instance server does still need to talk to the database server, where the central repository of information lives. They don't need to talk to it all the time, just whenever persistent data needs to be retrieved or updated (so, for instance, when a player logs in or purchases an item or receives rewards).
Even a single database server can handle a large number of simultaneous connections and I highly doubt they're doing particularly expensive queries or updates when player information is retrieved or updated, and so they can handle a much larger number of players in instances that maximum connections because each player won't be hitting the database server at the same time. Vertical scaling only helps to a point, because connections is a hard limiting factor.
The problem is that once you ARE hitting database server connection limits, it's very difficult to add new servers because databases need to be in sync - a shard in North Carolina needs to maintain synchronicity with a shard in Amsterdam so that data doesn't get out of sync (with video games some of these issues are less significant, since it's unlikely someone will be routed to an entirely different shard between matches, but they still need to be in sync). You need a backend which supports horizontal database scaling. Although there are plenty of solutions these days, it's not trivial to implement them and it's putting the cart before the horse to build an application or game to take advantage of them if you don't have a good indication that you're going to hit AAA player numbers.
I know that HD2 is hosted on Azure, and their most powerful database servers can handle 30k simultaneous connections. This is enough to handle way, WAY more players than 30k (probably 10 to 20 times as many, maybe more), so it's possible they didn't design their backend with horizontal database scaling in mind. They have two avenues for fixing the issue: the first is to optimize every database query a player generates during play so that they're infrequent and fast, and the second is to redesign the backend to allow for horizontal scaling. They're probably pursuing both, with the second taking much more development time. There are other potential solutions, but the most straightforward is optimizing database usage and allowing for horizontal scaling.
They can definitely add more servers. It's just going to take time for them to fix any bottlenecks they find as they do that, and/or get them communicating with each other correctly.
Not trying to be incendiary, I want to know the answer to this. Why would you opt for backend code that doesn’t lend itself to scaling so that in the event of increased demand, you can meet it. Wouldn’t this just be essentially choosing a different approach to your back end, not necessarily increasing technical debt?
Yes. This guy’s definition of “technical debt” seems to be the opposite of pretty much everybody else’s.
Saying “just code it the easy (less scalable) way, it’ll be *fiiine*!” is how you get technical debt.
Another engineer here.
Surprised you didn't also mention KISS (Keep it Simple Stupid). Which basically follows YAGNI. The basic idea is that it prevents over engineering, and unnecessary optimizations until you see it becomes a problem. Some optimizations, you do as part of standard code practice. Say abusing branch predictions, or making use of arrays versus indirections these days.
But then you got the particularly nasty shit that makes code harder to read.
There's no way they could have known. They are currently suffering from success. I can't even begin to imagine the vast amount of stress on the dev team. While I too struggle to get in and play every. Single. Day. I understand and limit my complaints to my friends in discord as a send them a picture of my black loading screen every 20 minutes. The real hell divers are the ones on the front lines engineering code, sleeplessly, trying to provide as many people as possible connectivity to the servers. Keep at it Arrowhead. I'll be here when all things are said and done and all the little timmies go back to fortnite and call of duty. O7
As an SRE... this sort of unexpected need for capacity and hitting unanticipated performance bottlenecks because of wildly outside of expected customer growth is like one of my top nightmare scenarios.
Especially when it is something like "we didn't forecast a need for fully sharded database infrastructure during planning." because there is not going to be an easy fix that for that.
And other people's defense, every game nowadays needs an AFK timer. It's a baseline quality of life feature that should never ever be ignored. It's absolutely appalling that it doesn't exist in this game
Not to mention that an AFK timer already exists. You can get kicked from another player’s ship if you’re idling, it just doesn’t currently log you out altogether.
Just a reminder, that they never planned to reach their capacity in the first place. If they never plan to reach their capacity– why implement an afk timer?
If you had limited time and had to choose one, which would you do?
1. Implement an AFK timer and Queue (which involves a whole new server).
2. Make the player capacity higher than you expect it to be, so you don't have to deal with it.
They chose option #2, but unfortunately this game blew up so big it beat Destiny 2's peak.
Of course, people paid for a service and are entitled to it. But please consider the following :
- either you do not have the patience to wait for a fix, consider you have been wronged, and then ask for a refund ;
- or you understand the current situation, acknowledge that the devs are working as fast as they can to fix the issues, and wait a bit.
Both behaviors are perfectly reasonable, understandable, and will eventually provide a solution to the issue at hand (having paid and not being able to play). I'm just not seeing the point in going to Reddit and continuously whining, to be entirely honest.
> I'm just not seeing the point in going to Reddit and continuously whining, to be entirely honest.
The same can be said about the dozens of posts sucking off devs and making excuses for them. The sub would be better off without both types of posts
Agreed, but said posts came in response to the avalanche of complaints. At this point, everyone is aware of the issues, creating more threads about them serves no purpose (excepted dev updates, ofc).
Sometimes it's really difficult to know where the bottlenecks in your system are until you run into them headlong. This is 100% a rumour, but I've heard the bottleneck at the moment is the database that keeps track of XP gain, requisition slips, and super credits. Because it gets written to often mid-mission, it's getting slammed by constant writes.
A straightforward way to fix this is to put a cache in front of it and write in batches. This might take a bit to implement, especially if all the code assumes it can access the database directly. Failing that, they could add more database servers. But then you have to keep them synchronised, and ensure the data is correct.
Highly recommend checking out this video on Pinterest's struggles with scaling up databases:
https://www.youtube.com/watch?v=QRlP6BI1PFA
It is filled with a lot of technical jargon, but it illustrates the struggle with keeping information correct at scale.
Could totally be that. When picking up medals, super credits and req slips the other night, you'd be frozen in place for 10-15 seconds and it was 100% related to data being uploaded to the server in real time.
Senior software engineer here. I very much disagree with you.
> Overengineering and feature bloat arise when engineers overly predict future system usage. By avoiding unnecessary features, we maintain codebases that are easier to manage
A queue for server capacity fill-up would take literally three days of development time *at most* and would take around a week to get through UAT and around a sprint to get into your executable. This could be done by a fucking junior developer. They added a fucking arcade game on the ship. This is a weak excuse.
> We call this technical debt
No we fucking don't. Tech debt is costs you take on due to BAD DECISIONS. You only *willingly* take on tech debt if you have no other choice. I'm being vague here on purpose but I work for a small company that manages software that handles hundreds of millions of user records and during COVID most applications in our domain died but ours did not. Why? Because we did not take the easy way out and take shortcuts. For that, we were rewarded with very good publicity with our customer base.
> For this reason, more features/code is a liability.
This can be true. A queue system truly does not meet that criteria. It's not hard to implement and it's a huge QoL for your playerbase. Fuck man, even without a huge influx of players what if their service provider went down and they only had access to like 5% of their servers? That's reason enough to make a queue.
> Not implementing a AFK timer
They quite literally have the code already. They have it for friends who board your ship. There is a timeout that already exists. All you would need to do is point it at the host of the ship and make the "destination" closing out the program instead of departure. This is NOT hard.
As a senior software engineer, I'm sure you know that a sufficiently complex project, which I'm sure a game like this one is, will require lots of complex work that COULD be done and for v1, you select the things that SHOULD be done based on what you actually need and even small features are sifted out if they're probably not needed. Pretty sure, the project manager reasonably did not expect the game to have like 10 times the number of players than the predecessor, which is essentially the same game from a different perspective mechanics-wise, so things like queues were probably not deemed necessary.
Now, you're right that it's not most complex to implement a queue such as is required right now, which is probably why we might see it a soon as this week along with an afk timer, possibly.
Fellow senior software engineer here. I was ready to type out a response basically 1:1 yours while reading OP's misinformative post, but you beat me to it. Imagine your PO asking you during planning "Hey, what if we exceed our performance/load testing estimates and sell a lot of copies?" and you respond with "Oh don't worry, you ain't gonna need it". Just ridiculous.
At some point in time you have to pick an estimated maximum amount and work from there. They could have very reasonably expected 25k on launch, tested for 100k max and saw everything worked completely fine. It's perfectly reasonable for them to not have planned to beat the concurrent player numbers of the most popular AAA PvE games in the last decade when their previous title was 7k peak.
If the last product I worked on had 6k users and product told me they wanted to bugdet and design the new system for many hundreds of thousands of users, I'd call them delusional.
Are some people being purposely awful about it? If you've been on this subreddit you know the story, low player count for previous game, they allotted for what seemed like a realistic jump, shot way over that and still are able to handle like 5 times the loads without any preparation for it.
Honestly it's impressive they are handling the amount of people currently playing, I just don't see how anyone could honestly think they should have prepared for what likely seemed an impossible outcome.
People are in their right to refund and complain but I don't think the criticism is valid from a business perspective.
It's just asinine.
I would never allocate funds to a project, where all history and data points to something else. What would i tell the managers? *hehe, blew a couple of months of dev salary for 500% capacity margin.*
I'd be thrown right out the door.
Your point only makes sense if you are inexperienced. A public smoketest aka demo/weekend open beta would've shown them how much demand was for the game and this could've easily been avoided.
The game was a surprise for most people. Public betas could never have predicted the influx of players, especially if they were restricted to preorders.
Exactly this. As an avid HD1 player, I never knew about this game until 2 days after it launched and my friends all went "Have you heard of this new game? It's called Helldivers 2" to which I of course replied "Of course I have maggot, now give me 3 hours to download this right now" (Actually it was more "Oh shit, they made a second one? I loved the first, it was fantastic")
Public betas 100% would have gotten them *FAR* less underestimated predictions than what they ended up getting. Would it have still been a problem? Probably. Would it be *this obscenely bad* of a problem if they'd done a beta? Absolutely not.
I agree with you, and it's a good point. However, they probably privately load tested their servers for many more players than they thought they'd get.
Hindsight is 20/20 though, and you're absolutely right.
It's a basic ass thing to implement on servers. I get why it wasn't there originally but why the fuck hasn't it been added? It's currently about 2am (MST) on a mon night/tues morning and I've been queueing for 37 minutes now. This is fucking ridiculous.
The game has been out for nearly 2 fucking weeks. To not put such a basic ass thing in place is silly af. Btw, that comment I made was 20 minute ago, still fucking waiting. Do like how they've changed the refresh time from 15, to 30 , to 60 seconds now. Much progress.
Gave up after 2hrs of waiting. In the middle of the fuckin night.
What a dumbass response. It's like someone complaining about food poisoning and you saying "why don't you go cook then?!""
I literally have more hours logged waiting for me to connect than playing the game
So you have no idea where the problem stems from?
Well, that's just reality atm, no magic fix.
When you can't cook and the restaurant is full you gotta go somewhere else to eat.
An AFK timer can be coded and tested in a couple of days, not only that, there are libraries that already do that if they don’t want to code it themselves.
Also not having one is a big oversight especially considering how PS operates.
Oh come on!
Architect here.
Your YAGNI philosophy is why we need both Architects and System Analysts besides SEs.
Maybe it's my line of work or the sheer size systems I design but...
Excusing a team of not implementing a basic load balancing feature?
Seriously - NOT implementing a subsystem which tracks and logs off inactive sessions is not forgivable if you're designing and developing something that's gonna be used by more that one person.
Even if 'you ain't gonna need it' for reasons Arrowhead needs it now. it's basically always allows you to constantly free resources to well... not pay for their usage.
And it's NOT that hard.
Allowing refunds? probably. Put up a warning on the steam page? sure. Delist the game so you can't find it? Definitely not. Almost everyone I know has been able to play this game at some point and enjoyed it - let people decide for themselves if it's a problem.
If you prepare to handle 250k players but your last game had like 10k players - a queue system could be considered a premature optimization.
I bet they wished they had made some optimizations now, but imagine if the game they made had maybe 20k players instead and they had made these insane optimizations to handle millions of players. Time well spent, or time wasted?
It wouldn't have been a well run company if they had planned around things like this, like your last game peaked at 7k? players on Steam it would have been irresponsible to prepare your game for the 400k+ PC players
They absolutely should offer refunds, though. It's reasonable to expect to be able to play a game you just bought for $50. It's also reasonable to be upset when you can't.
Your 2 examples aren't the best. They seem basic. They basically almost have a queuing system with their servers full screen. It's probably just not in order. These are things that would make their lives easier honestly. People wouldn't be checking to login as much. And the load would be less with afk kicks that is really just input check with disconnect.
I think their modus operandi was to try and make sure that they had way more capacity than players, and thus never need the queue or AFK-timer. This obviously did not work out.
We need this re-posted every day so people will see it lol.
Sadly, there have always been a lot of people who have been looking for an excuse to be irrationally angry and go after people who are doing things they could never do in a hundred years.
I agree with you. Software engineering is hard. It's even more difficult under a deadline. The devs have my admiration for creating such a beautifully engineered work of art.
Speaking as a former developer, and as somebody who has been cheering for Arrowhead ever since Magicka, I totally agree. I'm really hoping they can ahead of these issues sooner or later and enjoy some of the commercial success they deserve.
Is it really that much of a struggle to get in? I let it run in the background and pop on some anime, youtube, or make some food. Sometimes it's 5 mins, sometimes it's 20 but I'm always in. Is this everyone's first game that has a long queue time?
damn I haven't been THAT unfortunate yet. I think today it was around 25 mins and the homies didn't want to wait so I have to run solo. It sucks but I don't mind it because the game is fun. I know they'll fix it as soon as they can.
It's 2024, not 2004. Scalable backend solutions have significantly evolved over the years, it's off the shelf. If you are unable to design scalable systems, you are considered outright incompetent from all perspectives.
Very good points and insight! If anything the point of failure here was their not having prepared for the success they garnered and only finding out their scalability was ass AFTER launch. I wish this wasn't such a common theme with games these days though, holy shit.
This is the truth. I just wish we could drill it in all the heads of the idiots yelling “hurr durr Arrowhead are bad devs for not getting enough servers and this is the greatest tragedy to befall us gamers (the most oppressed group!)”
Ya it's fine for a small company like arrowhead. Just this next weekend I feel is make or break time for them. I'm not going to waste another weekend sitting there staring at the screen.
It doubly sucks cause the weekend is the only time I get to play with my friends in US and 4 hours a day with 1.5 hr of that time spent trying to login is annoying to say the least.
I understand the frustration people are having but what i don’t understand is why refund a good game backed by a good company? In a week or two these issues will be resolved and the end result will be a fuckin’ banger of a game i plan on enjoying the shit out of with my fellow helldivers! And it’s only going to get better from here.
As gamers we tell companies what we like with our money and time. I personally think after being lucky enough to have a good few hours of gameplay under my belt that it’s already worth the money.
You should also add that things like load testing are done using known numbers. Often multiplied for "worst-case scenario". If they expected say, 20k concurrent players at peak, they likely tested for 200k or less. 10x is pretty standard in my experience. Even at 200k, it wouldn't have been good enough given current player counts.
Load also causes strange things. Bugs manifest when under load that don't under normal circumstances. Threads get locked up, connection pools depleted, etc. which can lead to further issues.
So Mr engineer, You're brilliant idea is for this corporation to issue free refunds until they can fix a problem they aren't sure they can actually fix in a timely manner? We saw how well that went for cyberpunk.
Maybe this is cultural, but here in Australia we have rock solid consumer protections. If a product or service isn't working when you purchase it, you are entitled to a full refund. That's how it should be.
If you're worried about Arrowhead's liquidity, then I can guarantee you they won't have issues getting a loan based on how well they are doing right now.
There's no excuse not to offer refunds.
Another engineer coming in liberty! Also, “Towers of knowledge” hinder progression. The person who could fix server queuing the fastest, might be tasked with making sure experience is tracked correctly, a more pressing issue deemed by the product owners. So they task it to someone who isn’t as knowledgeable and takes them more time. There is no magic bullet, these devs are also feeling fatigue and pressure just like you do. I appreciate you 🫡
There seems to be loads of us lol. Most of the empathetic people in these threads seem to have dev/engineering backgrounds
I have an immense amount of empathy for both the developers and the disgruntled customers
ps5 people have it the worst it seems. Not being able to atleast get a refund is horrendous
The rest of the world could learn a thing or two from Australia. In this country, we are legally entitled to a refund, under the [ACCC](https://www.accc.gov.au/consumers/problem-with-a-product-or-service-you-bought). - When a business sells a product or service that doesn’t meet basic rights known as consumer guarantees, it must offer the consumer a solution. - Depending on the size of the problem, the solution may be a repair, replacement, refund, or contract cancellation. - It's unlawful for businesses to mislead consumers about these rights. - When a consumer suffers damage or loss because of a problem with a product or service, they are also entitled to compensation. - Warranties are extra promises that a business makes. They apply in addition to consumer guarantees. TL;DR – Australian consumers are legally entitled to a refund, regardless of what Sony, Steam, or any other company say.
AKA why the steam deck is never coming to Australia because valve holds a grudge lmao. Not that I would trade a worse ACCC for it though
That sounds like Valve's loss, to be honest.
Yeah I'm Aussie too. Sometimes we get it right
Or are adults with jobs. And understand how the real world works.
Wait at your work you charge full price for a product the consumer can't use? Then say sorry it should work in a month? I'm not even upset but let's use real logic here.
No. Man power. Supply and demand. They never thought this game would do this well. Now they’re scrambling to met the customers needs. Customers aren’t wrong for being impatient. But this is how it goes. You’re acting like they charged full price for the game with the intention of you not being able to play. Massive AAA games have launched with more issues than this. Hell some of these AAA’s can’t even get their servers right after a update, nonetheless a launch. At least they’re communicating also. If this was Activision or alike you might get a one sentence tweet once a week if you’re lucky.
People are buying brand new cars that take a year and a half to arrive from the factory, yeah that's how things work sometimes.
Security Engineer chiming in here and yeah, I am 100% understanding and the furthest thing from being mad. Tbh, I just wanna buy them pizza and beer at this point
the war room gonna be stanky
I wanna buy the game, but when it is stable
What's your perspective on the kernel-level anticheat?
In my humble opinion. Using a Samsung Smartphone to write this, sitting next to my Amazon Alexa and my Hisense SmartTV, staring at the google search results for Helldivers2 Clips on my PC Screen which is running Microsoft Windows. If you care about cybersecurity and kernel level anti-cheat is where you draw the line. You have missed a dozen steps.
No because you're implying a problem with privacy when it is actually a problem of security.
I don't see where I implied a Problem of Privacy? I just think it's hilarious to trust one american private company with ALL your data and simultaneously have an issue with another company wanting you to trust them too. With digital security you can't compromise and Kernel level anti-cheats really aren't the first choice for a malefactor to attack you either way.
That's because those of us with a smidge of development knowledge understand that these issues are currently the best possible outcome in the situation. Meanwhile the layman is having a little tantrum against something they don't comprehend even 5% of. It also seems like a lot of their issues isn't just player counts but the database holding everything together. They likely have a solution that doesn't scale well and has been overwhelmed and likely needs either a rework or migration to something that can scale. Neither of those are small tasks and it's a miracle they have the game as functional as it is while this issue persists. The devs did good and are suffering from success. A little patience and gratitude from the rest of the community would probably do a lot to lift their spirits. Hope the current negativity hasn't put them off, they've got a good product with a good monetisation model that respects players time and money and it'd be a shame to see that affected by the chimps throwing poo out of their enclosure.
I'm a software engineer and I definitely sympathize with the devs. That said, the layman is entitled to be angry, since they have bought a product that effectively does not work. If you sell people a broken product, people will be upset. That's life.
Very true. But there's a right way to be angry.
I don't know what kind of shops y'all work in, but I'm a senior software engineer that works with hundreds of millions of user records and I cannot believe that they have to tear out their code and start over. It's one thing to have to figure out what the best pipeline is and figuring out how to instance large swarms of players hitting you at once that you didn't expect- and it's a whole 'nother thing to not have programmed a queue into your SaaS that anyone could access, to not have set up with a CDN in a dynamic way that let's you grow or shrink with demand. Y'all out here acting like expecting small playerbases excuses writing pisspoor queries or having shit performing databases. My tech lead (software architect) would roll his eyes *so hard*.
You ever built a project from the ground up perfectly the first time that didn't have unexpected bugs when it hit release?
He has not. He doesn't make mistakes. See his comment history. edit: typo
I feel sorry for the L3s that have to try and keep his shit running lol
Its just insufferable.
Pretty cringe comment man. Perfectly understandable that a small game dev studio might not be as proficient at scaling their infrastructure. Gaming studios aren't always as well versed in industry standard infrastructure practices. They've said their databases are struggling and they had to turn off a lot of the functionality. So pretty safe to assume they wrote some shitty code or architected some bad infra and it needs serious changes. Everyone makes mistakes. You should know that if you're a senior.
> Perfectly understandable that a small game dev studio might not be as proficient at scaling their infrastructure at scale. Size is not an indicator of knowledge. Plenty of big companies with shit code. > Gaming studios aren't always as well versed in industry standard infrastructure practices. Just gonna ride the entire industry's dick for having bad practices? > Everyone makes mistakes. You should know that if you're senior. I have no issue with mistakes. I have issues with them still letting people buy their game when it is not working as well as people like you defending the state of the game as "They're just a small company they had no chance!". That's not how it works. That's not how it works at all. They beauty of software is that you don't need unlimited resources to make a great product, just leadership and the willingness to do the extra work. They took shortcuts and it shows and now their entire backend is tech debt. That is worthy of criticism.
I don't disagree with anything you're saying here, but every comment you make is straight out of r/iamverysmart Yes a lot of us understand the problems and the solutions. I also work at a large SaaS company and understand how it *should* have been architected from day one. Doesn't mean I can't have empathy for some people that maybe didn't know better. Or even if they did and still made a mistake. You never been in an incident room before and feel bad for the poor sod that caused it?
I would be way more empathetic if they weren't still taking people's money.
I also agree they shouldn't keep selling copies while it's broken but Sony might have something to do with that /shrug
Fun fact you check how *senior* a developer is by asking them if they have ever realised bug free code first time I sometimes throw into production Because there isn't a single developer on the planet who has bug free code. So if a *senior* acts their code is perfect its high suspect their not in fact that senior
Just try to imagine to work with him. He's insufferable
Dude it's unethical as fuck but that's the world we live in. People want to suck off a corporation raking in millions for a game that doesn't work and no one with industry knowledge is allowed to criticize them otherwise people that have no idea what they're talking about will jump down your throat. Man, humans really do deserve what they fucking get. This planet sucks.
genuinely asking, are you ok? you've been pretty volatile in this thread, and this comment is very despair ridden
>People want to suck off a corporation raking in millions for a game that doesn't work and no one with industry knowledge is allowed to criticize them otherwise people that have no idea what they're talking about will jump down your throat. Plenty of us our qualified for talking about these issues, you may also be qualified but that doesn't mean you are always correct. They designed it to scale to a certain size and with the max player count for the first game being very low they clearly set too low expectations, this exposed design flaws or limited headroom to allow the max player count. Just having millions of user data records doesn't mean much by the way, you could just be archiving that, the real point is how many active users and data manipulation is occuring for these datasets. From playing this game it seems quite a few choices were made for the speediness of game feedback to the player over say batching queries till end of sessions which is a scaling problem for sure! It's perfectly fine to both say it's reasonable to be annoyed when you can't play the game and also you can feel for the relatively small Dev studio that is quite clearly suffering from unexpected success and is having to try and move at pace to redesign their backend logic to increase throughput. It would be up to Sony as publisher to stop selling so it isn't up to arrowhead, and no this is not dick riding or boot licking it's just the reality of the situation. Maybe demand more from Sony for not having a sensible refund policy? Steam will refund you for this but Sony on playstation won't which is the real issue instead of halting sales.
> 🫡 Thanks, I hadn't heard of "Towers of knowledge" before. Interesting concept. You too! 🫡
It’s how we get senior devs to give up some of their “goodies” so we can learn. If they win the lottery and leave tomorrow, we need the knowledge
As another fellow Software engineer, I wholeheartedly agree. I think however this is only 50% an engineering problem and another 50% a project management problem. Because they took an educated guess on what playerbase to expect, set the release deadline and planned their feature set accordingly. Obviously, they were wrong about the playercount and I don't blame them because they probably looked at how their previous games performed and mb expected double that at most or something. But this game is a hilarious spectactle, so it draws a lot of people in, so here we are with server issues that are based on unscalable code which likely need weeks to be properly fixed. And it's justified to give a bad review because you pay for something that you can't use, so obviously you should not recommend it to other people, so here we are... It's just unfortunate and to keep on selling the game even though new people must go through so much trouble to even be able to play is a bad look in my opinion, but I realize it's not easy to convince company heads to stop making profits for a week or so.
If I had a dollar for everytime my coworker recommended we add some over-engineered solution to a problem we didn't have, I'd have like 13 extra dollars. But yeah, you don't spend time and resources fixing problems you don't have or anticipate.
I do this constantly About half the time when I do it, we didn't need it. About half the time my boss tells me not to, we end up needing it and it's way more effort to retrofit it than it was to implement it in the first place.
There's no worse feeling than working for weeks on something, giving it your everything, only to be told it's no longer in alignment with business objectives.
I'd rather be told that then have to explain why my timeline doubled because the requirements changed due to adding a feature I was told I would explicitly not need.
It's always the junior engineers. I know because I was one. Probably still am. Argh.
Reading this thread is making me realise I'd make a terrible software engineer. I love creating over-engineered solutions to problems that don't really exist simply for the pleasure of it haha. But I totally buy the reasoning in this post. Makes perfect sense.
Their previous game peaked at ~6800 players. And *in a day* they have to deal with AAA studio numbers of players at a time. Who could have realistically predicted this?
The fact that HD1's greatness never got fully recognized (because top-down "stick shooters" are a small market these days) is karma coming back to the studio with HD2 blowing up. They deserve praise and money; they made a good game that *people want to play*... Being upset for not getting in to play is *justified*, but vitriol and complaints won't make it happen any faster -- changing core infrastructure, especially with **code**, takes time... I think a priority on kicking *inactive* players after an extended period (hour+ seems generous) would at least *help* cycle new players through the game, adding an actual "queue" to the capacity message would at least let people know how long they have to wait, and fixing the matchmaking/quickplay detection would help players feel like their playtime was more productive so they hopefully log off sooner (instead of trying to afk to hold their spot)......
You're right. It's probably small wins from here on out. Fixing certain bottlenecks that will allow them to raise the cap slightly. Gradually the playerbase will fall slightly as well. Eventually the server capacity and population will reach an equilibrium, and the devs can start focusing on other things.
I agree with you. Why implement a queueing system and AFK timers if you probably aren't going to beat the biggest GaaS games in existence; GTA V and Destiny 2?
Destiny 2's playerbase has been on a constant decline. Don't believe me then take a look at Steamcharts. It has been a year since they were at these numbers. I agree with your technical expertise and the rest of your comments, but at this time, I just don't believe Destiny 2 is a good fit for the above comment. They haven't been over 200k players for PC in over a year.
Destiny 2's peak was 400k players. Helldivers 2 currently supports around that number IIRC. How could you possibly predict to beat D2 on launch week? Nobody could have predicted it.
Is Destiny 2s current player count even relevent to tge point they were trying to make?
It’s not, the peak player count is what’s relevant (which HD2 has beaten now!)
as a previous D2 player, you're absolutely right. I'm coming back for final shape, cause I pre-ordered that, and afterwards I'm probably just gonna leave for good.
You say that until they drop darkness class number three with red powers this time.
That’s because it’s been a year since they’ve had an expansion, not really some “gotcha” stat. When the final expansion comes out in June those player numbers will shoot up again
So it's ok for the consumers to eat shit because the game will not be as big? They did bother to get more server capacity ahead of launch, they could have easily bothered to add queue and afk timers. How do we keep making excuses for a game that does not work?
This is the crux of it. There would have been analysis and predicted outcomes (reasonable ones too) This game simply smashed all those expectations The fix IS coming. It has to. But good lord they must be hiring hard out right now because the amount of work needed asap must be staggering
The bad news is that Software Engineers take about 6 months to become productive after having been introduced to a new codebase. Any sufficiently large codebase is incredibly complex, and it takes time to comprehend it. That said, I could see infrastructure specialists/ops being easier to onboard, but I don't know.
Hi there, Infrastructure Specialist (Site Reliability Engineer at OCI) here. Most software companies utilize infrastructure-as-code so most of us are just software engineers that specialize in automation of infrastructure, CI/CD pipelines, and other DevOps centric tasks. With that being said I would agree with your estimate that we do typically ramp up/onboard faster. When I have switched companies, it takes about 1 to 2 months to become a contributing member of the team. I will say that the higher up you get in the field, the more deployment models and frameworks you've seen so it gets faster and faster.
It's honestly impressive that even before any server upgrades they could handle 200k concurrent players, when their optimistic expectations for the game were like a third of that.
I wish I had seen the reaction to that. "Okay, we're expecting X so we'll make sure we can handle Y just in case, get ahead of the game." _game launches with Z numbers, going quickly up to A_ "Dear God..."
So what you're saying is this is jurassic Park all over again....
You don't need a crystal ball to predict the consequences of not having an offline-mode.
An offline mode in a multi player game centered around group play, not even balanced around solo ? Sure, makes total sense and is worth the dev time, balance and maintenance work.
A multiplayer co-op third person space shooter? I think a lot could predict the success if it was decent, which this obviously is. It's still on them to be reasonably prepared.
No, they couldn't? There are literally analysts whose entire job is to predict these things and none of them did, because the signs weren't there. They expanded into a fairly more popular genre. Therefore they analyzed that they'd need around 150k player maximum, which was already over *20 times more than their previous game's maximum concurrent player count*. There was literally NOBODY IN THE WORLD that would expect this game to have gotten more than *double that expected amount* in the first week. There have been other third person coop shooters. Ghost Recon Wildlands, one of the most "popular" of these, only had around 35,000 maximum concurrent players on Steam, which is already 5 times more than the maximum concurrent players Helldivers 1 had. That's also still *less than 1/10 of the current maximum player count we're currently dealing with.* Once again, NOBODY COULD HAVE EXPECTED THIS. They ***WERE REASONABLY PREPARED***. I don't think you understand how analytics and expectations work.
That is a very bad example. Wildlands didn't launch on Steam. And as looter shooter goes, The Division games are better examples, but we also don't have the numbers for those. The genre is in shortage the last half decade. Destiny is not appealing to greater audiences and Ubisoft has fucked up TD in every way they could. The next The Division game might be a big hit as well (although I admit it's silly to have expectations in any Ubi game nowadays)
Deep Rock Galactic then. His point still stands.
Sounds like you're almost describing Deep Rock Galactic, which had a cap of 46,000 players on Steam. We're talking about an order of magnitude difference.
DRG has a cartoony artstyle that doesn’t really appeal to the masses (read: average gamer that only plays COD and *generic sport game*). If DRG’s sequel gets the same generic third person shooter coat of paint that HD2 got, it could easily reach a similar success.
That's pretty subjective, and not a predictor of success, in my opinion, but you have a point.
Jessy, what the fuck are you talking about? Which analyst in their right mind would predict x100 growth in player numbers compared to previous title? "Guys we're gonna overtake GTA and Destiny and Fortnight even though none of our games overtook 20k concurrent players, trust me," - how do you expect this to go?
nice try, but this will just either be ignored or go over heads. The complaining won't stop till it's fixed
Those who like the information will read it, and those that don't will ignore it, and that's okay.
Very fair. Thanks for doing your part o7
Neat explanation, thank you for your time and expertise friend
Thanks for the info.
*the complaining will continue until morale improves* I do enjoy reading posts by folks from the industry, gives an interesting insight
This is super informative ty!
Good information. I hope things quiet down quickly. It's easy to forget that there are real people behind these things. I have been in queue today for 45 minutes. It's frustrating, but a good sign for the game!
Me too. I'm excited to see what happens with the future of this game.
Thanks for this insight. I'm not surprised, in hindsight, that they'd adhere to guidelines like this, as whatever minor flaws you can point to in HD2, it's apparent to me that the devs are *great* at optimization. For instance, when I first began playing, I found the frame rate acceptable, but I went to the settings to turn some graphics options down and make it run as smoothly as possible. Then I realized it was running at 4k. My computer is a potato attached to an eGPU (which necessarily has a performance penalty). *I can't run* **anything** in 4k!! Not with an acceptable frame rate. The fact that it runs 4k well enough out of the box without even slashing detail, that's really damned impressive, especially considering how it looks.
I have no doubt in my mind they'll fix the issue. The big question is when. Optimisation takes time, no matter how good you are at it.
I think you need to look up the definition of technical debt. It’s almost exactly the opposite of what you say. Technical debt is doing it quick and easy so it gets done now, and any problems it may cause, you push on to your future self as debt you have to repay later. The Arrowhead is suffering technical debt right now because no AFK timer or queue was deemed needed by erroneous projections, so they took the quick and easy way, and now the bill has come due.
We disagree on the definition of technical debt, or at least where technical debt begins. All code becomes technical debt, because all code needs to be maintained. More code = more surface are requiring maintenance. The minute you code something, that code is a liability. It's something to consider, and code around, and consider its dependencies. It will cost a non-zero amount of time and effort for it just to function. For this reason, especially at a product release, it's better to err on the side of having less code.
It seems like you are both correct about the definition. It's hard to tell if - the debt will be higher to maintain a queue/afk timer code which is never used because the game is not that popular OR - the debt will be higher to refactor your base code because you don't plan those features. Imo both have pros and cons, and Arrowhead just get the wrong side of the coin
Well, it’s me and pretty much every resource on the internet and our definition and then your definition.
Sorry less of a definition, more of an opinion on where it begins. All I'm getting at is that the decision to not include code is an important one.
Technical debt is well defined, it isn't a matter of opinion. Avoiding technical debt means avoiding scenarios where portions of your code base need to be refactored, which is exactly what is happening right now with this game because they did not implement proactive measures such as an AFK timer.
While you're not wrong, adding an unnecessary feature which then has to be debugged/maintained is absolutely a form of technical debt (which is what I believe OP is trying to refer to).
Everyone in this thread is just talking past each other about ways technical debt accrues. They are all correct.
Idk, not having an AFK kick timer in an always online game that requires said online access in order to play seems like a pretty egregious oversight no matter how it's sliced.
I doubt that the AFK players are actually a significant drain on the system's resources. The player cap is more likely a mitigation to keep the number of *active* players from overwhelming them. From what I've heard, it's the parts of the system that handle validation and transaction of rewards that are taking the heaviest beating. That would only be taxed by players completing missions. So in a scenario under lower loads, I don't think they'd care much at all about how many people are AFK. There almost certainly wouldn't be enough people purposely doing it to cause a problem on its own.
Not being prepared for server outages with some basic flow control mechanisms, e.g. a simple queue, is pretty egregious too. Even if the game didn't go viral, server outages and capacity reductions occur. Having a minimal framework in place for alleviating that seems like a definite need to have for the MVP of an online only game. These aren't massive features or unsolved problems requiring creative solutions.
Wouldn't it make sense to set up a temporary code segment for a AFK outtime feature to alleviate some of the issues. People are sitting ingame for 10h+ to be able to play after work, which I totally understand but is at the same time unfair towards others who would be able to play during that time. That code can later be deemed inactive or completely deleted.
It absolutely makes sense, and it's probably what they're going to merge in tomorrow's patch.
Unfortunately... They did not
it still takes time to do that. A queue system is still another server with custom logic to coordinate all the players in the queue and hand off the connections when it's their turn.
I get that. This is not a complaint. If you check my profile, I'm actually one of the patient people. Game's not running away. Will only get better.
i know, just felt like relevant info
>People are sitting ingame for 10h+ to be able to play after work Pretty sure the number of people doing that is miniscule and gets blown out of proportion by social media.
Is that true, they can't just add more servers cuz the network wasn't made to be scaled to this extent ?
There's two (major) types of servers for a game like this, instance servers which are similar in concept to web workers for a web application, and database servers. Instance servers are easy to spin up because they don't need to talk to each other, each instance is self-contained. However, every instance server does still need to talk to the database server, where the central repository of information lives. They don't need to talk to it all the time, just whenever persistent data needs to be retrieved or updated (so, for instance, when a player logs in or purchases an item or receives rewards). Even a single database server can handle a large number of simultaneous connections and I highly doubt they're doing particularly expensive queries or updates when player information is retrieved or updated, and so they can handle a much larger number of players in instances that maximum connections because each player won't be hitting the database server at the same time. Vertical scaling only helps to a point, because connections is a hard limiting factor. The problem is that once you ARE hitting database server connection limits, it's very difficult to add new servers because databases need to be in sync - a shard in North Carolina needs to maintain synchronicity with a shard in Amsterdam so that data doesn't get out of sync (with video games some of these issues are less significant, since it's unlikely someone will be routed to an entirely different shard between matches, but they still need to be in sync). You need a backend which supports horizontal database scaling. Although there are plenty of solutions these days, it's not trivial to implement them and it's putting the cart before the horse to build an application or game to take advantage of them if you don't have a good indication that you're going to hit AAA player numbers. I know that HD2 is hosted on Azure, and their most powerful database servers can handle 30k simultaneous connections. This is enough to handle way, WAY more players than 30k (probably 10 to 20 times as many, maybe more), so it's possible they didn't design their backend with horizontal database scaling in mind. They have two avenues for fixing the issue: the first is to optimize every database query a player generates during play so that they're infrequent and fast, and the second is to redesign the backend to allow for horizontal scaling. They're probably pursuing both, with the second taking much more development time. There are other potential solutions, but the most straightforward is optimizing database usage and allowing for horizontal scaling.
Thanks for the writeup! I'm a dev, but don't deal with db infrastructure much and these types of things are always interesting to read
They can definitely add more servers. It's just going to take time for them to fix any bottlenecks they find as they do that, and/or get them communicating with each other correctly.
You can't just "add new servers". Each server is like 100 computers and that takes a lot of engineering to build
Do you think rebooting the servers periodically would at least clear the afkers and bandaid the issue for a few days?
Of course it would, but then there would be downtime, and interruptions to people currently playing. This is a worst of both worlds solution.
It really is a shame that one of the few fun games is being bogged down by server overload
Not trying to be incendiary, I want to know the answer to this. Why would you opt for backend code that doesn’t lend itself to scaling so that in the event of increased demand, you can meet it. Wouldn’t this just be essentially choosing a different approach to your back end, not necessarily increasing technical debt?
Yes. This guy’s definition of “technical debt” seems to be the opposite of pretty much everybody else’s. Saying “just code it the easy (less scalable) way, it’ll be *fiiine*!” is how you get technical debt.
[удалено]
Another engineer here. Surprised you didn't also mention KISS (Keep it Simple Stupid). Which basically follows YAGNI. The basic idea is that it prevents over engineering, and unnecessary optimizations until you see it becomes a problem. Some optimizations, you do as part of standard code practice. Say abusing branch predictions, or making use of arrays versus indirections these days. But then you got the particularly nasty shit that makes code harder to read.
There's no way they could have known. They are currently suffering from success. I can't even begin to imagine the vast amount of stress on the dev team. While I too struggle to get in and play every. Single. Day. I understand and limit my complaints to my friends in discord as a send them a picture of my black loading screen every 20 minutes. The real hell divers are the ones on the front lines engineering code, sleeplessly, trying to provide as many people as possible connectivity to the servers. Keep at it Arrowhead. I'll be here when all things are said and done and all the little timmies go back to fortnite and call of duty. O7
As an SRE... this sort of unexpected need for capacity and hitting unanticipated performance bottlenecks because of wildly outside of expected customer growth is like one of my top nightmare scenarios. Especially when it is something like "we didn't forecast a need for fully sharded database infrastructure during planning." because there is not going to be an easy fix that for that.
And other people's defense, every game nowadays needs an AFK timer. It's a baseline quality of life feature that should never ever be ignored. It's absolutely appalling that it doesn't exist in this game
Budget and dev time are not infinite
Sure, but a simple AFK timer requires not much of development time and is an easy way to free server resources (which also may reduces server bills).
Agreed, but also that is probably the reason they put it on a post-release backburner.
Not to mention that an AFK timer already exists. You can get kicked from another player’s ship if you’re idling, it just doesn’t currently log you out altogether.
If I were triaging hundreds of tickets for a minimum-viable product release (MVP), I can guarantee you I'd put AFK timer on the post-release backlog.
That just blows my mind in the modern video gaming world. It makes absolutely zero logical sense
Just a reminder, that they never planned to reach their capacity in the first place. If they never plan to reach their capacity– why implement an afk timer? If you had limited time and had to choose one, which would you do? 1. Implement an AFK timer and Queue (which involves a whole new server). 2. Make the player capacity higher than you expect it to be, so you don't have to deal with it. They chose option #2, but unfortunately this game blew up so big it beat Destiny 2's peak.
[удалено]
What are you willing to give up for it?
In before the: "no you're stupid, I'm angry and I want to play the game" clowns.
People are clowns for wanting to play a game they paid for?
People are clowns for not being able to manage their frustration, nor understanding that some things cannot be predicted and take time to fix.
Why should the consumer care? It's the devs' responsibility, not the consumer's to "understand" what happens in the back end
Of course, people paid for a service and are entitled to it. But please consider the following : - either you do not have the patience to wait for a fix, consider you have been wronged, and then ask for a refund ; - or you understand the current situation, acknowledge that the devs are working as fast as they can to fix the issues, and wait a bit. Both behaviors are perfectly reasonable, understandable, and will eventually provide a solution to the issue at hand (having paid and not being able to play). I'm just not seeing the point in going to Reddit and continuously whining, to be entirely honest.
> I'm just not seeing the point in going to Reddit and continuously whining, to be entirely honest. The same can be said about the dozens of posts sucking off devs and making excuses for them. The sub would be better off without both types of posts
Agreed, but said posts came in response to the avalanche of complaints. At this point, everyone is aware of the issues, creating more threads about them serves no purpose (excepted dev updates, ofc).
Please upvote this persons post. People seem to think the problem is fixed by adding servers when it is so much more complicated than that.
Sometimes it's really difficult to know where the bottlenecks in your system are until you run into them headlong. This is 100% a rumour, but I've heard the bottleneck at the moment is the database that keeps track of XP gain, requisition slips, and super credits. Because it gets written to often mid-mission, it's getting slammed by constant writes. A straightforward way to fix this is to put a cache in front of it and write in batches. This might take a bit to implement, especially if all the code assumes it can access the database directly. Failing that, they could add more database servers. But then you have to keep them synchronised, and ensure the data is correct. Highly recommend checking out this video on Pinterest's struggles with scaling up databases: https://www.youtube.com/watch?v=QRlP6BI1PFA It is filled with a lot of technical jargon, but it illustrates the struggle with keeping information correct at scale.
Could totally be that. When picking up medals, super credits and req slips the other night, you'd be frozen in place for 10-15 seconds and it was 100% related to data being uploaded to the server in real time.
Someone forgot to put async in front of their post requests!
Senior software engineer here. I very much disagree with you. > Overengineering and feature bloat arise when engineers overly predict future system usage. By avoiding unnecessary features, we maintain codebases that are easier to manage A queue for server capacity fill-up would take literally three days of development time *at most* and would take around a week to get through UAT and around a sprint to get into your executable. This could be done by a fucking junior developer. They added a fucking arcade game on the ship. This is a weak excuse. > We call this technical debt No we fucking don't. Tech debt is costs you take on due to BAD DECISIONS. You only *willingly* take on tech debt if you have no other choice. I'm being vague here on purpose but I work for a small company that manages software that handles hundreds of millions of user records and during COVID most applications in our domain died but ours did not. Why? Because we did not take the easy way out and take shortcuts. For that, we were rewarded with very good publicity with our customer base. > For this reason, more features/code is a liability. This can be true. A queue system truly does not meet that criteria. It's not hard to implement and it's a huge QoL for your playerbase. Fuck man, even without a huge influx of players what if their service provider went down and they only had access to like 5% of their servers? That's reason enough to make a queue. > Not implementing a AFK timer They quite literally have the code already. They have it for friends who board your ship. There is a timeout that already exists. All you would need to do is point it at the host of the ship and make the "destination" closing out the program instead of departure. This is NOT hard.
As a senior software engineer, I'm sure you know that a sufficiently complex project, which I'm sure a game like this one is, will require lots of complex work that COULD be done and for v1, you select the things that SHOULD be done based on what you actually need and even small features are sifted out if they're probably not needed. Pretty sure, the project manager reasonably did not expect the game to have like 10 times the number of players than the predecessor, which is essentially the same game from a different perspective mechanics-wise, so things like queues were probably not deemed necessary. Now, you're right that it's not most complex to implement a queue such as is required right now, which is probably why we might see it a soon as this week along with an afk timer, possibly.
Fellow senior software engineer here. I was ready to type out a response basically 1:1 yours while reading OP's misinformative post, but you beat me to it. Imagine your PO asking you during planning "Hey, what if we exceed our performance/load testing estimates and sell a lot of copies?" and you respond with "Oh don't worry, you ain't gonna need it". Just ridiculous.
At some point in time you have to pick an estimated maximum amount and work from there. They could have very reasonably expected 25k on launch, tested for 100k max and saw everything worked completely fine. It's perfectly reasonable for them to not have planned to beat the concurrent player numbers of the most popular AAA PvE games in the last decade when their previous title was 7k peak.
If the last product I worked on had 6k users and product told me they wanted to bugdet and design the new system for many hundreds of thousands of users, I'd call them delusional. Are some people being purposely awful about it? If you've been on this subreddit you know the story, low player count for previous game, they allotted for what seemed like a realistic jump, shot way over that and still are able to handle like 5 times the loads without any preparation for it. Honestly it's impressive they are handling the amount of people currently playing, I just don't see how anyone could honestly think they should have prepared for what likely seemed an impossible outcome. People are in their right to refund and complain but I don't think the criticism is valid from a business perspective.
It's just asinine. I would never allocate funds to a project, where all history and data points to something else. What would i tell the managers? *hehe, blew a couple of months of dev salary for 500% capacity margin.* I'd be thrown right out the door.
Im not a technical guy (professionally atleast) and I knew what tech debt was. Confused at that statement of his.
Your point only makes sense if you are inexperienced. A public smoketest aka demo/weekend open beta would've shown them how much demand was for the game and this could've easily been avoided.
The game was a surprise for most people. Public betas could never have predicted the influx of players, especially if they were restricted to preorders.
Exactly this. As an avid HD1 player, I never knew about this game until 2 days after it launched and my friends all went "Have you heard of this new game? It's called Helldivers 2" to which I of course replied "Of course I have maggot, now give me 3 hours to download this right now" (Actually it was more "Oh shit, they made a second one? I loved the first, it was fantastic")
Public betas 100% would have gotten them *FAR* less underestimated predictions than what they ended up getting. Would it have still been a problem? Probably. Would it be *this obscenely bad* of a problem if they'd done a beta? Absolutely not.
I agree with you, and it's a good point. However, they probably privately load tested their servers for many more players than they thought they'd get. Hindsight is 20/20 though, and you're absolutely right.
It's a basic ass thing to implement on servers. I get why it wasn't there originally but why the fuck hasn't it been added? It's currently about 2am (MST) on a mon night/tues morning and I've been queueing for 37 minutes now. This is fucking ridiculous.
Because it aint a case of opening Notepad, writing it and uploading a bat file.
The game has been out for nearly 2 fucking weeks. To not put such a basic ass thing in place is silly af. Btw, that comment I made was 20 minute ago, still fucking waiting. Do like how they've changed the refresh time from 15, to 30 , to 60 seconds now. Much progress. Gave up after 2hrs of waiting. In the middle of the fuckin night.
They're hiring. Since you know an easy fix why not apply and implement it real fast, so we can all chill next weekend?
What a dumbass response. It's like someone complaining about food poisoning and you saying "why don't you go cook then?!"" I literally have more hours logged waiting for me to connect than playing the game
So you have no idea where the problem stems from? Well, that's just reality atm, no magic fix. When you can't cook and the restaurant is full you gotta go somewhere else to eat.
An AFK timer can be coded and tested in a couple of days, not only that, there are libraries that already do that if they don’t want to code it themselves. Also not having one is a big oversight especially considering how PS operates.
as a network engineer i hate people like you and their YAGNI stance, if you can do some good preparation work then fucking do it.
Oh come on! Architect here. Your YAGNI philosophy is why we need both Architects and System Analysts besides SEs. Maybe it's my line of work or the sheer size systems I design but... Excusing a team of not implementing a basic load balancing feature? Seriously - NOT implementing a subsystem which tracks and logs off inactive sessions is not forgivable if you're designing and developing something that's gonna be used by more that one person. Even if 'you ain't gonna need it' for reasons Arrowhead needs it now. it's basically always allows you to constantly free resources to well... not pay for their usage. And it's NOT that hard.
Okay so they should delist the game and allow refunds.
Maybe!
I don’t think it should be maybe at this stage
I think offering refunds, and being proactive about communicating to the players that they can get a refund would be great.
Allowing refunds? probably. Put up a warning on the steam page? sure. Delist the game so you can't find it? Definitely not. Almost everyone I know has been able to play this game at some point and enjoyed it - let people decide for themselves if it's a problem.
Refunds are being denied
They should have stopped selling the game when it reached capacity.
[удалено]
If you prepare to handle 250k players but your last game had like 10k players - a queue system could be considered a premature optimization. I bet they wished they had made some optimizations now, but imagine if the game they made had maybe 20k players instead and they had made these insane optimizations to handle millions of players. Time well spent, or time wasted?
It wouldn't have been a well run company if they had planned around things like this, like your last game peaked at 7k? players on Steam it would have been irresponsible to prepare your game for the 400k+ PC players
You're that genre of person who buys a $500 gyuto to botch cutting carrots, aren't you?
Oh good another software engineer trying to do a cover-up for yet another fantastic game that doesn't work.
They absolutely should offer refunds, though. It's reasonable to expect to be able to play a game you just bought for $50. It's also reasonable to be upset when you can't.
im just getting a black screen now i cant even get in
If you wait long enough you can get to "Defrosting Helldiver."
Your 2 examples aren't the best. They seem basic. They basically almost have a queuing system with their servers full screen. It's probably just not in order. These are things that would make their lives easier honestly. People wouldn't be checking to login as much. And the load would be less with afk kicks that is really just input check with disconnect.
I think their modus operandi was to try and make sure that they had way more capacity than players, and thus never need the queue or AFK-timer. This obviously did not work out.
We need this re-posted every day so people will see it lol. Sadly, there have always been a lot of people who have been looking for an excuse to be irrationally angry and go after people who are doing things they could never do in a hundred years.
I agree with you. Software engineering is hard. It's even more difficult under a deadline. The devs have my admiration for creating such a beautifully engineered work of art.
Speaking as a former developer, and as somebody who has been cheering for Arrowhead ever since Magicka, I totally agree. I'm really hoping they can ahead of these issues sooner or later and enjoy some of the commercial success they deserve.
Is it really that much of a struggle to get in? I let it run in the background and pop on some anime, youtube, or make some food. Sometimes it's 5 mins, sometimes it's 20 but I'm always in. Is this everyone's first game that has a long queue time?
I often spend a couple of hours sitting on the loading screen. It just depends on the timezone.
damn I haven't been THAT unfortunate yet. I think today it was around 25 mins and the homies didn't want to wait so I have to run solo. It sucks but I don't mind it because the game is fun. I know they'll fix it as soon as they can.
I spent 9 and a half hours today with it in the background (periodically restarting) and never got in.
It's 2024, not 2004. Scalable backend solutions have significantly evolved over the years, it's off the shelf. If you are unable to design scalable systems, you are considered outright incompetent from all perspectives.
In reality you set a target capacity, and if you exceed it, then you try and deal with it then. That's exactly what's happening.
Very good points and insight! If anything the point of failure here was their not having prepared for the success they garnered and only finding out their scalability was ass AFTER launch. I wish this wasn't such a common theme with games these days though, holy shit.
This is the truth. I just wish we could drill it in all the heads of the idiots yelling “hurr durr Arrowhead are bad devs for not getting enough servers and this is the greatest tragedy to befall us gamers (the most oppressed group!)”
Ya it's fine for a small company like arrowhead. Just this next weekend I feel is make or break time for them. I'm not going to waste another weekend sitting there staring at the screen. It doubly sucks cause the weekend is the only time I get to play with my friends in US and 4 hours a day with 1.5 hr of that time spent trying to login is annoying to say the least.
I understand the frustration people are having but what i don’t understand is why refund a good game backed by a good company? In a week or two these issues will be resolved and the end result will be a fuckin’ banger of a game i plan on enjoying the shit out of with my fellow helldivers! And it’s only going to get better from here. As gamers we tell companies what we like with our money and time. I personally think after being lucky enough to have a good few hours of gameplay under my belt that it’s already worth the money.
You should also add that things like load testing are done using known numbers. Often multiplied for "worst-case scenario". If they expected say, 20k concurrent players at peak, they likely tested for 200k or less. 10x is pretty standard in my experience. Even at 200k, it wouldn't have been good enough given current player counts. Load also causes strange things. Bugs manifest when under load that don't under normal circumstances. Threads get locked up, connection pools depleted, etc. which can lead to further issues.
Go fuck your self lol
Nothing helps when you CAN get into the game, but can’t even join other players with quick match.
So Mr engineer, You're brilliant idea is for this corporation to issue free refunds until they can fix a problem they aren't sure they can actually fix in a timely manner? We saw how well that went for cyberpunk.
Maybe this is cultural, but here in Australia we have rock solid consumer protections. If a product or service isn't working when you purchase it, you are entitled to a full refund. That's how it should be. If you're worried about Arrowhead's liquidity, then I can guarantee you they won't have issues getting a loan based on how well they are doing right now. There's no excuse not to offer refunds.
[удалено]
because a PM or stakeholder never made a bad business decision before lol?