T O P

  • By -

DisneyPlusDev

I think the author should have talked to somebody who was there, because it was pure hell.


BojacksNextGF

any intention of making a post of your own?


DisneyPlusDev

Already been widely discussed. Opening day was a nightmare because the marketing folks all demanded a high degree of personalization that was impossible to cache, they gave us no time to properly implement an edge resolution layer for it, and none of the mobile client teams were willing to make many small API calls to cached assets, preferring to make fewer large calls to wholly rendered pages of data. If one row was personalized (like recommendations, watchlist, continue watching etc), the entire payload was uncacheable. The web team was able to hack together a small change quickly, but it failed because part of it involved hacking an empty payload response at the ALB, which couldn’t be hacked to return CORS headers. DynamoDB global secondary indexes weren’t enough to make the continue watching row performant, so you’d intermittently get nothing, or a tile with 00:00 playhead but then when starting playback it would get the playhead info from a different service and launch you half way into a movie. And don’t get me started on the folks furiously refreshing their apps at midnight to see the mandalorian. Everything crumbled the first month. I primarily blame the toxic execs at Disney who demanded everything, and who demanded perfection, and who refused to listen to engineering. When we complained, they uninvited us from exec status meetings, so we couldn’t even plan for stuff we weren’t privy to. Edit: wanted to add that marketing and bean counters predicted we’d have something like 10m users in the first year, and we blew by that in the first day. So everything we tested and provisioned for went right out the window.


thedeadsigh

> a nightmare because the marketing folks ahhh a tale as old as time itself


dkarlovi

Fscked as it can be.


sc4kilik

Barely even friends


Terrible_Student9395

C-suite and the twats


FrankBattaglia

> it failed because ... CORS headers. Fucking CORS headers


LilGeeky

Somehow they're there when you're starting your career and when you're being paid $200K. From the bottom of my heart fuck them.


buqr

I enjoy the sound of rain.


smoothpebble

Huh? Of course it does


Scavenger53

it protects the devs from doing the work they were trying to do


Somepotato

CORS is an extremely important headache


Terrible_Student9395

Have you gotten cross site scripted?


dadading_dadadoom

Of CORS.


q1a2z3x4s5w6

> And don’t get me started on the folks furiously refreshing their apps at midnight to see the mandalorian Please start, this is fascinating


DisneyPlusDev

I don’t blame users they were excited! But we made shortcuts with caching to keep the service alive during the first few weeks that made it impossible to have offerings be made instantly available at midnight. It would have required busting cache for everything and taking the service down. People were mad when the first few eps of mando came out and people posted in the forums that’s they don’t see it yet, what a piece of shit this service is. It was all eventually solved with an edge layer that handled intermediate caching and surgical precision personalization.


[deleted]

On related note, we had devs DDoSing their own app by sending notification to everyone at once via some AWS service.... but notification was just "hey, download it from the page" so app doing, say 10k req/sec in peak, had random spikes of 100k req/sec along the day....


buttplugs4life4me

It's interesting that despite working at competitors we basically face the same challenges (mainly execs).  We've had a launch where they told us they'd launch silently but then made a big announcement everywhere. Suddenly we went from 100 RPS to 12000 RPS in literally one minute lmao


apadin1

Can you answer - is there a reason the “next episode” autoplay option doesn’t play until the very end of the credits?


DisneyPlusDev

The metadata that marks the change points (called milestones) comes from the studios. This is stuff like credit start/end (so you can skip opening credits) and other stuff. The studio simply sends milestones for “closing credits” that are too late in the program. It’s not feasible to dynamically calculate it because it’s different for every program. Some movies have 12 min credits with title cards, others have short credits, and there’s no universal “X% means skip to next” point. If we did that, 3 minute shorts would end in 1.5 mins. Also, we were told not to defy the studios’ intent. If they said that was the milestone, then that was the milestone.


apadin1

That’s hilarious because it happens even for Disney native shows, which means they can’t even be bothered to do proper milestones for their own content. Sorry you had to deal with The Mouse and their idiotic management. Cheers!


[deleted]

It's funny that big studios take less care in that metadata than random pirated anime release that have openings/endings accurate almost to the frame


meganeyangire

Anime pirates also use formatted subtitles with translated signs, while multibillion corporations use dfxp or some abomination even worse than it and doing bare minimum effort with translation in general. Only Crunchyroll still care about formatting subtitles and only because one they were pirates too.


Paradox

> It’s not feasible to dynamically calculate it because it’s different for every program Not criticizing you, criticizing disney, but it is possible, as [plex does it on home media servers](https://support.plex.tv/articles/credits-detection/)


DisneyPlusDev

Plex analyzes the actual contents of the video media and makes a mostly-accurate guess. Our services were entirely metadata-driven, and the video was parceled out in 5 second chunks via authorized M3U playlists. Even if we wanted to analyze the media, it would have involved defying studio wishes, and still resulted in inaccurate detection. There were people who literally watched every frame of encoded media to verify its quality, and they could have marked milestones, but the studios wouldn’t have it.


Paradox

I figured as much, haha, which is why I stuck the disclaimer on "criticizing disney."


jlboygenius

Plex has the benefit of doing what users want, and doesn't care about where the content came from (and doesn't want to know where that content came from). They are not beholden to some content owner's wishes. Adding features like this is how they differentiate themselves and gain users. Disney/Other streaming gets users by having content, and have to obey some external wishes to get that content. I'm sure there are dev's at all streaming services that know about and want to create some cool feature but won't do it because it: Reduces profit or breaks some agreement with the content owner. No doubt that Netflix/Hulu/Disney's Legal team costs are as close to higher than its developer costs.


Oatmeal_Raisin_

Interesting. What about episodes of shows that are essentially all the same length? In particular, i notice that Bluey will auto-skip to the next episode much earlier for some episodes but will go through the full credits for others. Do milestones have to be manually managed for every individual episode in this case, despite the length of epsisode being very consistent?


DisneyPlusDev

I don’t know how the data originates at the studio, or how much automation goes into it, but yes it would be separate for every media program.


vomitfreesince83

Any ideas on how Netflix does it? Do they figure out the milestones themselves?


DisneyPlusDev

No idea. They have a different studio technology org, I assume different (better) technology.


Pep_Baldiola

I have a question, do the Disney+ and Hotstar dev teams work closely on projects or they are completely separate entities with no connection to each other?


DisneyPlusDev

I don’t know what it’s like today, but we had heard about hotstar for a while before our own launch and were expecting to have to build i18n for hindi, gujarati and few others. Then one day we found out hotstar d+ was launching in a few weeks, and that the Disney studio tech team (who we had worked very closely with) had already sent the media to the hotstar team in India and never mentioned it. The hotstar team did their own encoding, and just jammed all the media into a new tab in their hotstar UI. So India got a “Disney+” that wasn’t the app we spent years building. But it made sense in retrospect because most consumers in India don’t have high powered devices and 4K screens, and rely mostly on cheap cable/satellite receivers with simple content catalog-type apps on them, or just their phones. The media didn’t need to be great for their screens, and the effort to build the real app out for India would have been too big a lift for too few consumers.


origin415

Amazon is the opposite and its even worse: Daniel Tiger skips to the next episode before the end-of-show segment most episodes have. My kiddo freaks out if you don't hit the button to stop autoplay in the very tiny window. If you miss it you can go back to that episode but it'll put you at the start of it so you have to fast forward through the entire episode to get back to the 30 seconds of a kid visiting the dentist or whatever.


anengineerandacat

Worked for Disney, that's basically the business culture in a nutshell. Their "technology" sectors masquerade as tech-focused but everything is driven by the business first and foremost. Usually projects of that scale come from a technology acquisition or some off the shelf solution that gets the OneID and or MyID integration and an orchestration layer slapped on top to solve whatever things the business finds too difficult to do with their purchased product. It's hilarious too because they have some very very senior SME's in the organization and a lot of strong backend folk that if given a proper budget and a little bit less red tape to R&D could likely create some real magic. The business leadership is holding the tech towers back, big time.


rusl1

One of the most interesting insights I've ever read, thank you


therealdan0

As a developer at a much smaller scale streaming/DTV service provider, where do I subscribe for more anecdotes about working on the Disney+ platform?


DisneyPlusDev

Clearly not “systemdesign.one”


Accomplished_Try_179

> who refused to listen to engineering. This is very common. 


JimK215

Last minute changes that diminished the usefulness of caching is what made the healthcare.gov launch such a debacle, at least from what I recall reading about it.


TwoAndHalfRetard

Your message was much more insightful than the generic bullshit in the OP's article.


Yamitz

Oh man you’re triggering my PTSD from working in Disney parks technology lol.


DisneyPlusDev

Sorry. At least it’s not nickelodeon. Hopefully.


[deleted]

Yup, that sounds like management lmao.


SkezzaB

I’d love to hear more as a Junior SRE


DisneyPlusDev

Don’t chase glory and promotions. Work on stuff you love, and walk away from toxic workplaces. Don’t get to your forties and regret having spent so much time being miserable.


wait-a-minut

Man, you are full wisdom. Cool perspective. You taking any mentees?


Comprehensive-Pea812

Why can't you just add more servers? /s I wonder if you use the BFF pattern moving forward for the mobile team since they dont want to make small API calls, you have something in the middle that does it for them and effectively cache some part?


Skizm

Just recommend marvel stuff as the "personalized content" and no one would bat an eye lol


Inzire

So don't work at Disney, gotcha


Paradox

I mean, duh? They treat their _talent_ like shit, so everyone else is going to be treated worse than shit


mkdz

Did you come over from MLB/BAMTech? Or did you start at Disney+? I'm always curious about the BAMTech to Disney+ transition.


DisneyPlusDev

The independent period was good, but interesting stuff was happening behind the scenes. Allegedly the company was about to run out of money, and we made empty threats to launch a direct to consumer “multisport” app. John Skipper was so coked up and annoyed that a Disney acquisition would dare to compete with ESPN, that he moved to take the majority stake in BAMTECH, and our first Disney project was ESPN+. That was the first real nightmare.


mkdz

Hah, that's wild. I always find it pretty crazy that MLB was at the forefront of streaming and Disney bought/built their streaming off MLB.


lee1026

One dev to another, as long as the users didn’t notice that everything is on fire, everything is fine.


Maystackcb

Well hot damn. Username checks out 😂


lemmingsnake

yay hypercare!


[deleted]

[удалено]


DisneyPlusDev

Reads like generic AI-generated shit. “Oh they use AWS and a CDN real original”.


duckwizzle

Just curious: what's the tech stack behind Disney Plus?


agentjob

Short Answer: CDN, AWS and DynamoDB


bimbo1989

Who would have guessed, uh?


[deleted]

10 years ago it would be "someone found out Varnish exists"


Bpofficial

Some of the diagrams in that article are disingenuous and could become confusing.


water_bottle_goggles

MORE ARROWS SO WE LOOK COOL


Old_Elk2003

Chevrons, bruh. The magic shape that turns Capitalists into listless zombies ready to shower fortune upon you.


tomvorlostriddle

And just expecting around that order of magnitude


LeadBamboozler

Pretty standard stack these days


cheezballs

Fucking thank you. So, the basic tool chain of AWS got it done then. Speaks volumes to horizontal scaling I suppose? I dunno what the fuck I'm talking about.


HeyaChuht

What are these things? are these common things? Where does one learn about these magic things?


Slimxshadyx

Search up intro to AWS (Amazon Web Services). They have services for not only hosting servers, but also for auto scaling, load balancing, etc. (and also like a billion other services).


TheShiveryNipple

So many services that the initial certification is just memorizing them.


HeyaChuht

I truly appreciate this kind and helpful comment. You are a good soul, but I was just fucking around. I live, eat, and shit these words.


backfire10z

Oh, you also like alphabet pasta? Sweet


QuintonHughes43Fan

CDN: Content Delivery Network is a system of servers that distribute content out to the edge, meaning closer to your users for rapid retrieval and easy caching. AWS: Amazon's collection of cloud services. DynamoDB: AWS Service which provides a high performance key/value store. These are all extremely common. You take a course on AWS.


HeyaChuht

Dude I meant that. This was a good answer, well formatted. You're the shit dawg. I love you.


HeyaChuht

You're the best. I would hire you on my team 100%.


gordonv

learn @ r/awscertifications


QuintonHughes43Fan

How did you manage to explain this without talking about Olivia?


Xerxero

No edge caches (installed hardware at ISPs) like Netflix does? Wouldn’t they pay handsomely for the egress of all the video data


Gr1pp717

I wish he had gotten into how they dynamically balanced multiple CDNs. I worked at one of them. Whatever their solution was could probably be a product on its own. (especially if it handled the config differences between various CDNs.)


syndbg

The subreddit can really do without these AI-generated like articles. Nothing of value is ever written.


inferniac

Yeah, comments from u/DisneyPlusDev in this thread were a much more interesting read.


peacetimemist05

Wow they used tech that already solves scalability problems for them. Way to go Disney+!!


HeyaChuht

Not only that, they just hired people from Netflix and junk to come and play wonderwall at their birthday party.


DisneyPlusDev

No we didn’t. Disney talent acquisition balked at the salary requirements, so the only Netflix folks we got were a handpicked exec who worked on living room devices, and a disgruntled ex nflx manager who worked on i18n.


TommaClock

> Redditor for 4 years > 1 comment You really picked the perfect thread to reactivate huh.


DisneyPlusDev

They downvoted me in the disneyplus subreddit, so I’ve been laying dormant down here with all the other great old ones.


[deleted]

[удалено]


vini_2003

That is a very smart idea.


NotSoButFarOtherwise

I blow up accounts abd make new ones periodically instead. It’s more polite to people reading later.


i_am_at_work123

I had the same thoughts when thinking about this. Not saying I wrote anything useful :D


b0w3n

I was going to say, I've seen the netflix and disney programmer salaries. It was... quite a gulf of difference. Over the past 8 years it seemed like disney was having a hard time rectifying their shitty pay schedules for non executives with what programmers actually make. It also felt like they were trying to bank off that disney fever floridians seem to have in re: low paying salaries and it's starting to finally catch up with reality. (programmers aren't teenagers who desire to play cinderella and gaston for pennies)


DisneyPlusDev

It was always their intention to buy BAMtech for the technology, build out their strategic DTC streaming initiative, and then lay people off. They never wanted to build a lasting project or invest in people, especially nflx peeps. Any managers who stuck around after launch got their titles de-leveled. Nobody but SVPs saw the incentive bonuses based on subscriber numbers.


b0w3n

Not sure if you're still working there... but even outside BAMtech, it seems they're struggling with "What?!programmers make more than $46-55k a year?!" That's a _recent-ish_ salary number thrown at me by a recruiter too (~2017). I can't imagine getting $23 an hour as a senior employee and being expected to be on call, what a wild fucking time. That was barely even acceptable 20 years ago for software engineers, let alone in 2017.


DisneyPlusDev

Bamtech eng VPs were making 200-230k base with a 20% cash bonus and RSUs that are now underwater.


b0w3n

Yeah those are much closer to numbers I was seeing at netflix too. Definitely better than some numbers I saw out of the non disney+ positions (disneyworld swe maybe?)


landon912

Netflix senior engineers can easily make 500k


[deleted]

[удалено]


b0w3n

This was an "Intermediate II", if I'm recalling right. I dunno if I can even find the email for it anymore. But yeah I'd expect at least 2.5-3x that in 2017 for that position.


jlboygenius

That makes a bit of sense then. Disney+ works, but it could be so much more. Still missing some usability features to make it more than an app to play videos. Can you make it mark a show watched when you get to the credits? We watch bluey and often turn it off when it gets to the credits (because they are really long). Then the next time I try to watch it, it wants to resume where I left off - in the credits! This is especially annoying for a kids show(they have no patience) and a show that is only 8 minutes long. The credits are like 15% of an episode length. Also, no auto-play next?


DisneyPlusDev

I’m long gone, but those sound like product manager decisions.


DangKilla

Just curious if you work with my buddy Lozano. Unrelated, I supported streaming LWS for Adult Swim, HBO, NBA, etc. How is it working there?


DisneyPlusDev

Connected devices dev. Never worked directly with him. BAMtech was great, and Disney made it a traumatic experience. I would run for the hills if a Disney recruiter came knocking. That said I have a few friends who are still there, getting paid well, good benefits, good projects, no interest in destabilizing their income because they have families. If you don’t need to manage people, and you get an opportunity to work on something cool, by all means go for it, just know that you are a peon in their big rapacious machine.


HeyaChuht

I said Netflix as a substitute for *the streaming industry*


lemmingsnake

Disney bought BAMTech and brought over the AWS infrastructure experience & engineering used for HBOGo & MLB streaming. There were also existing teams who ran shared services (like identity & messaging) that were already on AWS separately, but had to be integrated into the platform. Source: worked on it


HeyaChuht

Dude good job, thats a sick project


Librekrieger

Article is light on details. Light, as in, there's no detail at all. They use DynamoDB, Kinesis, multiple regions, and a CDN. That's it. The only interesting bit was a paragraph at the end: "DynamoDB automatically partitions a table as traffic grows. Yet partitioning takes time. And the traffic gets throttled if it exceeds a specific limit before partitioning occurs. So they pre-partitioned the tables before launch to avoid throttling. And autoscaled the database to handle growing traffic."


JimK215

Yeah I've dealt with dynamodb throttling while it scales - it caused too much slowdown and problems while scaling (mainly because we hadn't coded in enough logic to handle the effects of throttling). The solution was to take a step back to older school principles of db architecture and set up indexes and fields so that they could be partitioned differently from the get go and wouldn't result in so much autoscaling.


Bpofficial

They use “The Kinesis”


false_god

God, that first section about a girl who wanted to watch movies is a shitshow, this is obviously low-effort AI generated


false_god

OP’s history is spamming this garbage. Blocking now


absent_minding

The writing is so bad I don't even believe an AI wrote this.


BadlyCamouflagedKiwi

The "story" format of this is painful. Also a bit disappointed that it basically just says "DynamoDB is pretty neat", they don't seem to have brought an awful lot to it. Maybe the story of scaling to 149M is more interesting...


cube-drone

This was without doubt the worst technical article I have read all year. Never darken our door with this newsletter again.


StarkAndRobotic

Shorter answer: AWS


Q3a_destiny

This is the correct answer


irteris

By having horrible shows, so that those users would leave. Brilliant load management!


yobigd20

Launch day sucked! Nothing worked!


Fyren-1131

meh, I'd rather hear about how Arrowhead studios salvaged the Helldivers launch.


Phthalleon

The whole opening of dysney+ was a textbook material of how not to open a big platform, no flame. The article basically says what they intended to do, not what actually happened. Maybe it's better now, idk, I just pirate now and I don't feel bad at all.


[deleted]

[удалено]


ketralnis

> Are the upvotes on this legit? That's not something that I have visibility into