Apple’s painfully unstable live stream was caused by bad JavaScript

By

appletvtruckschedule
The new landscape view of the iPhone 6 Plus. Photo: Chris Roman

Apple Watch might have stolen the spotlight at the Flint Center yesterday, but for many fans, tuning into the first 30 minutes was as impossible as scratching sapphire crystal glass.

Fans trying to watch the event via Apple TV and Safari were greeted by colorful bars and an Apple TV truck schedule. Even if you could get connected to the live stream, understanding Tim Cook and Phil Schiller was nearly impossible thanks to a Chinese translation track play loudly over the main feed.

It was an absolute disaster for the first 30 minutes from a company that executes marketing events with zen-like precision, but according to Dan Rayburn at Streaming Media, a few simple errors is all it took to bring down the show. After digging into the meta data and code from Apple’s site, Rayburn found the root of the problem: A couple lines of JavaScript.

Apple decided to add some JSON (JavaScript Object Notation) code to the apple.com page which added an interactive element on the bottom showing tweets about the event. As a result, this was causing the page to make refresh calls every few milliseconds. By Apple making the decision to add the JSON code, it made the apple.com website un-cachable…which causes a huge impact on the performance when it comes to loading the page and the stream”

Because of all the refresh calls from the JSON code, it looks like it artificially forced the player to degrade the quality of the video, dropping it down to a lower bitrate, because it thought there were more requests for the stream than there was.”

Apple only relied on Akamai’s content delivery network for the video stream, but their servers couldn’t cache the page because of the JSON code. Any performance problems on the page also impacted the video because Apple embeded it directly onto the web page.

Ok, but what about the annoying Chinese translator yapping over Tim and Phil for the first 27 minutes? That’s all on Apple. An employee must not have set up the encoder right, and the primary and backup streams were out of sync (that could also explain the dual songs before the event).

Rayburn says he believes the encoders were rebooted after the event started, which is why Apple TV and iOS users received “could not load movie” and “you don’t have permission to access” error messages when trying to access the feed.

Source: Streaming Media

Deals of the Day

  • Tom Nikolic

    Oh man it really was painful to watch the first 30 mins. I became so unenthusiastic and sad but then I saw the watch and I was happy again

  • Demi Buraga

    who cares? at the level Apple it is, i think errors and next excuses/explanations are unacceptable! Because of those “errors” the live stream was a FIASCO!

    • It was a live stream. That Apple provided for free. We all went into this knowing the massive amount of excitement and interest it was generating. Was it annoying, yes. Was it life stopping? No. I remember having to watch live updates in a web page for previous keynotes. Web pages you had to manually refresh. Given the sheer size and scope of this live stream, who cares if the stream was a bit flakey for a bit. Even Apple provided a live blog of the event!

      • mike077

        I don’t quite agree. We tech consumers have got to stop getting this backwards. Apple wasn’t doing anyone a favor by providing a free stream. It was a key part of their marketing strategy. The viewers were doing Apple the favor by watching it. By screwing up this event, for which many customers set aside part of their valuable day, they did a disservice to their most loyal customers. IMHO.

    • Adrayven

      yes, so whens the hanging? do you have a spare white hood? I forgot mine, and don’t forget the rope!

      err.. wait.. it was just a show and the show was recorded.. ahhh yea.. over -drama much?

    • Tallest Skil

      Grow up. It was a problem, they fixed it, it will never happen again, and the people responsible are certainly fired. You were not harmed in any way by 30 minutes of stuttering video and east Asian language.

    • aardman

      I was traumatized and permanently scarred by the live stream glitch. Were you damaged the same way? I feel PTSD coming on.

    • mike077

      There was no excuse for it. They had MONTHS to prepare and it was the biggest part of the biggest Apple event in years. They should have dress rehearsed it several times and rechecked the feed prior to beginning the program.

      You are right. Steve would have first verbally fried everyone associated with it, and THEN fired their asses — as they should be.

      In the end, reputation is vital and this preventable, foreseeable snafu does not bode well for people entrusting themselves to Apple’s competence.

      If this is taken as a subtle harbinger of things to come at Apple, then their reputation for reliability took a big hit. Not good.

  • ericbrady

    It was so frustrating trying to watch! (no pun intended)

    • Superalias

      No pun detected.

  • Mark Edwards

    I just gave up in the end and relied on tech blogs during and after the show. Technologically I think some of the new hardware looks reasonably interesting, and the Apple Payments system could be what finally takes NFC payments into the real mainstream, but generally I’m left feeling relatively uninspired by this years crop of stuff.

    • fastasleep

      Good thing nobody cares.

    • Tallest Skil

      You shouldn’t have given up in the end. Past the first 30 minutes it worked flawlessly.

  • I’ve worked in IPTV distribution my self for a few years… the encoder issues happen. What the company providing the feed should have done is to have a 1:1 backup of each encoder. Not much you can do now… learn the lessons… improve and go on. It’s no use crying over spilled milk.

  • duh

    Steve would have destroyed the streaming team with nothing but a glance and his thoughts…

  • Vatsaev Aslan

    Nobody gives a shit why it happened, just makes sure it never happens again.

    It was a total disaster, not that Apple streaming was always good (it was shit) but this time they took it on a whole new level of incompetence.

    • rattyuk

      Here’s the thing. Every single day this September, sometimes up to 3 times a day, Apple is doing a live stream from London’s Roundhouse featuring a line up of artists from all walks of music.

      It works flawlessly. Apple does know what it’s doing here.

      The problem here was that it looked like they had idiots on the sound desk and something else probably an unprecedented demand on the stream.

      However JSON had absolutely nothing to do with it.

  • It was a colossal screw up, but with the intricacies of web development, a single character out of place can destroy a website, it’s a ridiculously simple mistake to make

  • Sure, blame it on Javascript. First of, the contents of the page with Javascript enabled has no effect on it being able to cache or not. Stuff gets done on the clientside, but the basic page can be cached normally.
    Secondly. You can just as easy send the same stuff to thousands of clients. Thing is: you have to prepare for the amount of traffic it requires. And yes, it takes a bigger cut when clients refresh the whole page as the whole list of images and content you submit with XHR is going to be loaded.

    This isn’t a Javascript issue (hell, the page was working fine, but i just couldn’t get the stream going at all), its the bandwidth and performance of the servers that was lacking and not properly setup for the task. Sure, the site itself could be a bit heavier and the clientside stuff can take up more performance, but thats not what this is about. Apple-streams in the past have proven that its the lack of available bandwidth, not the website itself. And if you are going to add stuff that make it worse, well then its just going to be a disaster.

    They should’ve used Youtube or Twitch to give people a second source for watching the broadcast. Now you were down to a badly coded website which in turn made you focus on getting it going, not being able to focus on the content that was presented (which to be honest wasn’t much as well)

    • Kr00

      You clearly don’t understand how JSON code for tweets over javascript, embedded in a web page works do you? Because if you did, you wouldn’t make such an uneducated guess on what you think you know. Gather some actual information before making such dumb comments next time.

      • mjd14

        I would say that anyone who says ‘JSON code’ doesn’t know what they’re talking about. JSON isn’t code it’s a data format.

        The explanation offered in this article makes no sense at all.

      • Yeah exactly. Its not JSON, its Javascript that handles the JSON and there are plenty of good frameworks out there that show that this solution of Apple could be much better.

      • VaxMask

        Auch, that gotta hurt once you figure out what JSON actually is.

      • Kr00

        Well thanks for the google search translation of JSON morons, but if it is to be embedded via javascript or HTML, it has to be written, i.e., coded into HTML, javascript, etc. Isn’t that how you embed any content into a webpage? Thats how I’ve been doing it for 15 years.

        So while you idiots have had your childish fun, the original poster gets off making totally ridiculous comments, yet you chose to ignore his total lack of knowledge. Clap clap clap. Well done. Next time I’ll make sure I link to your google searches, just to keep you morons happy. Ok with you?

      • VaxMask

        Then it’s not the JSON that causes the problem, is it? It’s either the HTML produced by the JavaScript which generated content from the JSON, or the JavaScript code itself. “JSON code”? JSON is merely a way to format data, just like XML or CSV. You don’t put logic into JSON, JSON holds data. Unless the JSON data for some reason were represented incorrectly as invalid JSON; but then again, it’s probably a PEBCAK error when the API was made.

        If you’ve done this for 15 years, give yourself a pat on the back, will you?

      • Like mdj14 says: JSON is a format, not a programming language.

        I’m currently working on a few AngularJS projects and they handle these kinds of JSON requests pretty easily and they shouldn’t be much of a hassle.

        Sure, refreshing the page is not handy but thats not what happend. At all. It just requested new data and changed the DOM. This might seem as a refresh, but is just updating certain data in the DOM.

        The reason the site went down is because too many people tried to watch it. That the Javascript also had an issue getting its data via JSON is not the main problem.

  • Barnaby Jones

    The stream also didn’t show in Chrome on my mac, only in Safari. Going fullscreen enlarged the browser but the video was cut off, using less screen space instead of more.
    When the tweets/blog underneath the video updated, the video itself started stuttering or jumping back and forth. The buttons ‘resume from paused’ or ‘play live’ didn’t work at all.

    Terrible user experience, unworthy of Apple.

    • MagicBoyUK

      Apple streams have only ever worked in Safari and Quicktime in the days before HTML5.

      Chrome is a pile of crap on OS X. Badly optimised code that flattens your battery twice as quick as Safari. Don’t even get me started on the Windows version that installs itself without admin rights, “Don’t be evil” my arse.

      • While the site seems to block anything other than Safari on iOS or OSX, it does have a normal HTML5 video tag with a .mov file as source. That should work on Windows (if you have quicktime) so they block it on purpose. Thats not really user-friendly. Especially if you are trying to get people to buy new stuff. You are effectively blocking a group of buyers. And if they would block them but allow streaming via youtube or twitch that wouldn’t be much of a problem, but now it really is

  • Diana

    “the annoying Chinese translator yapping over Tim and Phil” You mean the person *speaking Mandarin*? Doing her job? that someone else screwed up? “yapping”? Really?

    • C

      Yeah… the yapping comment sounded a little rough there… but to be honest, she was talking over Tim and other speakers the whole time and it was annoying. Not her fault though, she was just doing her job. The Apple team screwed this up, all in all though I feel like people are excited about their products.

    • William D

      For some Americans, whether they go abroad or stay at home, anything non American-english is yapping.. That’s why many of them seem to shout in American-english when they go to France or Germany or Asia and make the smallest effort to speak the native language.

  • bahlung

    And I was blaming my ISP for bad internet during the show..

  • Pearl

    Flash and javascript are archaic and need to just die.

  • Fab

    This sounds total bullshit to me, the AppleTV stream was as unstable as the webpage.

  • C

    Thanks for the explanation… I love Apple but that presentation was not done well. I had to load the stream multiple times. They lost some of their audience. Apple needs to make it available livestream on non-mac devices. Why not evangelize to PC? Bad PR.

    Still love Apple though!

  • MagicBoyUK

    Well that was a cock-up. No need for the smart arse webpage, the old quicktime version worked fine for the last umpteen events.

    I gave up and went to cook some food after 5 minutes.

    I’m getting used to Apple cock-ups. The 2011 MacBook Pro with AMD GPUs made of chocolate is a case in point. #mbp2011

  • chapps

    That doesn’t sound right. Because the stream was identically screwed up via the Apple Events app on Apple TV. Any hinky JS on Apple’s website shouldn’t affect the stream in the app … unless they’re calling the web page code. Confusing.

  • nostatic

    For some reason, Apple never seems to get big launches right. Not only was this live stream a disaster, but I remember the past three times I’ve tried to buy an iPhone online the day it goes on sale only to meet with huge server overload issues. While it’s almost impossible to prevent the occasional glitch or mishap, Apple prides itself on perfection so one would think they’d have basic stuff like this down cold. Guess we can’t snicker at Microsoft anymore.