Why Consumer Reports’ MacBook Pro battery tests were wrong

By

MacBook Pro
The new MacBook Pro has great battery life.
Photo: Ste Smith/Cult of Mac

The new MacBook Pro is set to receive a “recommended” rating from Consumer Reports after working with Apple to discover why tests showed the laptops suffered from unreliable battery performance.

The culprits? A hidden Safari setting and “an obscure and intermittent” bug.

It turns out that the testers at Consumer Reports got horrible battery life because they were using a Safari setting that turns off browser cache. The setting is used by web site developers and isn’t normally used by consumers.

Now that a fix has been implemented, Consumer Reports is redoing its tests.

“We have now downloaded the software fix and are rerunning our battery tests with the fix in place on the same computers previously tested,” Consumer Reports wrote in a blog post. “If the battery life results are consistently high, the ratings score for MacBook Pros would rise, and those laptops will then receive Consumer Reports’ Recommended rating given their performance in all our other evaluations.”

Fixing the tests

Consumer Reports originally rated battery life as “highly inconsistent” on the MacBook Pro after test machines got between 3.75 hours and 19.5 hours of battery life on the same machine. It was the first time the well-regarded reviews organization failed to recommend an Apple laptop.

Immediately after Consumer Reports published its test, Apple SVP of Marketing Phil Schiller tweeted that the company would work with the organization to understand the strange battery test results. After looking into the matter, it was discovered that Consumer Reports use of the hidden Safari setting triggered a bug that reloads icons.

The following statement was released by Apple concerning the new findings:

“We appreciate the opportunity to work with Consumer Reports over the holidays to understand their battery test results. We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook we’ve ever made, we respect Consumer Reports and we’re glad they decided to revisit their findings on the MacBook Pro.”

Deals of the Day

  • Ari Rando

    So what does everyone think of that? I’ve owned Macs and MacBook Pro for some time, so I’m a user (not a PC guy here, lol). But still, there’s something weird about this all along. I honestly didn’t ‘get’ whose play this was.

    • WiscoNative

      It wasn’t anyone’s “play.” It was Consumer Reports finding a bug in Apple’s code, causing poor battery life. They reported it, Apple investigated and is working on fixing it.

      • Ari Rando

        Hmm, that’s what I was trying too figure out. Because in customers general use, that ‘bug’ wouldn’t show. I was thinking why it would on CR’s testing. But that’s probably why then. Thanks for the shout!

      • charlene.sturgis

        Look over following article to figure out how an individual mommy was able to make $89,844/year in her extra time on her laptop computer without selling anything>>> FL-Y.COM/3m17

      • Furutan

        What fix? It was NOT a bug – it was an idiot at Consumer Reports who went into the browser preferences, turned on the Developer Tools menu (something that no consumer ever sees) and then _turned_off_the_browser_cache_. Once the dunce turned it back on, the battery life was exactly as Apple advertised.

      • Glen Baerett

        Where in the Apple Manual is this functionality disclosed?

      • Domenic

        What’s a manual?

      • Ari Rando

        That’s what I’d understood from the post as well.

  • Furutan

    This is annoying, as it was because a moron at Consumer Reports demonstrated their lack of expertise by turning on Developer Tools and turning off the browser cache. When they turned it back on, the battery life was exactly as advertised.

    Years ago CR did a shootout with desktops. All the PCs had Ethernet cards. The Mac had gigabit Ethernet on the motherboard. The guy at CS (being a PC guy) simply looked for a dedicated Ethernet card in a slot and, not seeing and PCI Ethernet card on the Mac, he decided that the Mac did not support networking. The Ethernet port was right there on the back – you could see it in their own photograph. Even so, their comparison grid said the Mac did not support networking and in the text of their conclusion they stated that the (supposed) lack of networking made it a bad choice.

    Naturally, they never published a retraction.

    Why did this bug me? Because the school where I taught had been given a grant for a Mac lab and the committee of teachers (none of which were computer gurus) read the CR article and decided to buy Dells instead. The educational software was years behind what they would have had on the Mac and the machines were such junk that 3 out of the 28 failed so badly that they had to be exchanged for brand new machines (a 10% hardware failure rate).

    • Glen Baerett

      You kinda lost me when you called the person at Consumer Reports a “Moron”.

      I bought a car from Consumer Reports after they wrote a review for it. Saved me about $5,000; and it was practically new. But you have to be a member for about 20 years (and ask) to get that kind of benefit.

      Another issue with your assessment– The School didn’t see the value in spending maybe $500 to pay someone for analysis, a report, and better (tailored) advice. Some people may have given advice for free for a reference. I question the leadership of the school for spending grant money without performing research. That’s a major purchase. It’d be similar to asking a car dealer if a Yugo is a car worth spending $50,000 cash on. His answer will always be “Yes, and I have the perfect one for you!”

      Ideally (because this is a staff position) They should have posted a job for a “Computer Tech” to get advice from people looking to maintain the computer lab.

  • Mac2020

    The damage has been inflicted even with no fault on Apple. Regardless of any apologies issued by Consumer Report in the future due to their negligence in conducting said test, it is already too late. The brand name of MacBook Pro has been tarnished beyond repair due to third party stupidity.

    • timtilberg

      You have got to be kidding me.