Why Apple Will Dominate the Gesture-Based Future



In the 50s, some futurists predicted food pills instead of meals. It never happened.

The biggest reason futurists fail is that too many predictions are based on the possible, rather than the desirable.

It’s now possible for anyone to take all their nutrition from pills. But people enjoy eating food. That’s why we don’t take pills instead.

If you want to predict the future, you need to deconstruct human nature. You also need to know what will be possible. Where these two things intersect is where accurate predictions can be made.

And that’s why I can already tell you what your iMac will be like in a few years.

Computers will be everywhere, of course — in the car, kitchen, bathroom and elsewhere — continuing the current trend. However, you’ll have a main desktop system that will essentially be a huge iMac set at a drafting table angle.

It won’t sit on a desk. It will replace the desk entirely. The screen will swivel to flat, so it can be used as a desk or for more than one person to use at once (for games and so on), and it will swivel to vertical to be used as a TV, for presentations or for using the computer from the other side of the room.

The mouse will be non-existent. Keyboards will be made of software. (A minority of writers, programmers and older users will continue to use physical keyboards, which will be built to sit on the angled screen.

Most people will accept software keyboards because people will barely type. Most words and characters will be placed on screen through Siri-like voice interaction, and dictation.

What little typing takes place with a software keyboard will be heavily automated through advanced auto-correct.

The main way you’ll manipulate things on screen will be multi-touch. You’ll do this for moving things around, re-sizing, gaming and other tasks.

If you’re imagining a giant, advanced iPad plus a super-smart Siri, then you’re on the right track.

Here’s the part that doesn’t yet exist on your iPad: You’ll use a lot of in-the-air gestures as well. You’ll wave your hand to go to the next page, document or picture. You’ll to a quick “go away” gesture to dismiss applications.

While sitting or standing at your iDesk, or whatever, you’ll use mostly voice and multi-touch together. But you’ll get in the habit of continuing to use the system as you walk around the room, and for this kind of use, gestures will replace touch. You’ll change the iTV channel by waving your hand or talking. You’ll scroll down pages with hand gestures.

Our iDevices will constantly “read” us. They’ll know when nobody’s in the room. When someone comes in, they’ll know who it is, or they’ll know it’s a “stranger.”

Eventually, our computers will even read our body language to see if we’re happy or frustrated, paying attention to the screen or looking elsewhere — and change what’s happening on screen according to what we’re doing.

But why?

The reason this three-part interface of the future will happen, and is in fact is already in the process of happening, is that this is the interaction that human beings are hard-wired to want.

If you look at the whole history of computer interfaces, you’ll see a linear trend. As more compute power gets cheap, it’s increasingly applied to making the computer work harder to communicate in human language.

When we interact with other people and with the world, we use voice, touch and gestures. Huge parts of our brains are dedicated, in fact, to interacting with the world in this way.

Human language isn’t just about words. When we talk, we make facial expressions and use hand gestures, and all this is combined in the mind of the listener to receive the full measure of our communicated meaning. Computers will be taught to also understand all this.

Computer interfaces are not going in a random direction, nor are they going in the direction of what’s logical or possible. UIs are moving in the direction of interacting with us as another human would.

Understanding this fact is the secret to predicting Apple’s continued rise to dominance of the computing industry. Apple is the only company I’m aware of that understands the primacy of human nature, biology and psychology in interface design.

The number-one reason iPad has succeeded in the market and Android tablets have not is that Apple managed to eliminate the delay in motion after you swipe your finger across the screen. The iPad satisfies our need to interact with physical objects, while most Android tablets are unconvincing. The iPad thrills for reasons that don’t register consciously, while Android tablets annoy.

The same goes for Siri when compared to competitors. Siri’s playful, natural-language banter and ability to understand is just human enough to satisfy our need to interact as a human would.

Android users are confused by the success of Siri, or blame it on Apple marketing baloney or lemming-like Apple fanboy obsession. The reason they don’t get it is that they’re focusing on what the technology can do (take action on voice commands) rather than what’s satisfying to the user (understand random comments and respond with sometimes playful, human-like banter).

Using iOS is a little like interacting with objects in the physical world. Using Siri is a little like interacting with a person. The Apple devices of the future will be a lot like real life and real people. That that’s why we’ll love them.

Here comes gestures

As with all major improvements in interface design, gestures are being developed by many companies.

Microsoft has been selling its Kinect for Xbox 360 product for a year. It’s a run-away hit, and has spawned enormous customization and re-use by scientists, hackers and hobbyists. Microsoft announced this week a Kinect for PC product, which will come out next year for Windows 8, which itself has a multi-touch user interface.

The Korean company Pantech is already advertising gestures for its soon-to-be-released Vega LTE smartphone.

Dozens of other companies are shipping or working on similar technology, including Apple.

And researchers in universities all around the world have been developing gesture technologies for many years.

We can safely assume that all phones, tablets, laptops and desktops will get gesture control — especially Apple gadgets.

Apple has applied for and acquired several patents related to gesture technology. What’s interesting about Apple’s gesture patents is that they’re almost entirely for content creation, rather than content consumption or data manipulation.

For example, one group of patents is for creating CAD-like or gaming applications, where you do a multi-touch gesture on screen, but lift your fingers off the screen to render that object in 3D. Another patent uses in-the-air gestures for video editing.

Microsoft and Pantech use cameras for registering user gestures. But it’s likely that a range of sensors will be brought into play that register fine movements, as well as accurate distancing.

With the success of Microsoft’s Kinect, it’s tempting to believe gesture control is a Microsoft thing or a gaming thing.

But in a few years, I believe people will see it as an Apple thing. Why? Because as interface design delves deeply into satisfying the hardwired human desire to interact with physical objects and human beings, Apple will dominate this space because that’s what Apple is really good at.

Nobody can touch Apple in the ability to create devices that satisfy our innate human desires about how to interact with the world. And for that reason, the future belongs to Apple.

Very soon we’ll say hello (literally) to our voice-command, multi-touch and gesture-based iMacs — and say good-bye to peripheral input devices.

And we can also say good-bye to the dominance of Microsoft Windows.

Picture courtesy of DreamWorks Pictures and 20th Century Fox.

  • Jeroen Gunning

    I think you made a mistake in “Apple announced this week a Kinect for PC product, which will come out next year for Windows 8, which itself has a multi-touch user interface.” I don’t think Apple would be announcing any Kinect stuff ;)

  • bolthouse

    Thanks for the article. One thing tho, you cant eat your daily nutritions in pills unless youre willing to take thousands of pills per day. Please do research :) ps – so we might see pantech vs apple in court soon lol

    By the way, android users are not confused with the success of siri… i mean really? Cmon.. I know this is a mac site and I love my ipad 2 but you can’t be bashing competitors left and right. There are many users of other operating system and hardware here.

  • Gregintosh

    I don’t think the table concept is going to take off. Its just not necessary. Why would someone buy a big table computer when all their work can be done in a few taps on an iPad?

    It is not desirable for anyone but geeks to have MORE computing power available to them. Normal people just want to get things done, as quickly and easily as as possible so they can enjoy their entertainment, family time, etc. 

    If there was a way to get the work thats needed done as effectively without a computer, companies would be dumping computers. And that’s exactly what’s sort of starting to happen as companies issue iPads to sales people instead of laptops & projectors, and we’ll see more of that. 

    Process innovations in business are going to render computers less useful and accelerate the trend toward specialized mobile apps that do 95% of the heavy lifting in 10% of the time, at a fraction of the cost too.

    And for personal use, we want to consume content. Having a 6 foot table screen isn’t going to make you read more stuff or watch more videos because you can fit more on there. Instead of the iMac, many people will just opt for a tablet which is comfortable to hold, handles all the content, and can be put away easily.

    A more realistic prediction would be an LCD/e-Ink hybrid that knows when to use backlighting and graphics and when to use just text, to make it optimal for both uses, automatically.

    In the home, a big table computer is going to be harder to fit in than an iMac (which can be placed almost anywhere) or a laptop or tablet. You’re really talking about being in the furniture business then. That’s actually quite against Apple’s philosophy.

    Apple would minimize the presence of anything thats not needed at the moment. Just like the indicators disappear when not needed, they’d prefer that their iPads and gadgets wouldn’t interfere in your life when you didn’t need them – and then be instantly accessible when you do.

    The other trend you’re not addressing is mobility. As transportation technology improves, it may be feasible for people to want to live in many places and not just in one house. For example, people may want to travel the world rather than have permanent residences (as more people work wherever they want thanks to technology, the more this is possible) especially if a Europe to US flight takes only 1 or 2 hours. With super-speed transportation technology, why not rent an apartment in England then commute to Chicago, and stop over in Singapore for lunch? You won’t be lugging your Table computer with you, you’ll probably have your phone or small tablet.

    So again, gadgets that let you take them with you will be the future, not big pieces of furniture that will be hard to move.

  • prof_peabody

    The reasoning behind the “pills for food” analogy is incorrect.  The reason we don’t just take pills for food is because they *don’t* and *cannot* provide all the nutrition we need, not because people just didn’t like the idea or find it desirable.  

    Our guts need a certain amount of fibre and bulk to make things work.  Pills can never provide this.  The best you could hope for is pills for nutrition supplemented with lots of tasteless blocks of compressed straw for the bulk.  

    Futurism fails mostly because it assumes things about the future that turn out to be poorly thought out, and not so much because of any general failure of desire.  

  • Gregintosh

    That’s why in Back To The Future the movies we see hover cars in the year 2000 but no internet or cell phones. That’s because in the early 1980’s people could only think of the existing things getting better, but not true innovations.

    Just like now, we think of what would make computers and mobile devices better, when in reality all new categories we haven’t thought of may replace them. 

    On a micro scale, its like everyone wants to learn how to make a better netbook, and then someone comes out with a whole new category of product (iPad) that renders them mostly useless. 

    Maybe in the future there will be a device that will render smart phones, tablets, or even desktop computers useless. It won’t be a “better” desktop computer, it will be something else entirely. I won’t try to predict one because if I knew what that thing was I’d be patenting the crap out of it right now and not posting here. :-)

  • Finlay MacArthur

    Wait until you get Siri on your iPad, then you’ll know why.

  • GregsTechBlog

    We will never use computers that lay flat on a desk. Why not? Place your iPad flat on a desk, sit down, and use it for 30 minutes without lifting it up.
    How’s your neck and back feeling?
    If desktops survive, they will look like sleek, thin iMacs. Touch, gestures, voice control and even traditional keyboards and mice will survive.

  • Mssangol

    I don’t believe we can say good-bye to our peripheral input devices, it’s more likely that they’re incorporated into the devices. Like touch-screens that can slightly transform into keyboards. 

    The touch and gesture-based technology today lacks a dimension of “action-confirmation”. When we push buttons, we can feel that the button is lowered, with computer mice we feel friction, when sliding it on a desktop. I would like to think that it is desirable to incorporate such dimensions to the gesture/touch-based tech. It’s ridiculous that you need to look at the touch keyboard, and confirm with your eyes, that you are pressing the right buttons!

  • Shameer Mulji

    I agree that computers won’t necessarily lay flat but they will tilted similar to that of a drafting table.  

    Imagine an iMac as large as a drafting table & fully touch enables laying tilted in front of you running Final Cut Pro.  That could potentially be a lot more productive than using mouse & keyboard.

  • GregsTechBlog

    You’d still have a large portion of the screen below your line of sight. Also, if we still type at this point, typing would be a pain because you’d have to angle your wrists so much. 
    I’d see computers being hung on the wall in this future though, with a wireless keyboard, and integrated trackpad. But most people would be fine using the built in voice control for navigation and word processing. Eye tracking combined with voice recognition would be so perfect, I should go get a patent for it.

  • AdamC

    The catch words are ‘adjustable tilt’ of the device.

    Computer hanging on the wall?, I believe Apple have done that with the AppleTV and airplay (correct me if I am wrong).

  • casfian

    Think of Exo Desk – the Touch Screen desk! Saw that a couple of days ago! Cool!

  • GregsTechBlog

    You can’t mirror a mac display onto a tv through the Apple TV yet. 
    Technically, you could put a monitor on the wall, etc. But I think it’s more likely that people will use wall mounted computers than want to look down at a screen for even a second. 
    Why even have it as an option if no one will use it?

  • Liam Robichaud

    I don’t think voice will take over for keyboards. What about all of the times when the user is in a quiet zone or wants to type private things that they don’t want others in the room to hear? I think the feature will become entirely available but not replace typing.

  • metroworker

    I think Mike is on the right track. But in recent years there have been many ‘mock up’ future rooms (like at EPCOT and Disneyland, etc…) about the future home and work spaces. Computers will become more discreet and integrated. But there will be one central ‘hub’ that connects, syncs, and coordinates all the remote devices that will be in the living room, kitchen, bathroom, etc…even connecting to our phones, cars, etc…outside our home.

    In the meantime, I’m thinking there should be an iPad Pro — a large, A3-sized (approx 11×17 for our American friends) iPad for work spaces. These are intended to be in and around the office or at meetings. We talk about drafting tables, imagine architects or engineers running around with A3-sized iPads with work-oriented productivity apps. Drawing, large spreadsheets, even large musical instruments or sound boards, would be possible. Teachers having enough room to see student work + video conference with a remote student…

    Eventually, the iPad will be more like the iPhone in size but able to utilize virtually ANY surface to type upon or use as a display screen. Data, images, and other content can be ‘swiped’ from surface to surface simply because it is captured in a hub device that goes where you go. So, a sketch you start at home on your table can be transferred to your phone then once at work transferred to a work surface simply by waving your hand.

  • mlahero

    It’s an interesting article but saying such nonsense like “Android users are confused by the success of Siri” really dilutes the quality. I don’t see any Android owners scratching their heads about Siri. Try and be more objective please Mike.

  • Daniel Velazquez

    that central hub already exists, it’s called “the Cloud”

  • HerbalEd

    Even if food tablets can/will provide complete nutritional needs, people will continue to “eat” food because of good taste. Nutrition pills will no more replace real food than will artificial insemination for reproduction replace sexual intercourse. 

  • HerbalEd

    You are right on. I’m always amused when looking at a 1950’s sci-fi movie where the space ship has big dials and switches. Talk about no sense of the future.

  • macgizmo

    While I don’t disagree that Apple will dominate the Gesture-based computer market for a short time, I don’t think it’ll last forever.

    What we’re essentially talking about here is the MS Surface, only made in a way only Apple seems to know how to make – for common people. But I don’t think they’re going to replace traditional computing entirely, especially keyboards, anytime soon.

  • David Clark

    I feel like, sometimes, children write these articles…

  • Gregintosh

    The fact that Android users think Siri is no big deal, and even Google’s mobile person said that people don’t want that kind of technology in their phone shows that they are INDEED in the dark about Siri and what it means. They can’t seem to figure out that people DO in fact want their phone to understand them and make their life easier (which leads them to scratch their head about why Siri is so popular).

  • Gregintosh

    Probably mind control will be used for quiet areas, heck it might become the standard input for any time. There is a lot of research going into reading our thoughts and make the output useful. We can already control a mouse or even type on a keyboard with just our minds (granted, not practical setups for everyday use YET but as the technology gets better it will be).

    So I can see how maybe in the next 15-20 years we’ll be able to “pair” our minds with tech devices and control them entirely by thought alone.

  • Steffen Jobbs

    Wall Street believes that Siri has no advantage over any other “voice recognition” system and that it’s basically a gimmick that won’t even help drive Apple product sales.  Hardly a game changer.  It’s amazing how much hype Apple can get behind its products.  I think more consumers are interested in 4G than “intelligent assistants.”

  • AvoidDroid

    Wall Street ???? You mean those guys that lost virtually ALL my retirement savings ???
    Yeah those are great guys to listen to ! What’s amazing is that anyone at this point can give Wall Street ANY credibility for prognostications ! BTW SIRI encorporates artificial intelligence, a mite beyond “voice recognition”whether the geniuses on Wall Street acknowledge it or not .

  • AvoidDroid

    Agreed. And these are the same people who came away from the iPad launch saying, “Ha, Ha, Ha ! It’s just a big iPhone ! Ha, Ha, Ha ! ” The joke’s on who now? Regardless of what they are saying, you can bet they’re scurrying about trying to mimic SIRI asap. Why would they quit replicating EVERYTHING Apple does at this point ?

  • macowling

    I think this is spot-on Mike! Sometimes you can be a bit “pie in the sky” and I’m sure people will dismiss this article the same way, but I think what you’ve described here is the future, and Apple has a commanding lead. Maybe the geeks can’t see it and keep on telling us “Android already has voice recognition”, but it only takes one look at the queues at the Apple store of “normal people” to realise that you’re onto something here, people want a “real” user interface.

  • Steven Zahl

    Mouse will still be around.  Nothing beats it for cost and finger light operational effort.

  • Wouter Hermans

    Y’know… Ever since I got the Apple Trackpad, I haven’t touched my mouse…

    I wouldn’t go as far as believing we will Minority Report our way through Reddit in a few years, but the mouse is for me a thing of the past. Mainly because it is so limited, even with a large number of buttons and scroll wheels.

  • Wouter Hermans

    What I find odd is that MS Surface was there at the same time Apple came out with multitouch on small devices. And then nothing happened.

  • gareth edwards

    “you’ll have a main desktop system that will essentially be a huge iMac set at a drafting table angle.” – I’ve been thinking this form factor since my days at Uni, nearly 20 years agao. It’s the perfect solution for designers – bring it on PLEASE!!!!!!

  • arjwright

    Apple has done well with pictures under glass (http://worrydream.com/ABriefRa…, they haven’t advanced touch-based interactions and it doesn’t seem as though they will (Siri, then select-swype). If Apple manages to mediate voice as a spatial interaction (not just input and response layer) then yes, they can do well here. But leading? No. They pioneer the use of some behaviors to masses, they don’t create new. And especially as it is with gestures, they adapt on current memes. So, in light of these, no.

    Plus, what kind of sense does it mean to dominate gestures? How about dominating spaces ;)

  • Steven Zahl

    Yeah, but how much is yer Apple Trackpad?

  • WVMikeP

    Don’t confuse being around with being dominant.  Stylus-driven tablets are still around.  They sure aren’t dominant.

    Once the UI can support my needs as a photographer via finger rather than mouse (WHEN, not IF), the mouse will be dead to me.  However, it will be (and is) dead to many people already thanks to the iPad.

  • WVMikeP

    Geeks aren’t the taste-makers any more and they can’t seem to calculate why.

  • Shaunathan Sprocket

    MS Surface, courrier…  Microsoft had everything they needed to keep apple in check, instead, nero plays his fiddle…

  • Samuraiartguy

    I remember seeing a touch senstive drafting table display prototype at a Macworld Expo many years ago. It was like a Wacom Cintiq writ very large. Complete with a drafting arm, stylus and precision puck. But the thing had a power cable the size of my wrist and needed a Silicon Graphics workstation to drive it and cost OhMyF**kingGod. Not for the masses. But it was DAMN cool.

    We’ve been thinking about this stuff for long time.

  • Samuraiartguy

    I am falling in love with my Apple Trackpad, sitting opposite the screen from My Wacom Graphics Tablet. I don’t even turn ON the Magic Mouse. It’s too small, low and too teeny, and I have MAN-hands….

  • Samuraiartguy

    Elsewhere on the site someone is posted a comparison between Siri and the Windows Phone Voice Recognition technology. Guess which one actually works and is useful? No, No, No… you go look, don’t want to spoil it.

    And I’ve stopped listening to the thieves on Wall Street since 2008.

  • rod griffiths

    I liked the article, but the throw away line about living on pills got me going. The most energy intense food is fat at 9 calories per gram. Even assuming the most meagre diet you’ll need about a 100 grams of pills. I just weighed a few pills, most are well under half a gram, so you’d probably need 250 or more. OK so you could, in theory live on pills, but you’d be taking one every five minutes for most of the day and if you needs a small glass of water to get a pill down, you’d be peeing all day. 
    Food is more fun.