Ex-Apple engineer tells how the company’s manufacturing works

Almost all electronic products are still assembled by hand, even hundreds of millions of iPhones.

But that’s changing. Apple’s supply chain is rapidly automating using AI and robots.

At the forefront of this is an ex-Apple product design engineer, Anna-Katrina Shedletsky, who is using her expertise to help other manufacturers build their products.

On this episode of the Apple Chat podcast, we talk to Shedletsky about her new AI startup, Instrumental; Apple’s giant manufacturing operation; the role of product design; and much more.

If you’re curious how Apple makes its products, listen to the podcast or check out the full transcript below.

Instrumental founder and CEO Anna-Katrina and I discuss the current state of assembly-line manufacturing around the world, and in particular, how a large-scale manufacturers like Foxconn operate. We talk about the unique challenges facing large-scale manufacturers, including how to ramp up production for the holiday season, and how to manage the sheer scale of humans moving into and out of a factory.

We also discuss in detail how manufacturers and design engineers work together on optimizing the yield that comes off a production line. We discuss how companies determine what goes onto the “bonepile” (units not fit for consumers) and how the threshold changes depending on the product. And we talk about where automation fits in the process, and her views on automation replacing humans on the assembly line.

Finally, Anna-Katrina explains how Instrumental takes machine learning designed in the consumer space (for things like Netflix search recommendations) and applies it to the manufacturing industry.

Leander: Anna-Katrina Shedletsky spent almost six years at Apple, working as a product design engineer. What’s a product design engineer? They’re the folks who work out how to “pack the suitcase,” how to stuff all the technology into the beautiful enclosures, designed by Jony Ive’s industrial design team. The product design engineers also help figure out how those products get made in their millions on giant production lines in Asia. In fact, Anna-Katrina spent a lot of time on production lines, while working on four generations of the iPod, and then, as a product design lead for the Apple watch. Anna-Katrina has since left Apple and has used her hard earned expertise to launch a machine learning start-up. The name of her company is Instrumental and it uses AI technology to help companies find and fix problems on factory assembly lines.

In this episode we talk to Anna about big changes ahead for manufacturing, how Instrumental is laying the groundwork for truly intelligent factory robots and how giant companies like Apple design and manufacture products in their millions. It’s an extremely complex process where a lot of things can go wrong. Hey, Anna. Thanks very much for coming in today. You had a long career at Apple before launching your new company Instrumental. Can you tell me a little bit about your new company and what it does?

Anna-Katrina: Yeah. My company is Instrumental and what we do is we help hardware companies find and fix issues on their manufacturing lines, so that they can ship their products on time and at high levels of quality, that Apple like quality.

Leander: Okay. Apple like quality and Apple like numbers I take it, too, yeah?

Anna-Katrina: Potentially, yes. We work with a broad range of customers from very small start-ups to Fortune 500s.

Leander: Okay. This came to light because of your experience working at Apple.

Anna-Katrina: Certainly my time at Apple as a product design engineer exposed me to manufacturing in a way that you could not get exposed to in any other job. I was able to see some of the most advanced manufacturing lines in the world and some of the most rudimentary manufacturing lines in the world. This was a great survey of where is consumer electronics manufacturing today. That clued me in to the fact that there is a lot of changes that haven’t been happening in manufacturing over the last two decades, but that are starting to change now. That was an opportunity that I was made aware of because of the exposure I got in my role at Apple.

Leander: What are those changes?

Anna-Katrina: Consumer electronic devices are almost a 100% handmade, bespoke, hand-assembled. They are produced in these facilities on lines that are … Standard line like this a 110 meters long. A person is 0.6 meters wide. They are stacked down this line. Hundreds of hands touch a smartphone, regardless if it is an Apple smartphone or Samsung or Google smartphone. This is state of the art for assembly of these complicated, highly miniaturized devices. I think a lot of people imagine robots are doing this stuff already. You hear about automation in manufacturing in the news, but that hasn’t made it to consumer electronic assembly. Other industries a little further ahead, so-

Leander: Like cars?

Anna-Katrina: What’s interesting about cars is that there’s only a really small piece of the line that’s automated and that’s the piece that you see all the youtube videos with the big robot arms. That’s the body shop part of the line. There’s extensive part of interior assembly that robots aren’t doing, humans are doing those jobs today. There’s still a very human element. In the consumer electronic industry there’ve been opportunities to expand into new technologies as robotics arms have become more prevalent and CCD systems, essentially computer vision systems, actuators, all of this has creeped into various aspect of consumer electronic assembly, but not specifically the integration of all the parts, the final assembly, the put the display, put the enclosure, all the speaker, battery, all that stuff is still hand done by people, hundreds of people.

Leander: But you mentioned changes. It’s changing?

Anna-Katrina: Yes. Foxconn, Flextronics, Jabil, those are the top three contract manufacturers in the world and there’s a long slew beneath them as well. They’re all very interested in pursuing automation. There’s actually initiative in the Chinese government to supporting, where they want to replace 800,000 workers by 2020. This is a government sponsored initiative that Foxconn and Flextronics and others are really interested in trying to replace these people with machines. That work is ongoing and Foxconn releases press releases about how many workers they’ve replaced. This was the evidence of a crack in a wall that had previously kept out this kind of technology. Finally, it’s now cheap enough and viable and nimble enough for the types of activities that humans are doing. That creates an opportunity for a company like ours, which really leverages the data that those types of robots could collect.

Leander: Okay. What exactly is a product engineer? What is that role?

Anna-Katrina: I was formerly a product design engineer. Product design engineer is kind of fancy way of saying mechanical engineer. In my role I worked on three different iPods and then I lead system product design for Apple watch series one. As a product design engineer I did a variety of things. Starts in the architecture phase where you are responsible for designing the CAD, the company model of the new product for what I call ‘packing the suitcase’, which is getting all the parts to fit into that beautiful industrial design and negotiating that process with all of the stakeholders involved. Then, actually taking that design, selecting materials, doing validation on the part level, going to China very frequently, doing these engineering builds. This is all very standard process throughout the industry, but doing these engineering builds and essentially, my role as product design engineer was to validate that it was possible to build units at mass production yields, at mass production speeds on one line. Once we were able to validate that, we got to hand off our role to operations who would then scale up that to many lines and very high volume in mass production.

Leander: I’m interested in how you interface with the different teams. The product design, they’re in between the industrial design team, that creates the phone factor and the operations team that works out how these things are made in factories at massive scales.

Anna-Katrina: Yeah. There’s many teams at Apple. Absolutely industrial design team is holding the vision of what the product should look like and even at times how it should function and what the customer experience should be. There’s an operations team also called manufacturing design team, who really treats the manufacturing process as a product that also should be designed. How the parts are made, what are the specific tooling pads for this uni body enclosure or this particular part. Where do we put the gates on the injection molding tools.

There’re people who are really good at all of these aspects who are responsible for that kind of expertise. There’s people who are good at painting and coding parts. A lot of plastic components get painted, even though it doesn’t look like they are painted. There’s these teams. There’s also these additional team that are excellent at a key functional area. There’s a battery team, there’s a display team, there’s a touch team. Each of these key components has a team where they focus on their experts [inaudible 00:07:44] who know how to develop awesome acoustic systems and work together with the broader team to ensure that the product has awesome acoustics and meets the requirements.

Leander: And everyone’s working simultaneously and concurrently together or does it go from one team to another?

Anna-Katrina: There’s a lot of working together throughout the whole process. Operations is involved very early on, but they ultimately essentially take over from R&D and really scale up the process. They’re involved early on to make sure that as a product design engineer I don’t design something that’s not going to be manufacturable at volume.

Leander: They make sure that you’re not designed something that isn’t going to be manufacturable?

Anna-Katrina: Yes.

Leander: Too many negatives-

Anna-Katrina: Yeah, sorry.

Leander: Right. They make sure that you’re making something that could be made. You’re designing something that could be made.

Anna-Katrina: Absolutely. We work together on that. That’s a push and pull, because absolutely at any of these top consumer electronics devices companies, they are pushing the limits every single product cycle. There’s new technology that hasn’t existed before. There’s thinner wall sections, the rules get broken. That’s a push and pull, but to make sure that we make good choices around risk from a production standpoint and a design standpoint, we work together very closely. Then, ultimately design and R&D gets to step away as we start to scale production. This is very common as a process in the industry.

Leander: In general, how long is this process?

Anna-Katrina: It can vary for product to product. It can vary based on the urgency of that particular product or it can vary because maybe the idea for what the product should be isn’t totally clear yet and some prototypes have to be built at scale to really understand is this product working? Especially in a new space, where there wasn’t previously a product like that. For first generation product.

Leander: For example?

Anna-Katrina: For example, some products could potentially run on a scale of months, like six months. Some products could take years, many, many years. Just lots of iterative cycles. Really depends on the product. We, in our capacity, at Instrumental get to work with other Fortune 500s as well who have similar processes and again, similar scales on their schedule. Some of these schedules are super urgent, super fast, nice and tight. Some of them are … There’s time that has been designed in for exploration of what the product should be.

Leander: Right. Presumably, products like cell phones, which tend to get refreshed every year, these are the kind of short product cycles that you’re talking about.

Anna-Katrina: Depends. Because some of the technology and some of these products could have started being developed years ago. When it’s ready, then it could potentially go into a new product.

Leander: It appears as though these things get refreshed every year. But actually some of them have been in a pipeline for three, four, maybe longer.

Anna-Katrina: Potentially. Some of the components, some of the technologies, absolutely, potentially. These new technologies don’t get invented in those six months timeframe that you have to run a program or in the 18 months timeframe. You can look at other companies like Samsung or LG in terms of OLED technology, for example. Took them many years to figure out how to … They were showing at CES, flexible OLED displays for years before they shipped a phone that had an OLED display in it. You have to count the development of that new technology in the schedule, but the actual schedule of the product itself is pulling the technologies that are ready, pulling those together into a new product and then executing on a design that makes that product real.

Leander: How complex a process is this? It sounds really horrifically complex. A lot of things can go wrong.

Anna-Katrina: It is actually quite complex. What’s really interesting about it is that there’s not a lot written about the best practices for how to do this process-

Leander: Because?

Anna-Katrina: I think this is a very secretive industry. In general, people are trained not to talk about what they do. I didn’t use the word ‘watch’ as a verb, for the period of time I was working on the watch program. Like ‘to watch TV’. No, we’re going to use some other verb.

Leander: Oh, it was so engraved into you that you couldn’t use that.

Anna-Katrina: Never use that word. There’s just this secrecy around how this stuff gets done. Because there’s secrecy, there’s confusion. There’s even confusion on the team around what is an EVT build, is a great example. Before you get to production units that you’re building for customers, you do these practice builds. This is industry standard. These practice builds are essentially groups of prototypes that are built, so that you can test and iterate on the design, make sure the design is ready for mass production.

Leander: That acronym you used, EVT is-

Anna-Katrina: EVT means Engineering Validation Test. This is a term that’s defined on Wikipedia as Engineering Validation Test and then there’s very little other information about what it means. Actually, we wrote, Instrumental wrote a guide that goes through in details, based on our interviews across the industry what is an EVT, what are the entry criteria, exit criteria, what do you need to accomplish in this build to actually get to EVT maturity. Essentially, we’ve got these key on the internet that nobody had written content for. These, we’ve been told, have been printed out and sent to manufacturers around the world, pinned up in an engineering war rooms around like, “Okay, have we reached EVT level maturity? Can we move to the next stage?” There’s just a lot of confusion around what these processes actually mean, but there’s this general consensus in the industry that you do several builds, you are ever moving towards a more mass production capable process. Then when you’re ready, you turn it on and hope for the best.

Throughout that process things can go wrong and that’s where we come in. Between first prototype and first production unit, which could be, as I said, six months to 18 months or longer, depending on the product. There’re hundreds, if not thousands of issues that need to be found and resolved by engineers. Today, any one of those issues can actually block a product from being released and launched to customers or delay the launch of that product. Today engineers rely on very manual processes to find these issues. They get on planes and fly to China or wherever the factory is. Often in Asia. They stand on manufacturing lines for hours, trying to be in the right places at the right time to see that operator one time out of a hundred pulls the release liner from left to right instead of right to left. That created the difference that creates a problem down the lines. That’s a problem too in the design. If you’ve designed a product that’s so sensitive, as product design engineer I’m always thinking about this.

If I’ve designed a product that’s so sensitive that that difference matters for whether the product’s going to pass or fail or be successful, then maybe there’s a design issues that I need to work on as well, not just a process issue or operator training issue. These are the types of things that come up during development that have to get hammered out. Part quality is another very common one. Process, like glue, large amounts of glue are used in consumer electronic products to hold them together. I think a lot of people don’t appreciate how much glue there is. Glue seems like this great material, because it fills any gap size and you can put it anywhere. The glues are pretty nasty. They are really difficult to control. Their strength is based on a lot of parameters and chemicals eat them. There’s a lot of things that go into selecting and validating a glue process. If you’re gluing, that’s a problem.

These types of things are where issues come out of. Then, there could be issues with the design. I could have actually designed parts that at times interfere. Or that something that can be put in backwards and it still fits. Those could be design issues. Those are the types of issues we’re helping our customers to find during development. And then, during production we’re enabling them to continue monitoring to be able to understand if there’s process shifts or when they should take tools down for maintenance or if there are quality shifts or tools qualifications that have to be done. Our data’s helpful in assisting the issues that arise in the production.

Leander: Instrumental, you’ve developed a camera equipped station. Is that right?

Anna-Katrina: Yes.

Leander: This goes in various places in the production line to look for these kind of issues.

Anna-Katrina: Yes. We have a combination of hardware and software. We really view ourselves as a software company or as a data company. The reason we built hardware is because we know that the people we’re selling to are really busy. They don’t have time to integrate expensive software and doing all of these steps. We need an off the shelf system that just works.

Leander: Some of it they can plug in to their currents and production lines.

Anna-Katrina: They can plug in to their current line. We build a piece of hardware that’s very simple. It’s a station that goes at key places on the assembly line. We take high resolution images at key steps of assembly, where you can see key actions.

Leander: I’m sorry, what exactly is a station? What do you mean by that?

Anna-Katrina: A station is an overloaded term that’s used to refer to essentially a specific space on the line. That space is usually 0.6 meters wide, because that’s the width of a person. A station could be a space where a person is sitting on the line. It’s a specific step or operation that’s going to happen. We have overloaded it to also mean that we build a box, we build a box to control the lighting to take high resolution, high quality images at these key states of assembly. We capture that data. We get that data out of China into our software. Our software enables our customers to view their images from anywhere to compare and understand variation and anomalies, using machine learning techniques. Then we also have the ability to do first pass failure analysis, which is when something goes wrong, what is the first thing you’re going to want to do. We let our customers do that through the software, which is to virtually disassemble a unit and even take calibrated, Ad Hoc measurements. Again, this is after the fact.

Leander: Let’s say they find a problem, they don’t necessarily know where that problem originated. They’re going to be tracking back station after station to see okay [inaudible 00:18:23].

Anna-Katrina: Yes. Here’s a good example. Say that you’re working on a wearable product. Wearable product probably needs to be water-resistant, because maybe you wash your hands, it’s on your wrist, you wash your hands, splash resistant. Maybe it’s out in the rain, if it’s headphones or something like that. Wearable product needs to be water resistant. You could have a water resistance failure in a reliability test, where you’re actually trying to demonstrate and validate that you have a water resistant design. That failure could be caused by a bunch of different root causes. Our software enables our customers to actually go back and disassemble the unit virtually from anywhere. From their desk here in California and see oh, this one’s missing a screw. It just didn’t have a screw and that screw is important for holding the water seal. I can see that that’s the reason why I have a water seal problem. We can go one step further from root cause and actually help our customers get to corrective action. What that means is I know there’s missing a screw, I can look at nearest neighbors on the line to understand were there more units missing screws.

I can use our machine learning techniques and machine learning algorithms to actually see if there … Automatically look through population of a 100 units and see if there’s automatically one that’s missing a screw as well. I can understand when that happened on the line, same shift, same day, 15 minutes apart, after a break. These details are really helpful for me to understand how to actually … Not only understand there is a missing screw, but how to go fix that problem, so it doesn’t happen again. That’s the kind of operational process issue that when you’re building a 100 units, these happen [onesie, twosises 00:20:05]. We call them [onesie, twosies 00:20:06]. One or two units will have this issue. But if you’re building a million units a day, that’s a lot of units that are going to have that problem. The whole idea is solve it when there’s only one or two. Then you never have to suffer the fallout of tens of thousands of units that might have this issue.

Leander: You mentioned yields in production lines. People get impression I think that it’s a 100% yield. But it never is, is that right?

Anna-Katrina: I initially went to go to Apple to be a product design engineer, because I wanted to learn how to build millions of things. I went in with the now naïve idea that in order to build millions of things, you had to be perfect. If you’re building a million a day of something and you have one percent fallout, that’s a crazy amount of units that is not going to get shipped. If you just think about the space those units take up, where you put them? How do you repair them? I just-

Leander: We’re talking about iPods, right? In terms of your career.

Anna-Katrina: I’m spouting numbers of a million a day. It could be hundreds of thousands a day. In general, just this idea of yields and the interesting aspect there is I assumed that that meant perfection. It doesn’t mean perfection. It means is very common in the manufacturing process to have procedures and processes that are not a 100% yield. Those yields could be as low as 80% yield. That means you put hundred units down the line and only 80 are going to be good. What do you do with the extra 20? Sometimes those get scraped, sometimes they get repaired or reworked. In that rework process there’s yield fallout as well. There’s a lot of challenge around this idea of what do you do with the units that don’t need the spec. One thing is you can prevent those failures from being made. The other thing you can do is you can change the spec. Is the spec the right spec?

The spec is does this scratch or does this performance, whatever it is, speaker performance, microphone performance. Does it meet the requirements of the product to make sure that the product is a high quality customer experience. That’s what a spec means. If you were to build a product that had a scratch, that was out of spec means it’s too big to ship. For a company like Apple has really staked Apple like quality. They don’t ship anything that has things that people notice. If it has a scratch, it’s probably not going to get shipped. But a start-up should probably ship that scratched unit. They are not Apple and they can’t withstand that kind of fallout.

For some products, the fact that there is a scratch on it when it’s coming out of the box still most people won’t notice. Some of these things can be very, very, very minor. Some of them are big. If the microphone does not work, then it’s not meeting the spec and it’s going to be a bad customer experience, regardless of whether you’re a large company or a small company. That is the problem that you need to fix. That’s how you would use software like ours to actually go and trying to figure out how you could improve that yield.

You can also use our software to understand what the spec should be. Because today if you want to understand okay, here’s the various levels of scratching. A, B, C, all the way down. If we pick, it’s this percentage yield. If we pick B, it’s this percentage yield. Each one going down, the yield is going up. If you accept C level parts, you get a 100% yield. But if you only want to ship A, it’s 95% yield. That is done by doing these massive yield studies, where you actually count and inspect parts through human process. Takes weeks. There’re people whose job is to do these types of yield studies. You can use software like ours to very quickly understand what those yields actually are for varying levels of defect, which is an exciting application. That could mean maybe upstream, let’s talk about a part upstream. You’re building a housing enclosure and [inaudible 00:24:04] and there’s a small bur. It’s on the inside of the product, so it’s not going to affect the outside cosmetics. Do you throw that part out, because it’s technically out of spec? Or you change the spec?

Leander: [crosstalk 00:24:18] changing the spec.

Anna-Katrina: Understanding that oh, we’ve built parts that had burs like this and they did not have reliability failures or customer impact kind of failures. Maybe we should change the spec. Then we don’t throw out all these parts and we save resources. This saves money, saves time and all of that.

Leander: You were saying the ones that fail inspection, the ones that fail the spec can be up to 20% or so.

Anna-Katrina: For some parts, some processes. Not necessarily finished goods.

Leander: Do you have good specific examples you can talk about?

Anna-Katrina: No.

Leander: Okay. Was worth asking. You mentioned earlier, some products that people have been looking forward to get delayed. That was a good examples last year.

Anna-Katrina: Yeah. There’s many examples from last year. Just looking at an article, it does. But yeah.

Leander: Rile them off. Which ones were you think of?

Anna-Katrina: There’s a GoPro drone. There was a DJI drone-

Leander: Yeah, yeah. The one [crosstalk 00:25:15].

Anna-Katrina: There’s Apple AirPods, of course. There were more, just forgetting them. There were several products where it was very clear the product have been delayed. This happens very commonly. I think people expect it in Kickstarter products. They don’t expect it from these large corporations that they admire, that they … Of course they have their act together. It’s Apple, it’s HTC, it’s DJI. These are great brands.

Leander: Right. These aren’t the first. [inaudible 00:25:41] for decades.

Anna-Katrina: They’re great brands. They know what they’re doing. They have smart people. they have tons of resources. I think the key takeaway is it’s really hard. It’s really difficult to build high quality product and these companies do not ship stuff that they don’t think meet the customer expectation. They’ll hold stuff. These product delays are really prevalent and I think the key point is it’s upsetting for the customers who are very excited. They’re going to buy a new iPhone, it doesn’t have a headphone jack, they got to use a dangle to be able to use their headphones, because their headphones are available yet. That’s upsetting to the customer. What’s likely more upsetting is the team that’s running that program, because they were likely going to ship when that product … Why would you ship a product two months later on purpose, right? That was probably not on purpose. Again, I don’t have any background details on this. But just logic would say.

Leander: Logic would say they run into a problem. Why would it be so late in the process? Because surely they must have gone through, like you said, those-

Anna-Katrina: Absolutely.

Leander: Engineering test validation builds.

Anna-Katrina: Absolutely. In the context of figuring out what product we needed to build at Instrumental, we interviewed hundreds of engineers across the industry. Large companies, small companies, multiple positions inside these large companies. The top two reasons for that delay that they have self reported are the late discovered issue and the amount of time it takes once you’ve discovered an issue to actually get to a root cause that can fix and validate that root cause and then be ready to ship again.

Leander: Wow, so that takes months.

Anna-Katrina: It can. You could imagine that you have something that only fails after 500 hours of [heat soap 00:27:22] testing, of thermal testing. 500 hours is about 20 days.

Leander: 20 days?

Anna-Katrina: I think so. Did I do the math right?

Leander: Don’t ask me.

Anna-Katrina: 500 hours is a really long time. You can do … Let’s say you realize you have this problem. You have confirmed that this is required to ship the product for the customer experience that you want, based on other data that you have. You need to pass this test. You can do a new configuration, a new experiment of like, “Okay, I’m going to build this type of unit, do this one change and let’s if these pass the 500 hours test.” You have to wait 500 hours to get the results of that test. Then once you do that, you then have to validate at volume that you didn’t break anything less with the fix. This could take weeks. I don’t know if that was the issue particularly with Apple AirPods. It could have been an issue like that, it could have been a software related issue. Often, a lot of these issues don’t become apparent until you get to volume and you don’t get to volume until you’re almost ready to ship. That’s that late discovered issue.

Leander: Do you have an example of that? How can you not discover when you get to volume? Isn’t the whole point of doing the engineering validation thing?

Anna-Katrina: Remember we talked about the [onesie, twosies 00:28:32]?

Leander: Mm-hmm (affirmative).

Anna-Katrina: Right now, the best tools that engineers have to find these issues are performance based tests, that they design and put on the line to measure the speaker and measure the microphone and measure the antenna performance. They have those tests and then they have measurements that they set up measurement plants for, that do geometries. Then, they have right place, right time on the line to spot it. Those are the tools they have at their disposal. Those only cover certain areas. It’s very easy for … You could have 20 antenna failures in the development build, like an early build. That’s a lot of failures. The antenna engineer will go through each one of them and will do failure analysis, process and disassemble the unit, try to come up with what the root cause for this failure. They might find that oh, this one in this particular unit, it appears that the operator has smashed the antenna and bent it. It’s not an antenna issue, it’s a process issue. What’s very common, this is actually a true story for one of our customers.

What’s very common is that that antenna engineer is like, “Okay, not my problem. Check off the box.” But the information doesn’t get transferred to the person who can actually make the change on the manufacturing line. But because they were using our product they noticed that there is this issue. Someone else was looking at the antennas and saw that this one was clearly bent. Yes, it failed the test, so it would never have been shipped, even if this was in production. But it also flagged in another way, which is just it’s visually off, it’s different. They were able to make a change, a fixture change on the line to prevent that type of damage. They never suffered future issues for bending of the antenna this way, because the antenna was protected by that fixture. This is exactly the type of issue that if you went into production and you’re ramping multiple lines and you have operators who have gone through operator training, but are still very new and green and haven’t done the operation many times yet, could cause that damage and you could have a huge bone pile. Or if you have-

Leander: Bone pile?

Anna-Katrina: A bone pile.

Leander: Okay. An industry term.

Anna-Katrina: This is an industry term for a pile of units that you won’t be able to ship, unless they’ve been repaired.

Leander: I see.

Anna-Katrina: It’s a bad thing. You do not want to bone pile. The size of the bone pile is often quantified in dollars. To keep in mind that every single one of these units is a problem in dollars.

Leander: Right. Okay. What kind of numbers are we talking about? These assembly lines. Foxconn employees, is it a million workers, assembly workers?

Anna-Katrina: I read that there is 800,000 people that work at Foxconn Guanlan, which is the fair labor association went and did an investigation there a couple of years ago. There’s a video online of what this facility looks like. These places are huge. They’re massive to do this type of scale.

Leander: How big?

Anna-Katrina: Foxconn Guanlan is called Foxconn city. These are self-contained cities that include not only the manufacturing lines, they include little stores and they include all sorts of restaurants. There’s a little operator dormitories.

Leander: I heard they consume 3,000 pigs a day. Something like that.

Anna-Katrina: Could be completely reasonable. It’s a lot of people. Then there’s all these … What’s very interesting about China … I spent a lot of time in China, over 300 days in the past five years or so. What’s very interesting is outside these factories, these little factory towns spring up, the predominant demographic of who’s working in these factories are 16-24 year old young people. What does a young person what to spend their money on? That’s what’s in these factory towns. There’s a lot of cell phone stores and company stores and clothing, like fast fashion kind of clothing stores. And karaoke clubs and stuff like that. But there’s not a lot else. It’s very interesting dichotomy in the type of culture that builds up around these large places. They are really large. They can sustain-

Leander: 800,000 kids, yeah. The factory itself, is it the size of big airports or-

Anna-Katrina: Bigger.

Leander: Bigger? The buildings themselves-

Anna-Katrina: Again, the standard line length across the industry for consumer electronic devices, I don’t know who came up with this, but it’s a 110 meters.

Leander: How long is that? That’s like longer than a football … How long is a football field?

Anna-Katrina: 100 yards.

Leander: Yeah, just a little bit loner than a football field, yeah?

Anna-Katrina: Yes.

Leander: That doesn’t seem very long actually, to be honest.

Anna-Katrina: These rooms will feet a 110 meter line and they will often have at least 20 feet on the other side of that. So they can get forklifts and stuff through. Then there would be 10, 20, 30 lines on the same floor, lined up next to each other.

Leander: All making the same product.

Anna-Katrina: At times they’re making the same product. If you have a product that’s shipping at volume. If you’re shipping a million a day, you’re having tens of lines, if not hundreds of lines for some of the components. Again, it’s not just the device, there’s all the pieces that go in the device that have to get made.

Leander: They’ll be making the screen and then they’ll be one line-

Anna-Katrina: Different vendors make different parts. But yeah, essentially they’re made … Anything that involved hand assembly is made on these types of lines. Sometimes there’s other manufacturers who use cell manufacturing. If you’re doing a very small sub-assembly, where there’s only four or five steps, you won’t use a 110 meter line, you’ll see a small little table area with people doing these operations. If you’re building something that’s very sensitive, like a display or a PCB, a printed circuit board, the main logic board inside of the phone, you’re often using highly robotized equipment. There’s very little people engaged in that process. Something like a display, you take the people out because you need a very clean process.

Leander: No dust, no contaminants.

Anna-Katrina: If there’s a spec of dust between you and your pixels, you’re going to see it in some conditions. In order to get clean enough environment, these are almost entirely robotized. Again, PCB assembly has been robotized for a long time. Actually very much automated. On a line like PCB assembly, it’s very common to get super high yields. 99.8% yield, very, very common to be able to achieve this where it’s actually incredibly difficult to be able to achieve yields like that on a complicated product in the actual final assembly of the product. There’s a human factor and humans do things not on purpose, of course, but humans make mistakes. The screwdriver is swinging on the line and it hits a part and now there’s a scratch on it. These thins happen.

Leander: Right, yeah. These factories, it’s a size of [inaudible 00:35:14] 800,000 people-

Anna-Katrina: Yeah and then multiple floors.

Leander: Okay. Oh, really? Okay.

Anna-Katrina: Yeah, multiple stories. What’s really interesting about Foxconn Guanlan is they actually have multiple … There’s the street level and then there’re sky walks on the second floor, because there’s so many people during a shift change that they needed additional … They needed to open up a second line for people to be able to get from building to building. It’s just a lot of human movement going on.

Leander: For something like a highly anticipated product that might be coming out in the fall and tens of million of people are going to be buying them on day one, when they have to start making these in order to fulfill that demand? [inaudible 00:35:55] you don’t have to be specific about that. What about Samsung for example?

Anna-Katrina: Let’s talk about the industry at large from a consumer electronic standpoint. Consumer electronic, the main market is the holiday shopping season. That’s the main market window for phone, for headphones, for watch, for some IOT home product-

Leander: Almost everything, yeah?

Anna-Katrina: The things that you would buy for others as gifts. You might have noticed that a lot of consumer electronic products come out in September or October, because they’re trying to capture that holiday market. The ones that are coming out in December, those are the late ones.

Leander: They’re the ones that have made a mistake and they-

Anna-Katrina: Something happened. Something happened. Maybe they didn’t make a mistake, but something happened. In order to hit this market you want to be shipping September first, the industry standard. You got three months then to sell. These are all peak months of sales. You’re ramping up your quantities to be able to at peak volume in that September, October timeframe. Then your volumes are expected usually to go down in January. People don’t buy a lot of consumer electronic devices in January, expect the ones that they wish they’d gotten as gifts perhaps. This, again, is very standard across the industry. This is not Apple specific. In order to be able to be shipping at peak volumes, you need to have ramped up to volume and when you start with one line, it’ll be very common for one line to only make … Again, this could be any type of product, but if there’s humans involved, humans can only move so fast. A very fast human operation could be nine seconds, which means your line is running at a certain number of units per hour. Maybe you’re producing between 3,000 and 5,000 units per shift on that one line.

If you need to be making a 100,000 units a day, you can do the math to figure out that you need to have 20 lines doing that. Or if you need to be making 20,000 units a day or 5,000 units a day, you only use one line. You don’t need to replicate lines. If you have to do that replication, all of those lines require operator training and bring up and validation. You don’t just turn it on and like, “Okay, it’s good. Everybody just do it now.” Because these operators have been built anything before. They need to train and understand how to actually build a product to create something at the yield that you got on that first line. That first line is called the Golden Line. That line is the control group for any … Any future line gets compared to that first line and its yields. It needs to meet a certain criteria before it’s allowed to make units for customers.

Leander: Excuse me, so for a big product that’ll be coming out at the end of September, that golden line, where would that be set up normally? In January or-

Anna-Katrina: Part of my job as a product design engineer or mechanical engineers in other companies is to make that golden line. That may also include operations, manufacturing, design folks, etc. as well are involved in the process of making that golden line. When I said that my responsibility was to demonstrate … As product design, their responsibility is to demonstrate mass production yields at mass production speeds on one line, that one line is the golden line. The golden line is the first line that’s set up to build the first prototype on day one of development builds. That line, you try to not change anything on it if you can. What’s tricky about that though is because there could be months or years in between when you start building on that line and when you actually get to production. You don’t want to change anything, I’m an engineer. Control all variables.

This operator quits, then what? You have to bring a new operator. That operator has no experience, now you’ve changed a variable. Or this operation is slower than we thought it is, so we need to add three more operators so that the whole line can run at nine seconds, meaning each station takes only nine seconds. These are the types of things that happen normally. You might make changes, because you have problems. That changes the line as well. The golden line is constantly evolving to a certain point in development and then it’s locked. Essentially, it’s golden, don’t touch it. Every new line has to be compared to that one. If you’re trying to validate something new, if you’re making a change somewhere else. Let’s say you need to bring up another tool for an injection molding tool, that’s a big part of the enclosure, that you would validate that new tool on the golden line, because you know line is good, you can check to see if new parts are good or not. It’s used in a very careful way.

Leander: I see. Yeah. The beginning of this process. Right. I see.

Anna-Katrina: It’s the control for the experiment. This is a process that was invented by engineers. Think about how engineers would think about it. It’s like scientific method, try to control everything and then use that as the control for future lines. The golden line has been around for a very long time in its current iteration when it’s ready to go, that’s the point at which it gets replicated. That’s the point at which you start what’s called ramp.

Leander: Ramp. Okay.

Anna-Katrina: Ramp is where you’re building ever more units every day. The way you build more units every day is you might start a line two-

Leander: You want to do that carefully I take it, to make sure that you don’t get in somehow a disaster.

Anna-Katrina: Engineers are in control of this process, so yes. You don’t want to just turn on the second line at full speed, because you have a whole line full of operators. Could be 100 operators, could be 400 operators sitting in various parts of this process. First day you might only build a few and actually train every single person down that line. That’s somebody’s job. The line leader does that. Then the next day you might build tens, and you might build hundreds. If the peak capacity of the line is 5,000, you built up to that, but you test, you take a break and actually test. Like, “Okay, what was the yield that we build on a line when we built a 100? Was the yield high enough to try to build 500?”

There are people’s job on the operations side who really get into the details on this and they have models that I don’t understand around how you actually tune the nobs to ramp responsibly, where you’re not going to create a huge bone pile of units you can’t ship to customers. But also to ramp responsibly such that … It would be horrible if you stood in line for six hours and there wasn’t a product for you to buy, because that would just not the right experience for customer, right? You also want to make sure that there’s enough volume available.

Leander: In general this ramp process for large volume product would be what kind of timeframe? A month? Two months? Three months?

Anna-Katrina: Depends on the manufacturing engineer who is running the process.

Leander: I knew you were going to say that.

Anna-Katrina: Honestly, it really depends on the product.

Leander: I’m trying to get a certain idea of just the scale of this operation. I’m sure there a couple-

Anna-Katrina: Companies that are trying to ship for the Christmas season should be considering starting in the summer is a peak time for them. They got to get their product done, ready to go into volume and they should be starting to build in volume, some time in the summer to be able to release in the fall, to be able to release their product, because you don’t ramp in a week, unless you’re only going to have one line. Then, you’re already ramped. As the end of your development you’ve demonstrated you can build at mass production speed, you’re only going to build one line, because that’s the volume for the product that you’re building. 500 day, totally fine. Then, you don’t need to ramp. You’ve already ramped. You essentially just run that at sustaining on the ongoing basis. Other product you have to actually replicate. Depending on the amount of lines you have to replicate could take longer, could take weeks, could take months.

Leander: What about these cell phones that sell in the millions?

Anna-Katrina: Million a day is a lot. Again, this is tens of lines, right? You can make an assumption that each line is making somewhere between 3,000 and 5,000 units a day. You can do two shifts, maybe you could do three shifts, depending on the factory that you’re working in. You’re building 10,000 on the line a day. You need to build a million, so you need to have a 100 lines.

Leander: Are those the right numbers?

Anna-Katrina: I hope so, because I’ll sound really dumb if they aren’t.

Leander: Good.

Anna-Katrina: Cut that part out.

Leander: All right. I’ll have to go back to figure it out. Just wondering.

Anna-Katrina: Yeah, you’re building 10,000 a day. You want to build-

Leander: Is that a million … What was it? A million a week?

Anna-Katrina: Some products ship in a million a week, some ship in the millions per day, some ship a 100,000 a day. There is a whole black magic that goes into figuring out how many products to ship for a given market. I am not an expert on that. Because you have to make predictions around what your customers want. It’s very expensive to be wrong, to overshoot.

Leander: Right. Yeah. I’ll bet.

Anna-Katrina: But it’s also expensive to undershoot. There’re people whose job it is to do these kinds of forecasts I guess. Once you have your target number of what you’re supposed to ramp to, there’s a whole team that then figures out what is the fastest and cost-effective way to get there. And when you have a deadline looming, if you’re a large company that can afford a bone pile of some size, you might decide to make the trade off to we’re going to hold the launch date, but the yields aren’t as great as we wanted. So we’re going to eat the difference, but we’re going to hold the launch date. Because they’ve done the opportunity cost to understand losing those sales for that day versus the fallout we’ll have or have to eat or repair. Somebody’s done that math.

Leander: Yeah.

Anna-Katrina: Sorry, I just got really technical.

Leander: It’s okay. No, I like it. I find it fascinating. Because it’s something that you don’t see. It’s the Willy Wonka’s chocolate factory, right?

Anna-Katrina: It’s all behind the scenes. What’s really fascinating is it’s pretty much the same everywhere. Even though it’s pretty much the same everywhere, nobody talks about it. But it is very interesting stuff. How did this product get built and get in front of me and that I use every day? How did that happen? It looks perfect, how did that happen?

Leander: How big are the teams inside these companies? Not at the factories. The product engineering team, the operations team, the design team. In general, is it dozens of people or hundreds of people?

Anna-Katrina: It depends on the product.

Leander: A big company like Apple.

Anna-Katrina: In a big company there could be hundreds of people working on a program. What’s really interesting about consumer electronic devices and hardware in general, I always say hardware is hard. The reason hardware is hard is because it’s software plus a whole extra piece, which is hardware. Then the integration between the two. It’s actually three times as hard. There’s two extra things you have to do, besides software. There’s the software team that’s developing new features, that take advantage of the new functionality that might be in the new product. There’s the actual teams who are doing the work on the operations side or the engineering side. Then there’s all sorts of support, as well outside of engineering. Marketing needs to be lined up, business operations, channel part … There’s so much involved in actually getting a product out at a large company. At a small company it’s really fascinating to see how they take these hundreds of people operations and really shrink them down. This is why small companies buy our software, is because they see that theirs going to get amplification of their small team.

Large companies buy the software because they see that they’re going to save lots of money, if they can cut even 12 hours out of a process or one day off their schedule. Or ship on time, but they would’ve normally shipped on September 2nd instead of September 1st. That’s whole day of peak sales they get. That’s millions or tens of millions or even hundreds of millions of dollars that they can get in revenue by having that peak day of sales. There’s interesting dichotomy. But when you go down to a small company, they might have one mechanical engineer. They might have one operations person who interfaces with the factory. One marketing person or an outsourced firm. [inaudible 00:48:17] software team of three or four. You can shrink this down into a 10 person team or into a 50 person team and you can build a great product. It would be very challenging to do that at volumes with a team that small.

Leander: Right. Yeah. Going back to what we were talking at the very beginning when you said there’s an opportunity now. Presumably, we were talking about automation, much more automation. How does your box fit into when things are much more automated? How does that work?

Anna-Katrina: I’m really glad you asked this question, because this really speaks to the larger vision of what we are working on as a company. There is this sea change coming to manufacturing where automation is going to replace people. This is happening and I know that there are people in our society who talk about that as a bad thing, but this war was won a long time ago with John Henry. Back then. It’s just taking … Each individual thing has a battle, but the war’s been won. This is actually very important because automation enables the next thing. What is the next thing? The next thing is intelligence. We want, as a country and as an individual myself, I want to be developing that next thing on an even playing field with everybody else who might also be developing that next thing. Automation is a great thing to have because it means that we can be developing the next step. The next step is intelligence and the reason that’s important is it enables us to essentially have smart, intelligent engines, AIs that run these manufacturing lines, that are automated already. Someone has automated them.

What is the brain that is running that line and that brain needs to be able to do smart things that currently people do. When you have the ability to enable the machines to do what they’re good at and the humans do what we are good at, then you can create something that is a lot more efficient to get a product out. You can build anywhere, it doesn’t matter. It’s the cost of the land that you put the factory on, it doesn’t matter if that land is near people with expertise, because the expertise is in the software. That’s the vision that we see and this inevitable flow from automation to intelligence. That’s the space where we’re working.

Back to the original question, then why do you have a box that goes online with people? The answer there is that we are not making the assumption that the things that have already been automated provide data that’s actually valuable yet. We actually created a system to go out and get data that we think is much more valuable. It’s not currently readily available. That data enables us to build the first pieces of this analysis engine, this intelligence engine.

Leander: Yeah. [crosstalk 00:51:01] machine learning.

Anna-Katrina: Yes. With machine learning. These machine learning features that we have is just pulling back the corner, pealing back the corner of the drape on this whole intelligence engine that we’re building for manufacturing and demonstrating what is the power of this technology already and reaching towards the potential of what it could be in the future.

Leander: Right. Okay, yeah. Building intelligent automated-

Anna-Katrina: We built a box. It goes on lines where there people working or on transition lines where there’s people in machines. But it collets data that’s not currently available, that’s why we have to build the box.

Leander: Do you give thought to the social issues, all the practical issues, obviously. This is a hot button topic right now, given the election. Do you see manufacturing coming back to United States? Human manufacturing? Then the second question, which is related to that is if you take away those jobs, does it create new jobs, different jobs to replace those jobs?

Anna-Katrina: Yeah. I respect the multiple political views on this issue, because I think this is actually a very complicated issue. I think that stopping the flow of technology advancement in manufacturing would be a mistake.

Leander: And impossible, right?

Anna-Katrina: It’s inevitable at this point that automation will happen. There may be some operations that are done by people for a long time to come, but for operations that machines can do, we should let machines do them. My perspective on this is the reason we need to do that is so that we can be working on the next step. We can’s start working on the next step … We need to build a piece of hardware to start working on that next step, because the robots don’t exist. If the robots existed, we could just use the data from the robots. We had to build a piece of something to actually get to the next step. We want to be, I think, as a country, as a society, as a world, working on technology that saves resources, whether that’s money, time, materials and raw materials. We should be working on that.

Intelligence is this great opportunity and it’s also really powerful. I think the advent of intelligence in manufacturing is really what we should be thinking of. You can give a man a [inaudible 00:53:19] or you can give him a spoon and they’ll both be able to dig a hole. We would think that that would be ludicrous today. We should be embracing these technological advances and I believe that there’s a lot of work to be done on what happens then in society. I have to believe that every other technological advance in the past, like the advent of the company put a lot of typists out of work, but look at all of the jobs that exist now because of computers that would not have existed, if we were still using type writers. I guess as an optimist internally, I have to believe that that’s something that we’ll figure out.

Leander: Having worked on those line, I’ve worked in factories, those are not angelable jobs. You wouldn’t, I presume, would not be sad to see those assembly jobs go away.

Anna-Katrina: Certainly, some people will be sad to see those jobs go away. I think it’s important to keep in mind that there’s a very human factor here. But I also think it’s important to keep in mind that there’s a global society factor that artificially keeping ourselves behind is putting us behind at a much larger scale for what’s next. I think we have to weight those risks together and come up with solutions together. I think that there’s a lot of people that talk about the problems in this space, and not a lot of people working on or actually implementing solutions here. This is an area of focus that I wish more time was spent on.

Leander: The last question would be how many other people are working in this space? Are you alone or do you have competitors?

Anna-Katrina: This is a very interesting space in that it’s just starting to crack open. There are many large companies and large enterprises who are working on some of the ideas of what we are interested in, whether that’s on the appliance side, where they’re actually building the equipment itself, that does this automation or does measurement and creating this advanced technology or whether it’s on this manufacturing data site. There are a lot of companies that are plugging in to the machines that are available and they can take the data that’s on the machine and put that in the office of the line leader and they can review all the data and perhaps be able to understand that there could be insights, could be generated from that. We take a little bit of different approach in that we think the power here is the combination of data with insight. Data just tells you have a problem. We have a failure rate, the failure rate is this. Insight tells you what to do about that problem. Insight is something that intelligence comes up with, that’s something that people do today.

Honestly, I was a product design engineer, I was a pretty darn good one, I think most people who worked with me would say. But I was limited, because my head is full of meat. I can’t remember everything I’ve ever seen, I don’t remember the serial numbers, I don’t have a catalog of this data in my head. But computers can do that really easily. I think there’s this opportunity to let humans do what we do best and let machines do what they do best and leverage those two together. Machines can crunch lots of data that enables the creation of insights and that’s what we’re working on is building that intelligence. There are companies that have touched on various areas, but we think we are unique in this particular space in the market.

There are big players who are working on many problems in manufacturing, including where all this data goes, what collets it, the contract manufacturers themselves are building a lot of equipment and technology themselves. It’s an area of intense, intense development. I suspect there’s 10 companies that haven’t come out of stealth yet in the US and 10 more in China and 10 more in India. That this is an area that is growing and going to become ever more prevalent in the years to come.

Leander: Okay. There are other companies like you, start-ups.

Anna-Katrina: Probably, yeah.

Leander: [crosstalk 00:57:14].

Anna-Katrina: As a founder, it’s my job to be paranoid about everybody else. We don’t make the assumption that we’re the only ones working on this. That being said, we’ve got a great start. We have Fortune 500 companies that use our equipment and software, who have come back for repeat business. We have companies that are expanding their contracts with us. We’re currently deployed in multiple Foxconns and Flextronicses and multiple countries, three countries at this time. We’ve got a good initial reach into this market. But it’s just the beginning. We’re not cocky enough to believe that we’re alone.

Leander: You just read a lot about AI machine learning. I think this is a really good example of how it is becoming integrated into almost every aspect of our lives. The reason I ask about this sort of competition is it’s not just you who had this idea, this is a general trend that is going on across the industry.

Anna-Katrina: Machine learning is one of those terms that has become very buzz wordy. As a company we try to stay away from these buzz words as much as possible, but it is the technology that we use and leverage. Really the opportunity is that we are taking technology that’s been developed and evolved in the consumer space for things like Netflix, providing recommendations and in a restaurant. All that recommendation engine stuff is an early application for machine learning. Fraud detection on comments to push troll comments down. That is often done by machine learning. Fraud detection that pushes troll comments down is the same as anomaly detection, which is what our software does. Ours is in the photographic space, that’s in the natural language processing space.

Taking these technologies that have been developed in consumer and taking those and bringing them to an industry that honestly, get overlooked a lot. It’s very unsexy space. I think it’s really awesome, but it’s generally unsexy. This creates an opportunity between the two. That being said, one of the interesting questions I’m often asked is how do we come up with this idea to do this particular thing? Is it truly that unique. What we’ve always done as engineers is we want to get out and we want to talk to the people we’re going to build a product for. After we left Apple, I spent several months … We thought we were going to build robots, actually. We started as a robot company. We’re two mechanical engineers. Of course, we’d start a robot company.

Leander: Robot company, yeah.

Anna-Katrina:    But what we realized when we went and we talked to at least a 100 engineers, companies of varying sizes, and also factories, not just engineers, but also factories. What they wanted and what they saw. We realized that the robots were less interesting than the data and that there’s not a lot of software power being put in the manufacturing space. Even though there were engineers who could tell us, “Oh, man. You’re building that thing, I’ve always wanted that thing. I’ve always dreamed of that thing. The ability to see these images from anywhere, that would be awesome. I wanted that. I thought of that a year ago.”

This is a very common reaction but nobody’s actually done it. I think the unique advantage that we have as a company is that we’re building software for mechanical engineers, which doesn’t happen very often. And we’re mechanical engineers to start. We’re building software for real problems that we have faced and that this slew of engineers we’ve talked to have also faced. That’s enabled us to tune what we built. We very quickly scrapped the idea of building robots. Instead, focused on this power of data and software. Yes, two mechanical engineers have founded a software company. Doesn’t happen that often, but again, I think that’s what makes us special.

Leander: Okay. Very cool. Thanks so much and I wish you the best of luck.

Anna-Katrina: Thank you.

Leander: That’s all we have time for this week. I’d like to thank Anna-Katrina Shedletsky, the CEO and founder of Instrumental. You can find more information about Instrumental at Instrumental.com. You can also check out cultofmac.com. We have a couple of posts this week about Anna-Katrina and her company. That was Candy’s Corner, a weekly podcast about the world of Apple. New episodes come out every week. Please subscribe on iTunes or your favorite podcasting app. If you like the show, leave a review or rating, it helps a lot. Please check out cultofmac.com and follow us on Twitter or Facebook. On Twitter we’re @cultofmac. And Facebook is facebook.com/cultofmac. See you next time.