An amazing new augmented reality demo shows how the future of cut, copy and paste might work.
Created by Cyril Diagne, a 34-year-old artist, designer and programmer currently in residence at the Google Arts & Culture Lab in Paris, the demo shows images of objects being dragged from the real world and dropped into Photoshop on a Mac. That’s an AR application that any designer could get on board with!
4/10 – Cut & paste your surroundings to Photoshop
Code: https://t.co/cVddH3u3ik
Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF— Cyril Diagne (@cyrildiagne) May 3, 2020
The AR copy and paste demonstration takes advantage of some nifty machine learning advances. As Diagne writes on Twitter, the “secret sauce” is BASNet, a tool designed for salient object detection and background removal. Like Photoshop’s Magnetic Lasso, the software detects the edges of a physical object and then frees it from its surroundings. Diagne shared plenty more details on a dedicated Github page.
“This is a prototype that uses the phone camera to capture, extract and paste objects from your surroundings directly to photoshop,” Diagne told Cult of Mac. “Thanks to OpenCV SIFT, an image-tracking technology, the app is able to detect where the phone is pointing at the screen, making it a seamless experience. It’s part of a series of experiments I’m doing every weekend to explore how machine learning and AI can help create more digital interactions that are more natural. Basically using the algorithm’s ‘intelligence’ to remove the layers of abstractions we had to build in digital interfaces up to now.”
Coming soon to a phone near you?
In the demo, posted Sunday, Diagne carried out the copy and pasting using an Android phone and a MacBook Pro. Diagne said that the demonstration was not created using ARKit or Google’s ARCore, although doing so in the future could “unlock some very interesting scenarios.”
Diagne said he initially did not think about developing his AR tool any further. However, the response to the demo is making him rethink this. “Seeing the response and enthusiasm from creatives across disciplines, I think it would make sense to develop the feature into a mobile app,” Diagne said. “Technically speaking, there are some much-needed improvements that need to be done — but nothing infeasible.”
Right now, this is all still proof-of-concept stuff. While interested users can get it working as a demo, don’t expect it to work seamlessly. (You’ll also need a bit of coding knowledge to get it up and running.) As far as scintillating demos of AR go, though? This is pretty awesome. No wonder Tim Cook is so excited about augmented reality!
