In March, Matt Bratlien saw something odd in the spacious suburb of Silver Firs, north of Seattle. A six-wheeled robot with the Amazon Prime logo on its sky-blue carapace was driving up and down the sidewalks and curbs, watched by a company representative. I was surprised, excited, and very curious, says Bratlien, a partner at Net-Tech, an IT services company in nearby Bellevue.
Bratlien had encountered Scout, a delivery robot Amazon is testing in the area, including by ferrying real orders to customers. Heres what he didnt see: Countless digital clones crawling through a virtual copy of the neighborhood that Amazon created with scans of the area collected by lasers, cameras, and aircraft.
Amazon knows a lot about the world thanks to data from its vast retail business and cloud computing platform. It knows a 2-square-kilometer zone of Snohomish County in unusual detaildown to the position of weeds sprouting through the drainage grates. The companys digital copy mirrors the position of curbstones and driveways within centimeters, and textures like the grain of asphalt within millimeters.
That synthetic suburb allows Amazon to test Scout thousands or perhaps millions of times under varying weather conditions without swarming the neighborhood with bright blue rovers until they become a nuisance. The bots can run 24/7 in simulation, says Sean Scott, the executive leading the project. The practice resembles how Waymo and others working on autonomous cars invest heavily in simulations to supplement miles driven on real roads.
Amazon wants Scout to help it deliver more packages, more promptly. The company recently said it would offer US members of its Prime service free next-day delivery. In January, Amazon said it was testing six of the robots somewhere in Snohomish County. It now says the fleet has grown, but declines to say exactly where they roam. County Executive Dave Somers says Amazon consulted him and the sheriffs office prior to launching, and says he supports the project but doesnt know exactly where Amazon tests.
Bratliens Facebook post geotagged Scout in Silver Firs, a community of 21,000 with curving streets branched with cul-de-sacs; another person posted a photo of Scout in the same neighborhood. Months after Amazon announced Scout was in testing, Washington Governor Jay Inslee signed a bill that regulates delivery robots, limiting their speed and weight and barring them from jaywalking.
Matt Bratlien posted this image on Facebook after encountering an Amazon robot in March.
Amazons mapping and simulation technology is not just a research tool. It could also help Amazon deploy the robots to new neighborhoods when theyre ready for general use, by first testing them in simulations. We've built it such that we can scale up to an entire city, Scott says. By the time Scout rolls into a new town for the first time, its control system will have likely seen every seam in the pavement thousands of times before.
Amazon is a relative latecomer to the young world of delivery robots. Startup Starship Technologies, founded by two creators of Skype, and competitor Marble were delivering pizza and other food orders in early 2017. Amazons project appears to have begun in earnest late that year when it acquired Dispatch and the startups three cofounders joined Amazon.
None of these companies has yet proven that delivery robots can be reliable or profitable at broad scale. Most, like Amazon, deploy their robots with human chaperones who take control in case of problems; Amazons attendants also unload packages at customers homes. Kiwi, which delivers food to students at UC Berkeley, dispatches robots unaccompanied but pays remote workers in Colombia to steer them over the internet.
Amazons size and deep investment in logistics set the companys project apart. Scout joins a delivery fleet that includes 40 aircraft and 30,000 delivery vans. The company has deep experience with robotics; it employs more than 200,000 of them inside its retail operations to move shelves, load pallets, and sort packages, among other tasks.
Yet to drive safely around Silver Firs or any other neighborhood, Scout must cope with challenges not seen inside Amazons finely controlled warehouses. On sidewalks, Scott says, the robots are programmed to slow down and steer around people or animalsor stop if they come close. But making Scout reliably self-sufficient requires gathering as much data as possible.
Amazon built its simulated suburb in part with data from a cart similar in size to Scout that was towed behind a bicycle, capturing images using cameras and lidar, a type of 3D laser scanner used on autonomous car projects. The company filled in its map with 3D data from aircraft surveys. More recently, Amazon has experimented with a more powerful lidar too large for the bicycle cart, although Scott declines to identify how it travels.
At a glance, views from inside Amazons simulation can be hard to distinguish from photos of Silver Firs. Look more closely and you notice glitches like smeary foliage. All the same, Scott says his team has evidence that their fake world is accurate enough that algorithms trained on it also work on real world data.
Scouts control system includes computer vision software that labels pixels in images from the robots camera as grass, sidewalk, road, or other features. A version of that software trained on 400,000 hand-labeled photos of the neighborhood scored 0.98 on an internal accuracy test with a maximum score of 1. A version trained using only images from the virtual world scored 0.94. Scott says thats close enough to show that the simulator can help Scout understand the real world; progress doesnt depend entirely on collecting and labeling real photos. He predicts even higher accuracy from training with a combination of real and simulated data.
One of the robots trickier challenges is navigating curb ramps when it needs to cross the street. Collecting data solely by repeatedly driving the ramps would be time consuming, and potentially annoying to residents. Amazon engineers can minimize that by sending virtual Scouts through curb ramps at varied trajectories from different perspectives. It keeps us off the roads, Scott says.
Kevin Peterson, cofounder and CEO of Marble, which is testing delivery robots in Concord, California, says his company has experimented with high-resolution simulations but found they werent useful enough. Instead, he says, Marble has used a technique called generative adversarial networksalso used in AI art projectsto create extra data to train its robots software.
Amazon could use its simulations beyond training Scouts vision systems. A hot area in robotics research involves training control systems using a technique called reinforcement learning. Algorithms experiment many times inside a simulated world to learn a task such as how to manipulate objects through trial and error. Amazon could gain a significant advantage if it could make that approach work for its robots, says Peterson. If had to make a bet thats where Id say theyre going, he says. Scott says only that his team is exploring using learning algorithms more broadly for Scouts control.
Some unknowns about Amazons robotics project are questions of economics, not engineering. Scott says his team is aiming to make Scout 100 percent autonomousbut can also imagine robots that sometimes call home for help. Someone could manage hundreds of bots, he says. The robot-to-human ratio that makes Scout a practical delivery option will depend on how often it needs help, and the financial costs and payoffs of robot package hauling.
Scott says Amazon is focused on Scout for suburban deliveries for now, although Peterson of Marble says the economics are best in denser urban areas where drop offs are closer together. Amazon says Scout can travel miles at a time on a single charge, depending on the terrain and its loads.
Bratlien, who unexpectedly met Scout on the streets of Silver Firs in March, was prompted to think about how such robots might help his IT business. Net-Tech currently uses courier services to rush laptops to customers who need them right away. This could augment that, Bratlien says.