Seeing with AI: Flexible navigation in dynamic environments

ABB’s recent acquisition of Swiss start-up Sevensense Robotics AG, a leading provider of AI-enabled 3D vision navigation technology for autonomous mobile robots (AMRs), is set to transform the fields of logistics, manufacturing, and service robotics.

Subscribe to ABB innovation and technology news

_{Renaud Dubé, ABB Robotics – Sevensense, Zurich, Switzerland, renaud.dube@ch.abb.com
Marcin Dymczyk, ABB Robotics – Sevensense, Zurich, Switzerland, marcin.dymczyk@ch.abb.com}

Automated materials handling and transport in logistics and manufacturing centers, as well as in major retail facilities, offers vast benefits, including increased efficiency, profitability, safety, and flexibility in terms of labor fluctuations. Nevertheless, the vast majority of tasks and processes that could benefit from mobile robotics are still executed manually. Indeed, according to software marketplace specialist g2.com [1], in 2021, only six percent of warehouses employed mobile robots. And not much has changed since then. However, the trend in many industries toward mass customization, in other words, producing smaller lots of greater variety in shorter product life cycles, is affecting manufacturing as well as warehousing and logistics operations, which calls for increased use of flexible robotics.

This low adoption rate can ultimately be linked to the fact that most mobile robots sold today rely on expensive, legacy fixed-floor installations or 2D laser scanners – devices that can perceive the environment in only a narrow slice – as if looking through a mailbox slit. To know its location and how to navigate to its destination, a vehicle that depends on such outdated technologies requires a structured, static environment and can perform only very simple, precisely defined tasks – the opposite of the way modern warehouses, production plants and big box stores operate.

Automation solutions based on camera vision and powerful AI models overcome these limitations, as they offer a much richer and more intelligent perception of the environment. Automated vehicles equipped with such capabilities can easily navigate dynamic environments, execute complex tasks, and work in unstructured spaces shared with people »01. This technological progression is needed to bring autonomy to warehouses and to all other industries that depend on manual labor today. 

Visual AI for localization and navigation

The solution that makes the above-mentioned capabilities possible is called Visual Simultaneous Localization and Mapping (Visual SLAM). Using cameras, cutting-edge computer vision algorithms and AI models to perceive their surroundings, Visual SLAM-equipped robots build rich 3D maps of the environment to precisely localize themselves within it. Visual SLAM’s advantages are:

• No costly infrastructure installations required; the environment is perceived as it is and in real-time.
• Functions in dynamic environments such as warehouses where the layout may change frequently and where people and objects are constantly moving around.
• Extremely robust: perceives visual characteristics of surrounding such as light intensity, contrast, and shapes, thus avoiding confusion and allowing mobile systems to work alongside people and moving objects.
• Allows robots to navigate on ramps and uneven floors. Moreover, the number of mapped environment elements is so high that parts of the environment can be changed without degrading localization quality.
• Finally, observing the environment in 3D at all heights enables robots of different kinds to collaborate, exchange data and use the same map of the environment to localize, thus forming swarm intelligence »02.

Naturally, these advantages add up to considerable market potential. For instance, according to DHL Logistics Trend Radar [2], sales of mobile robots in the logistics industry are growing at a rate of 31 percent per year. This growth is facilitated by a technology shift from 2D laser scanner autonomy to 3D Visual SLAM autonomy, which is significantly advancing mobile robots in various applications and boosting their rate of adoption »03.

The Sevensense Visual AI offering has led this trend with two products, which are now included in ABB’s portfolio: Alphasense Position, the Visual SLAM positioning system for autonomous mobile robots, and Alphasense Autonomy, a Visual SLAM-based solution for autonomous navigation »04. The product is a plug-and-play kit designed to make any robot autonomous. The system’s hardware consists of an AI-enabled compute unit that runs proprietary algorithms and a set of cameras »07 that perceive the environment. Tight integration of the hardware and software enables optimal performance of the underlying software and AI models. It also allows customers themselves to efficiently develop highly reliable, high-performance robots powered by the technology and to add a range of paid services. This visual AI technology has also been integrated with ABB’s autonomous mobile robots and the company’s AMR Studio suite software. Users are already using Visual SLAM-powered Flexley Tug AMRs in their daily operations.

Paid services associated with Alphasense products

How vision-based positioning works

Alphasense Position software and hardware reliably estimate the 3D position, orientation and velocity of mobile robots. Thanks to sophisticated computer vision algorithms and AI models, Alphasense Position can reliably and very precisely (up to 5mm) estimate its own position even under the toughest conditions. The system meets or exceeds specifications on ramps and non-flat surfaces, in multi-floor environments, and even if some of the cameras are temporarily covered. It works both in- and outdoors, and in environments with very poor lighting conditions. Its built-in AI algorithms never stop learning. With each hour of operation, they leverage new experiences to update 3D maps of the environment, thus reflecting changes in the operational space and further enhancing the system’s long-term robustness.

One of the core benefits of Alphasense Position is its easy and quick setup procedure »06. For instance, the process of mapping – a precondition for positioning – is as easy as pressing a button and guiding the robot around. The resulting reduced installation time and skill set requirements shrink costs and lower barriers for customers. In ABB AMRs, Alphasense Position is tightly integrated into the AMR Studio user workflow, providing a single, user-centered interface to set up the robots, including the Visual SLAM system.

Alphasense Position serves use cases ranging from single robots working on their own, such as a scrubber-drier robot in a medium-sized grocery store, up to fleets of hundreds of mobile robots of different types in large manufacturing plants. In the latter case, units can intelligently interact and learn from each other by building a joint 3D visual map that leverages AI models and the latest data collected from the environment.

Alphasense Position works with any type of wheeled ground robot. Furthermore, Sevensense offers prototyping kits that enable customers to simply run VSLAM within hours on their own mobile vehicles.

A complete solution for autonomous navigation

Alphasense Autonomy provides mobile robotics platforms for materials handling, manufacturing, professional cleaning, and other service robotics applications with complete navigation and obstacle avoidance capabilities. By building on the positioning output of Alphasense Position and including additional information from off-the-shelf obstacle detection sensors such as 3D depth cameras, 2D laser scanners, and ultrasonic sensors, Alphasense Autonomy efficiently drives robotic vehicles while circumventing obstructions.

One key differentiator of Alphasense Autonomy is its ability to cope with challenging and unstructured environments, such as busy warehouses or crowded airports. The unique AI-based perception capabilities developed by Sevensense make mobile machines completely autonomous, without the need for costly human-in-the-loop interventions. Extra care has been taken to ensure that vehicles equipped with Alphasense Autonomy follow smooth and intuitive trajectories, allowing people working next to robots to feel safe and comfortable. Furthermore, the safety and reliability of the system is enhanced by a cutting-edge obstacle perception module that sees the world in 3D.

As is the case for Alphasense Position, the installation of Alphasense Autonomy is geared to minimizing setup time. Robots can plan their best and most efficient paths using only an onboard Edge AI. They can operate as single entities or, as members of a fleet, can share map and traffic information with each other. Additionally, Alphasense Autonomy accepts orders and commands from 3^rd party fleet management systems (FMS) using modern and accepted standards such as VDA5050. These developments ensure that Alphasense Autonomy is fully interoperable and can work with major FMS, including in heterogeneous fleets and with robots from other providers.

Alphasense Autonomy is used by a diverse group of customers with different levels of know-how in automation – either to upgrade existing navigation systems that have limited capabilities or to equip manual machines with autonomous capabilities. The product is available for purchase and has been validated and integrated by customers globally. For instance, Wetrok AG, a Swiss manufacturer of professional cleaning equipment, is selling its Alphasense Autonomy-enabled cleaning robots to customers around the world. Furthermore, several key OEMs and integrators in the materials handling industry are considering whether to adopt it. 

Fusing information to maximize efficiency and safety

ABB’s Sevensense technology is based on several unique components. Chief among these is a proprietary system that can orchestrate up to eight cameras and perform precise synchronization, automatic gain control, and camera exposure control to maximize the information content of images acquired by the cameras, regardless of lighting conditions. Based on a capable yet cost-efficient system-on-module (SOM) »07, this service enables Alphasense Position to run Visual AI workflows directly on the unit, with minimal latency or computational burden while benefiting from precisely tuned camera image signal processing (ISP). The associated proprietary hardware design, which addresses the technological challenge of transmitting high-quality synchronized camera imagery over long wiring, allows OEMs to freely and seamlessly position up to eight cameras anywhere within the chassis of their vehicles, thus enabling 360° surround view coverage. This seamless integration and robust view are unique in the market and significantly increase the performance of Visual AI localization and navigation.

Furthermore, Sevensense Visual SLAM technology comprises AI-powered proprietary sensor fusion algorithms to enable robots to precisely estimate their position. This is achieved by extracting discriminative elements from images, such as, for example, the corner of a window, and using these elements to build a representative yet compact 3D model of an environment. To achieve a high level of reliability and enable operations regardless of lighting conditions or perspective, detection and association of discriminative elements is made using AI-based models that run on fast on-board graphic processing units (GPUs). An example of a challenging scenario is shown in »08.

In order to effectively use the above-mentioned sensor types, robotics systems, including Sevensense products, require accurate calibration. The processes underlying calibration are complex yet essential in achieving both robot positioning and autonomous navigation. Calibration makes it possible to identify internal camera parameters, such as focal length and distortion of lenses, as well as the spatial transformations between different sensors, for instance inertial measurement units, ultrasonic sensors and time-of-flight cameras.

While complex, all of this can be easily done when mobile robots roll off the production line, thanks to Sevensense’s novel and proprietary methodology that automatically optimizes all calibration parameters at once. As a result, factory calibration can be performed by production personnel using only the robot itself, without requiring costly measurement instruments or iterative manual adjustments.

People first

Naturally, all the above-mentioned capabilities not only add up to precise and accurate interpretations of environments but also to a high level of safety for people. Sevensense computer vision techniques leverage the rich information content of visual images in order to obtain not only 3D distance information, but also much richer scene understanding, such as the detection of people and prediction of their relative motions. Through the combination of AI-based detection methods with 3D geometrical rules, robots equipped with Sevensense technology can detect and track people using cameras all around a vehicle. This crucial scene-based information enables the technology to adapt vehicle behavior accordingly, for example, by allowing vehicles to swiftly evade static obstacles that are blocking their path but drive more cautiously in the presence of people or come to a complete halt in front of people »09. Indeed, Sevensense technology is the first complete solution that uses multiple stereo cameras for detecting and tracking people to ensure safe and human-friendly navigation in crowded industrial environments. 

Lifelong Visual SLAM

One challenge for any type of positioning system is that environments change over time. This may be due to changing seasons or lighting conditions, changes in factory floor layouts, or simply in warehouse or shop inventories. Without smart algorithms, it is impossible to safely deploy mobile robots in dynamic environments. To master this challenge, ABB’s Sevensense offers a smart AI-based algorithm that enables building and maintaining always-up-to-date maps of environments »10. These maps incorporate data from multiple conditions, such as during cloudy and sunny weather, bright and dark lighting situations, and changes in the elements present in an area. These changes are detected automatically and swiftly incorporated into a lifelong map without any intervention.

All in all, Sevensense’s Visual AI technology enables more reliable mapping, particularly in dynamic and structurally repetitive environments. It also allows for a cost advantage over laser scanners, due to lower sensor hardware costs. A thorough comparison with alternative sensor technologies is provided in »11. 

The path to human-like intelligence

As the number of mobile robots in production centers, warehouses, logistics centers and retail facilities grows, complexity will increase. But to efficiently manage this complexity, mobile robots will need to be even simpler to setup and operate than they are today. Ideally, as they become increasingly capable of harnessing generative AI and large language models, they will be able to set up and optimize their configurations and interactions on their own. At that point, based on a few simple instructions from human operators, they will be able to autonomously explore the available space and plan the paths and flows of materials through that space. This revolutionary shift will increase the overall robustness and throughput of operations – and will likely generate unprecedented additional customer value. 

A rapidly changing service world

The evolution of QCS service from manual to digitally enhanced methods promises significant improvements in maintenance efficiency and system reliability. By embracing digital tools and ML, QCS field service teams can transition from reactive to proactive maintenance strategies, achieving better performance and reducing downtime in paper mills. Data security, customer readiness and alarm rationalization challenges can be addressed through thoughtful implementation and continuous improvement, ensuring a successful digital transformation. 

References
[1] g2. 50+ Warehouse Automation Statistics to Streamline Operations. Available: https://www.g2.com/articles/warehouse-automation-statistics [Accessed January 6, 2025.]
[2] DHL. Indoor Mobile Robots, Trend Overview. Available: https://www.dhl.com/de-en/home/innovation-in-logistics/logistics-trend-radar/amr-logistics.html  [Accessed January 6, 2025.]

Explore ABB Review