Northrop Grumman, Raytheon move forward in developing electro-optical sensors with built-in machine learning

Two U.S. prime defense systems integrators are moving forward with a military research project to develop a new kind of camera and digital signal processing to enable intelligent electro-optical sensors for tactical military applications.

Officials of the U.S. Defense Advanced Research Projects Agency (DARPA) in Arlington, Va., awarded orders collectively worth $25 million to the Northrop Grumman Mission Systems segment in Linthicum Heights, Md., and to the Raytheon Intelligence & Space segment in El Segundo, Calif., for the second phase of the Fast Event-based Neuromorphic Camera and Electronics (FENCE) program.

DARPA FENCE seeks to develop and demonstrate a low-latency, low-power, event-based camera and a new class of digital signal processing and machine learning algorithms that use combined spatial and temporal information to enable intelligent sensors for tactical military applications.

In June 2021 Northrop Grumman won a $15.8 million contract and Raytheon won an $8.8 million contract for the first phase of the FENCE program.

Neuromorphic describes silicon circuits that mimic brain operation; it exhibits low latency, sparse output, and extreme energy efficiency. Neuromorphic cameras offer sparse output, and respond only to changes in the scene, with accompanying low latency and low power for small-format cameras in sparse scenes.

Event-based imaging sensors operate asynchronously, and only transmit data from pixels that have changed, so they produce 100 times less data in sparse scenes than traditional focal plane arrays (FPAs). This leads to 100x lower latency at 100x lower power.

Despite their inherent advantages, existing event-based cameras are not compatible with military applications because military images are cluttered and dynamic. The FENCE program seeks to develop an integrated event-based infrared focal plan array with embedded processing to overcome these challenges.

The FENCE program’s primary focus is on developing an asynchronous read-out integrated circuit (ROIC) capable of very low latency and power operation, and a new, low-latency event-based infrared sensor with in-pixel processing.

The project also will develop a low-power processing layer that integrates with the ROIC to identify relevant spatial and temporal signals. The ROIC and the processing layer together will enable an integrated FENCE sensor that can operate on less power than 1.5 Watts.