VentureBeat Q&A: Hailo CEO Orr Danon says edge AI means ‘stream the insights,’ not the video

Developments in AI chip technology are coming rapidly of late, with reports of new processors from Google and Microsoft suggesting that Nvidia GPUs’ dominance of AI in the data center may not be total. 

Outside the data center new AI processing alternatives are appearing as well. This latter battle is marked by a group of embedded-AI chip makers taking novel approaches that conserve power while handling AI inference — a must on the boundaries of the Internet of Things (IoT).

Count Hailo among these chipmakers. The company endorses a non-Von Neumann data flow architecture suited for deep learning on the edge. Its chip combines a DSP, a CPU and an AI accelerator to do its work, Hailo CEO Orr Danon recently told VentureBeat. 

The company’s latest offering, the Hailo-15, can be embedded in a camera, can target massive camera deployments, and can offload the expensive work of cloud vision analytics, while conserving power. Behind this is the thought that it’s not helpful to push this kind of work to the cloud — not if the IoT is to make progress. (Editor’s note: This interview has been edited for length and clarity.)

VentureBeat: Nvidia certainly has become a preeminent player in the world of AI. How do you measure your efforts with edge AI using data flow ICs, as compared to Nvidia’s GPU efforts?

Orr Danon: To be clear, Nvidia’s main focus is on the server and the data center — this is not what we are optimizing for. Instead, we focus on the embedded space. Nvidia does have offerings there that are, to a large extent, derivatives of the data center products, and therefore are targeting very high performance and accordingly higher power consumption, and higher price, but extremely capable. For example, their next product target, I think, runs at 2 petaFLOPS on an embedded form factor. 

VB: Of course, they don’t quite look like chips anymore. They look like full-scale printed-circuit boards or modules. 

Danon: And that is of course valid. We’re taking a bit of a different approach: optimizing for power, looking at the embedded space. And that’s, I think, a bit of a differentiation. 

Of course, one of the big benefits of working with Nvidia is working with the Nvidia GPU ecosystem. But even if you don’t need it, you gain its overhead anyway. If you scale up it works okay, but especially when you try to scale down it doesn’t work very efficiently. That’s our space, which I think is a bit less of an interest to Nvidia, which is looking at the very big deployments in data centers.

Computer vision meets edge AI

VB: Still, the new Hailo chips have a lot to do. They can be embedded in cameras. It begins with the incoming video signal, right?

Danon: We have multiple processing domains. One of them is the physical interface to the imaging sensor that handles the auto exposure, auto white balance — everything that is classic image processing. 

Then, on top of that, there’s video encoding — and on top of that we have a heterogenous compute stack based on a CPU which we license from ARM that does the data analytics and the management of data processing. On top of that is a digital signal processor, which is more capable than the CPU for more specialized operations. And the heavy lifting is done by our neural net core.

Here the idea is that the processing of the neural network is not being done in a control flow manner, meaning executing step by step, but rather it is distributing processing over the neural network accelerator that we have inside the SOC [System on Chip].

Different parts of the accelerator are taking on different parts of the compute graph and flowing the data between them. That’s why we call it data flow. This has a major implication in terms of efficiency. The power consumption is going to be dramatically low, compared to the level of compute performance that you’re getting.

The internet of things with eyes

VB: The Internet of Things seems to be evolving into some individual markets, and a specialty there seems to be this vision processing.

Danon: I would call it “the IoTwE” — the Internet of Things with Eyes — things that are looking into the world. When you look at IoT, there’s no point in it if it’s just broadcasting or streaming everything that you have to some centralized location. That’s just pushing the problem to another space, and that’s not scalable. That’s very, very expensive. 

You know, the biggest sign of intelligence is being able to give a concise description of what you’re seeing, to not throw everything up. For example, if you ask what makes a good student, it is someone who can summarize in a few words what has just been said in the class.

What you need is very intelligent nodes that make sense of the world around them, and give insights to the rest of the network. Everything is connected, but you don’t want to stream the video, you want to stream the insights.

VB: Why pursue data flow architecture? Does the structure of the neural network influence the approach to the designs intrinsic in your chip?

Danon: That’s an important point. The whole idea of the data flow architecture is to look at the way neural networks are structured, but to provide something that doesn’t try to mimic them as a sort of  hard-coded neural network. That’s not the idea.

By understanding the concept of data flow, and how the processing is distributed, we can derive from that a flexible architecture which would map the problem description at the software level relatively simply and efficiently to the product implementation at the hardware level.

Hailo is a dedicated processor. It’s not meant to do graphics. It’s not meant to do crypto. It’s meant to do neural networks and it takes inspiration from the way neural networks are described in software. And it’s part of a complete system that serves [the needs of the applications] from end to end.