A Rare look at what Tesla Autopilot can see and interpret.

           

We previously reported about what Tesla’s Autopilot can see with its suite of 8 cameras around the car, but we rarely got into how the Autopilot understands and interprets what it is looking at through its computer vision system.

That’s changing today after a few hackers were able to overlay Autopilot data on top of snapshots of what the system is seeing.

As with our previous looks at what Autopilot can see, it’s thanks to our favourite Tesla hacker ‘verygreen’ who has been able to intercept the data that Tesla gathers from his own vehicle.

With the help of a few other Tesla hackers, they were able to overlay the data with the snapshots of what the vehicle can see, which results in an interesting visualization of what Autopilot understands when looking through its sensors.

It’s not a complete look at what Autopilot can see. For example, it doesn’t include the tracking of the lanes.

But it does show how the system is tracking objects in its field of view and a way that we have never seen before.

verygreen sent us some pictures and videos along with an interesting explanation of what they managed to do.

Here it is in his own words:

Would not it be great if Tesla published videos about how their cars see things, in a fashion similar to what Google does? Well, they don’t and so the exercise is left to hackers to try and piece things together.

Even though my unicorn car with hacked autopilot board is long lost, I still have a whole bunch of snapshots from those days. I am sure you have seen the picture. But what’s not widely known is that alongside those pictures, radar data was included too. Now TMC user DamianXVI (the same guy that wrote colour interpolator for originally black and white images) came up with a way to overlay the radar data on the pictures.

It looks like this:

 
Now a bit of an explanation about what the circles mean: circle color represents the type of the object:

• green – moving
• yellow – stopped
• orange – stationary
• red – unknown.

The circle size represents the distance to the object, the bigger the circle, the closer is the object (so we are not trying to encircle object, or approximate the size – the radar has no way of knowing this).
A thick circle means this object has a label and is therefore likely being tracked by the autopilot. Fading in and out is due to the probability of existence changing.
Sometimes the circle is in some empty space – this is likely due to radar having some problems with determining object elevation, try to look higher or lower for a relevant object. Also at times, radar cannot determine elevation at all (value of -5 is used in that case). The circle is drawn at elevation 0 in this case.
Also keep in mind that the radar reports coordinates in 3D and we need to project them onto a 2D picture. Sometimes there are errors in such conversions as you might expect.
Here’s a video with the radar data overlaid:

Courtesy of Electrek.co.uk