Lane Detection: How Cars See The World

6 min readApr 8, 2020

Your car was born with eyeballs, right?

No, of course not, your car isn’t that cool. But, what if it could be? What if all cars could see?

You’re probably thinking that mutant, eyeball cars aren’t possible, but you could strap a camera onto your car. This is what gives self-driving cars computer vision.

Computer vision is how computers can understand digital images or videos. Its goal is to seamlessly understand and automate tasks like the human visual system. This powerful tool is used in a variety of areas such as convenience stores, daily medical diagnostics, monitoring the health of crops and livestock, and the foundation of self-driving cars.

For this to happen, the images and video must go through a process to decode what’s happening, so your car can understand what it’s looking at and know what it should do. And that’s exactly what I did in my project! This algorithm was written in Python with the help of the NumPy and OpenCV libraries. You can find the full code on GitHub here.

This lane detection algorithm allows your car to recognize its environment. If it were combined with other software, cameras, and sensors, it would allow your car to self-park and, ultimately, make a self-driving car able to remain in its own lane.

The Process

For your car to be able to recognize lane lines, it needs to go through a few steps:

Gaussian Blur
Convert the image to greyscale
Turn into a canny
Create a mask using bitwise_and
Generate lines
Clean up lines

Gaussian Blur

First, the image must go through what’s called a Gaussian Blur. This reduces the unnecessary noise and smoothens the image, by changing the value of each pixel by a factor of the average value of pixels that surround it. To find the average value of pixels, a kernel or matrix of normally distributed numbers is passed across the image and sets each pixel value equal to the weighted average of its neighbouring pixels.

This process may seem complicated at first, but it just blurs the image and allows your car can focus on the most important thing — the road.

Without it, your car would be paying attention to irrelevant things like cracks in the road or tree branches. It would be like your car is texting while driving, and would you trust that kind of driver?

#This is what the code would look like.
blur = cv2.GaussianBlur(gray,(kernel, kernel),0)Canny

Canny

Then, the image is going to need to be converted into a canny. This converts the image to greyscale and then identifies the largest changes in pixel brightness, also known as the gradient. This turns the whole image to black, except for the sharpest changes in gradient. Now, your car is focused on the lane lines, mostly.

Pixel intensity ranges from black being zero (0) and white being two-hundred and fifty-five (255). Everything in the middle is a greyish mix of the two. This range wasn’t just pulled out of a hat, it is actually based off of the binary number system.

Binary is written using a base-2 number system, meaning it only uses two numbers: 0 and 1. A zero is “off”, while a one is “on”. Now, zeros and ones are used to represent numbers written in the typical based-10 number system. I find it difficult to explain by writing, so here’s a quick video that explains binary numbers.

https://www.youtube.com/watch?v=Ieq8AR8krrA

Now that you understand how to read and write binary numbers, the rest should make sense. The canny operation runs a five-by-five kernel (5x5) across the entire image to average out the pixel intensities.

When the computer notices a drastic gradient in brightness, in this case, three times the whiteness or blackness of a set of pixels, the computer draws over the sharp contrast in white or a pixel intensity of two-hundred and fifty-five (255).

def canny(image):
    gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    kernel = 5
    blur = cv2.GaussianBlur(gray,(kernel, kernel),0)
    canny = cv2.Canny(gray, 50, 150)
    return canny

The Mask

Now that your car is focused on the most drastic changes in brightness, the lane lines on the road are more visible. However, there is still some “noise” like trees and mountains that have caught your car’s attention. To solve this, I created a mask over the image to allow your car to keep you safe and focus on the lanes.

For this, I had to align the image on a grid and choose the coordinates for the area your car must focus on.

It’s close to being finished, but I still needed to use an operation called bitwise_and. This turns the mask into an array of zeros and compares the pixel intensities of the canny image and the mask. Bitwise_and only selects the homologous pixels (the pixels when multiplied together, give a pixel value of one), therefore masking everything outside of our region of interest.

#This creates a triangle and uses it to mask the rest of the image.
def region_of_interest(canny):
    height = canny.shape[0]
    width = canny.shape[1]
    mask = np.zeros_like(canny)
 
    triangles = np.array([[
    (200, height),
    (550, 250),
    (1100, height),]], np.int32)
 
    cv2.fillPoly(mask, triangles, 255)
    masked_image = cv2.bitwise_and(canny, mask)
    return masked_image

Generating Lines

Now, the image is finished and is ready to be used to generate lane lines so your car to stay on its side of the road. For this, I had to find the slope of the left and right lane lines and highlight them. I did this by highlighting the white pixels with blue (colour makes no difference).

This works, but the lane lines don’t show up as perfectly straight in our image because lines are just a group of pixels, so it’s a bit rough. To fix this, I merged some of the smaller lines into larger, cleaner lines by using a function called Hough Transform.

For simplicity, Hough Transform uses a voting system in Hough Space to have stronger biases for the larger pieces of the line. Therefore, individual pixels are worth less, while larger groups of pixels in the line are worth more. Then, this technique finds the equation of a line that passes though all of the smaller pieces of the line. By calling this function, it saves time having to loop through each of the pixels to find the equation of the line.

Finally, I layered the original copy of the image to the new processed image. To output this finished image!

Great! Now, I’ve given your car the gift of sight. I’m sure its very grateful for its newfound powers. Computer vision is an extremely powerful tool in the autonomous vehicle industry. With time, innovation, and hard work, we might just get cars with real eyeballs.

Key Takeaways

Self-Driving cars see using computer vision, which acquires, processes, analyzes and understands digital images.
An image must go through a Gaussian Blur, a greyscale conversion, and a canny before a computer can recognize what’s happening.
The processed image is used to find and highlight lane lines in the road.

Hey, I’m Kael Lascelle, a sixteen-year-old Innovator at The Knowledge Society! I have a passion for autonomous systems, especially self-driving cars, as well as sustainable energy.

I would appreciate if you could follow me on Medium and Twitter! Also, add me on LinkedIn, or send me an email.