Linetracking with opencv

Overview

This Tutorial will show how to make a line detection and find a center of line for a line following pi car using OpenCV.

In the line tracking process. I used Python and OpenCV to find lines in a real time video.

The following techniques are used:

1. Canny Edge Detection

2. Hough Transform Line Detection

Finally, I applied these two techniques to process video clips to find lines. There is a good instruction to line tracking in Car-Finding-Lane-Lines

Materials/Prerequisites

You will need a Raspberry pi 3 with python installed and Pi camera (Webcam works too). you also need to install OpenCV onto your Raspberry Pi. We can follow instructions to install OpenCV. If it is the first time to use pi camera, to be able to access the Raspberry pi camera with OpenCV and Python, we recommend to look at this instruction. For interfacing with the Raspberry Pi camera module using Python, the basic idea is to install picamera module with NumPy array support since OpenCV represents images as NumPy arrays when using Python bindings.

Process

If you are using Webcam, you can simply follow this link to realize line detection. The code is given below:

import sys
import time
import cv2
import numpy as np
import os

We start by importing our necessary packages.

Kernel_size=15
low_threshold=40
high_threshold=120

rho=10
threshold=15
theta=np.pi/180
minLineLength=10
maxLineGap=1

Next, we set some important parameters. They represent different things:

kernel_size must be postivie and odd. The GaussianBlur takes a Kernel_size parameter which you'll need to play with to find one that works best.

low_threshold – first threshold for the hysteresis procedure.

high_threshold – second threshold for the hysteresis procedure.

rho – Distance resolution of the accumulator in pixels.

theta – Angle resolution of the accumulator in radians.

threshold – Accumulator threshold parameter. Only those lines are returned that get enough votes ( >\texttt{threshold} ).

minLineLength – Minimum line length. Line segments shorter than that are rejected.

maxLineGap – Maximum allowed gap between points on the same line to link them.

Initialize camera
video_capture = cv2.VideoCapture(0)

From there, we need to grab access to our video_capture pointer. Here we grab reference to our Webcam.

#keep looping
while True:
    ret, frame = video_capture.read()

    time.sleep(0.1)

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
  
    blurred = cv2.GaussianBlur(gray, (Kernel_size, Kernel_size), 0)

Then we starts a loop that will continue until we press the q key. We make a call to the read method of our camera pointer which returns a 2-tuple. The first entry in the tuple, ret is a boolean indicating whether the frame was successfully read or not. The frame is the video frame itself.

After we successfully grab the current frame, we preprocess our frame a bit. We first convert the frame to Grayscale. We’ll then blur the frame to reduce high frequency noise and allow us to focus on the structural objects inside the frame. If Kernel_size is bigger the image will be more blurry.

The bigger kearnel_size value requires more time to process. It is not noticeable with the test images but we should keep that in mind (later we'll be processing video clips). So, we should prefer smaller values if the effect is similar.

  
#Canny recommended ratio upper:lower  between 2:1 or 3:1
edged = cv2.Canny(blurred, low_threshold, high_threshold)
#Perform hough lines probalistic transform
lines = cv2.HoughLinesP(edged,rho,theta,threshold,minLineLength,maxLineGap)

Then we perform canny edge-detection. When there is an edge (i.e. a line), the pixel intensity changes rapidly (i.e. from 0 to 255) which we want to detect. We introduced high_threshold and low_threshold so that if a pixel gradient is higher than high_threshold is considered as an edge. Similarly, if a pixel gradient is lower than low_threshold, it is rejected. Bigger high_threshold values will provoke to find less edges and lower high_threshold values will give you more lines. It is important to choose a good value of low_threshold to discard the weak edges (noises) connected to the strong edges. You need to adjust these two parameters to make the number of lines be acceptable.

Finally, we use cv2.HoughLinesP to detect lines in the edge images.

  
    #Draw cicrcles in the center of the picture
    cv2.circle(frame,(320,240),20,(0,0,255),1)
    cv2.circle(frame,(320,240),10,(0,255,0),1)
    cv2.circle(frame,(320,240),2,(255,0,0),2)
    
    #With this for loops only a dots matrix is painted on the picture
    #for y in range(0,480,20):
            #for x in range(0,640,20):
                #cv2.line(frame,(x,y),(x,y),(0,255,255),2)
    
    #With this for loops a grid is painted on the picture
    for y in range(0,480,40):
            cv2.line(frame,(0,y),(640,y),(255,0,0),1)
            for x in range(0,640,40):
                cv2.line(frame,(x,0),(x,480),(255,0,0),1)

This part of code is not necessary but it makes the frame more vivid when showing on the screen. We draw circles in the center of the picture and grids in the picture.

                
    #Draw lines on input image
    if(lines != None):
        for x1,y1,x2,y2 in lines[0]:
            cv2.line(frame,(x1,y1),(x2,y2),(0,255,0),2)
            cv2.putText(frame,'lines_detected',(50,50),cv2.FONT_HERSHEY_SIMPLEX,1,(0,255,0),1)
    cv2.imshow("line detect test", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything is done, release the capture

video_capture.release()
cv2.destroyAllWindows()

The last part draws the lines in the picture. If lines are not NoneType, we use a loop to draw the lines.

Our group is using pi camera so the code needs to be modified. Otherwise it will show Nonetype error due to an image not being read properly from disk or a frame not being read from the video stream. You cannot use cv2.VideoCapture when you should instead be using the picamera Python package to access the Raspberry Pi camera module.

There are two ways to modify the code to replace USB Webcam with pi camera.

One is swapping out the cv2.VideoCapture for the VideoStream that works with both the Raspberry Pi camera module and USB webcams. more details

We did another way. Firstly we import some pi camera packages

from picamera.array import PiRGBArray
from picamera import PiCamera

when initializing camera, we comment out video_capture = cv2.VideoCapture(0). Instead, we give code below:

#Initialize camera
camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 10
rawCapture = PiRGBArray(camera, size=(640, 480))

We also change the "While True" loop into:

for f in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):

Finally, we comment out "ret, frame = video_capture.read()" and add:

frame = f.array

Now you should be able to realize line detection with pi camera. However, you might find it is not satisfying because sometimes the video window suddenly disappears and the terminal gives you a Nonetype error again. In addition, the lines appearing in the window are more likely some sparse line segments or even dots. In fact, those problem stem from HoughLineP function. For more understanding of HoughLineP function, you can check This Link. Basically, cv2.HoughLinesP returns us an array of array of a 4-element vector(x_1, x_2, y_1, y_2) where (x_1,y_1) and (x_2, y_2) are the ending points of each detected line segment. (Each line is represented by a list of 4 values (x_1, y_1, x_2, y_2).)

The sample of cv2.HoughLinesP return value:

print(cv2.HoughLinesP) = [[[a b c d]]]

When we there is no line shown is the camera, cv2.HoughLinesP returns null(or None?) so that is the reason why the video window suddenly disappears. The code cv2.imshow("line detect test", frame) is in the if statement in which (lines != None). To maintain the window when there is no line detected, we could modified the code as below:

if(lines is not None):
        for x1,y1,x2,y2 in lines[0]:
            cv2.line(frame,(x1,y1),(x2,y2),(0,255,0),2)
            cv2.putText(frame,'lines_detected',(50,50),cv2.FONT_HERSHEY_SIMPLEX,1,(0,255,0),1)
    cv2.imshow("line detect test", frame)
else:
        cv2.putText(frame,'lines_undetected',(50,50),cv2.FONT_HERSHEY_SIMPLEX,1,(0,255,0),1)
        cv2.imshow("line detect test", frame)

After we understand the meaning of cv2.HoughLinesP function, we can modified the loop in the if statement since the old one "for x1,y1,x2,y2 in lines[0]:" only give us sparse line segments. The modified loop is:

for line in lines:
            for x1,y1,x2,y2 in line:

It would have a better result if we set both minLineLength and maxLineGap to be 0.

Now, we should be able to detect lines. How about the center of the lines? How do we find the center of it?

What we did is declare two variable x and y initialized to be 0. Then in the innermost loop of drawing lines from images, we set x=x+x1+x2 and y=y+y1+y2. Next we give another two variables xa and ya in the outer loop, which equal to x/(2 * the number of lines) and y/(2* the number of lines). Here, xa and ya represent the center of all the points(average point). Therefore, we can tell if our car is one the line or not by comparing (xa, ya) with (320, 240).

The final version of the code can be find in my github

Authors

Pi Car Discovery Group

Group Link

group project page

Weekly logs