Last year we used a Raspberry Pi 3 to perform vision processing which allowed us to implement auto targeting functions. We originally were using the vision processing provided for the RobotRio, but found that we could not process more than 15 frames/sec. even if we lowered the resolution to 320×160. With the Raspberry Pi, we were able to process about 40 frames/sec. At a resolution of 640×480.
The Raspberry Pi system uses the Raspberry Pi camera. Both the Pi and the camera can be purchased for a total of around $70. At some point I would like to put together a tutorial describing how to set up the Raspberry Pi, but that involves a large number of steps, some of which I don’t really remember at this point. To get around this problem, we are providing complete Raspberry Pi images which are already pre-configured. There are two images which you can use:
Pi3-xrdp-IP50-Master.zip – This image automatically connects to a specified existing WiFi hotspot. It is currently configured to conned to a hotspot named robotics with the password ‘raspberry’, and has a fixed IP of 192.168.0.50. You can, however, easily configure it to connect to any network.
Pi3-xrdp-AP-Master.zip – This image is configured as a WiFi hotspot (IP 172.24.1.1) to which you can connect.
Both of these images have a remote desktop server installed which allows you to connect to the Raspberry Pi using Window’s remote desktop application
These images also have a version of Eclipse installed which is set up to compile C++ programs. There is a project called NetworkVision-Multi which supports multiple regions as well as multiple colors.
Each of these versions implements a image processing server which runs on the Raspberry Pi and communicates with the RoboRio and the Driver Station via a network connection.
The following will describe the NetworkVision-Multi program:
The first server listens on port 5800 and process images and sends information about the position of the recognized regions to the RoboRio. To use this, the RoboRio would connect to this port on the Raspberry Pi and then it will receive an ascii data stream containing the following commands:
F [frameNo] [targetX] [targetY]
Specifies the start of a new frame. The command starts with the character ‘F’ and is followed by three integers.
- frameNo – specifies the frame number of the process frame.
- targetX – specifies the ideal horizontal target position as set by the driver station.
- targetY – specifies the ideal vertical target position as set by the driver station.
R [color] [left] [top] [right] [bottom] [topLeft] [topRight]
Specifies a region for the current frame. The command starts with the character ‘R’ and is followed by seven integers.
- color – specifies the color index for this region.
- left – specifies the left edge of the bounding box, in pixels.
- top – specifies the top edge of the bounding box, in pixels.
- right – specifies the right edge of the bounding box, in pixels.
- bottom – specifies the bottom edge of the bounding box, in pixels.
- topLeft – specifies the top left most position of the region.
- topRight – specifies the top right most position of the region.
Specifies the end of the current frame. The command consists of the character ‘E’.
In addition to the server that provides the region data to the RoboRio, a second server is started which provides information and images to the driver station computer. This server listens on port 5801 and when connected sends data packets consisting of a header which contains information about the image processing, followed by a jpeg data stream which encodes one of the frames. This can be used for display of the camera image. In addition, commands can be send to the server which allows the driver station computer to control the capture parameters.
The program used to view the images and control the server is a Java program which can be launched using the ImageViewer2017.cmd file that can be found in the PiUtils folder located in the RobotTools.zip file. The Eclipse project to build this tool can be found in the workspaceJava2017 folder of the same RobotTools.zip download.
ImageViewer2017.cmd is a Windows command file which starts the Image Viewer. Note that the last parameter specifies the IP address of the Raspberry Pi:
java -cp .;ImageViewer2017.jar imageViewer.ImageViewer 172.24.1.1
Before you run the Image Viewer, you must first start the Image Server on the Raspberry Pi. To do this you can use the SSH utility Putty.exe which is also located in the RobotTools folder. You can log into the Raspberry Pi using the user name ‘pi’ and the password ‘raspberry’. Once logged in, you need to run the program ‘NetworkVisionMulti’ located in the folder ‘networkVision’. When this program is running, your SSH terminal window should look like:
Now that you have the server running, you should be able to start the Image Viewer on your computer and connect to the pi. This should result is a display similar to the following:
You can control the parameters that control the processing of the images. The images are recognized by color and you specify a range for the Hue, Saturation, and Value components. Note that you can click anywhere on the image and the HSV values for the pixel at that location will be displayed below the image. This version of the Image Processor is capable of handling multiple regions and up to four simultaneous colors.
You can also control camera settings such as shutter speed, brightness. saturation and ISO. Finally there are horizontal and vertical target parameters that can be set. These numbers will be sent to the RoboRio as part of the region data.