Building software to control a robot based on a video feed is tricky. Generally, you get an idea how your algorithm should work so you implement it. But when you try it the robot does something completely different to your plan and you have no idea why. And the next time you try it, it does something different again.
Prior to the 2020 games we swapped from an onboard RPi screen to using a network connection to control the Pi. This gave us the opportunity to build an infrastructure to assist with the machine vision coding of our robot. The video pipeline of the software is shown below:
The game framework is the centre of the application on the robot. When it runs on the robot it receives a live video feed from a Raspberry Pi camera via Pi specific camera code and OpenCV. Individual video frames are passed to game specific video interpreters at a rate of 20 frames per second. The interpreters process each frame and direct robot movements (via the framework). They can also, optionally, mark up the image to assist with debugging.
The framework presents an IP socket interface. A PC application, “PW2020 Controller”, can be used to connect to the socket interface, normally via WiFi (although a wired connection is also possible). The PC application can display a live feed of the marked-up images, also at 20 frames per second. The application permits access to the game framework parameters and various robot control features.
In the image above the controller is showing a video feed which has been marked up by the line follower code. The line follower has partially marked up the line segment and is requesting a turn of 16 degrees. Sadly, this is wrong, the screen also says that the next turn should be left! Still, that’s the point of this system.
By default, the PC application saves a copy of the received marked-up images to a video file on the PC.
The most useful aspect of all this is that the game framework can also be used on a PC; note in the image above the application reports that it is connected to “localhost”. When used in this way, the video feed comes from a saved video file on the PC which was made as described above. So, it is possible to record the robot performing a short sequence (from the robot’s camera), then replay the sequence on a PC until the video processing can be finalised. Furthermore, this gets recorded too, so it is possible to run a sequence and step back and forth, frame by frame, to really drill down into the problem. This allows much quicker development. It also allows testing of new algorithms against an archive of known problem scenarios.
Team PiDrogen 