Robotic Arm with Computer Vision

Robotic Arm with Computer Vision - Picking Up the Object


Idea

The main idea behind this project was to create an environment where a robotic arm can execute various commands based on image analysis of a scene. In this article, I will describe each part of the project in detail. For the first task, I focused on detecting and moving a single object.



Environment

The setup consists of several components assembled together. I used an old table as the base, repainting it white to provide better contrast with the objects. The robotic arm, which I purchased from eBay, is mounted on the middle of the longer side of the table. The arm has six servo motors, including a rotating base and claws at the other end. The parts are made of aluminum and are quite sturdy.

Next, I cut and mounted perforated metal ledges to the corners of the table, securing everything together. I then attached an RGB LED strip to the bottom side of the top part of the structure. Finally, I placed a USB camera at the top of the setup to capture the entire scene.


Communication with Arm

The robotic arm is equipped with six servo motors. The quickest way to control these motors is through a servo controller, which allows for the management of individual or grouped servos. I chose a controller with a serial output and a custom protocol, enabling communication via USB with just a few lines of code in any programming language.

Example of a group operation:
#1P1500#2P2300#3P2300#4P2300#5P2300#6P2300T100\r\n
#servo-indexPtime-in-milis-as-rotation#servo-indexPtime-in-milis-as-rotation...Ttime-for-execution\r\n


Logic flow

The application consists of several independent components that communicate with each other.

  • The input from the camera runs in a separate thread, where it preprocesses the "interesting frame" (the frame detected after movement). The result of this preprocessing is a list of detected objects with their coordinates.
  • The interesting frame is then sent to the main logic module. Here, all the other modules are registered. If no module is active, the system tries to initialize the first module that satisfies the initial conditions. If a module is already active, the interesting frame is sent to that module. The module handles the logic and decides what to do next. It then sends movement commands to a queue for the USB communicator.
  • The USB communicator repeatedly reads messages from the queue and sends commands to the controller via USB. The controller then moves the servo motors accordingly.
schema of logic flow

Calculation of the Next Move

One of the most frequently used features will be picking up objects. Once the camera provides preprocessed input, the next step is to calculate the movements required to pick up the object. At this point, we have a frame with the detected object and the center of the arm. We also know the real dimensions of the table, the length of the arm’s segments, and the base height. The task is to calculate the angles for each servo in the arm so it can reach and pick up the object. We can break this problem down into two smaller geometric problems.

  • The first part involves base rotation (imagine viewing the arm from above). This is a trigonometry exercise where we know all the points of a triangle and two sides, and we need to calculate the angle. We then convert this angle to milliseconds for the servo controller.

  • The second part involves rotating three servos to tilt the arm (imagine viewing the arm from the side). In the initial version, the height of the object is unknown, so we use a constant. This problem is similar to the previous trigonometry problem, but here we have one side and angle for each segment. By substituting the right values, we can determine if the arm has reached the object. I used brute force to calculate the three angles, and the process took less than a second.



Modules

The goal is to create an application with easily insertable modules. Each module can be thought of as a series of moves with custom logic, extended from a template class. Each module is defined by a list of states. A state can change based on one of the following triggers:

  • Time Trigger: Wait for a specific time before executing the next move.
  • Interesting Frame Trigger: When movement is detected by the camera.
  • Command Execution Trigger: Wait for a signal from the USB controller indicating that the move was executed.

For example, a module for picking up an object could have the following states:

  1. Start (triggered by an interesting frame)
  2. Pick up the object (triggered by time)
  3. Move the object (triggered by time)
  4. Release the object and return to the default position (triggered by time)
  5. Verify if the object was moved (triggered by command execution)

First testing

While working with C++ and OpenCV, I encountered a small issue: you cannot display an image from a background thread; only the main thread can call the imshow() method. To solve this, I used a singleton instance to store the images, and the main thread then displays them.

One of the unresolved issues is detecting the object's height. It’s not possible to determine the height from a single camera. This could potentially be addressed by using a sensor at the end of the arm or another approach.

Despite these challenges, the initial tests for picking up objects were successful. However, more calibration is required. After calibration, machine learning could be used to identify the optimal spot on the object for successful picking. Additionally, the claws need to grip the object tightly enough to prevent it from slipping.

You can find the source code here: https://github.com/mbodis/rawcv




Comments

Post a Comment

Popular posts from this blog

Skate Tricks Recognition Using Gyroscope

Image
In this article, we’ll describe how to recognize skateboard tricks using a gyroscope. This sensor is already present in most smartphones, but if you’re not familiar with it, here is a description . Before we dive in, a small disclaimer: This project originated as part of a Slido hackathon with the goal of demonstrating that it’s possible to recognize skateboard tricks using a gyroscope. For simplicity, we’ll focus on recognizing just two basic tricks and won’t consider the skater’s stance on the board. We’ll break down the problem into several smaller tasks: recording the trick, storing the data, describing the trick, analyzing it, and, finally, recognizing it. Recording the Trick Let’s start with trick recording. To obtain precise data, the device with the gyroscope needs to be attached directly to the skateboard. Place it on the bottom of the board to avoid interfering with the execution of the trick. The device should also be compact and lightweight to avoid altering the board’s cen...
Read more

Play table

Image
Using proximity sensors for playing midi tones combined with LED visualization Description The goal of this project was to create a table-sized device equipped with multiple proximity sensors capable of playing MIDI tones. Each sensor is accompanied by LEDs that indicate the distance of the user's hand above the table. This interactive table can be used by one or more people simultaneously. Hardware Setup I began with a cardboard prototype to test the sensors and the underlying logic. Afterward, I ordered a custom plottered sticker with a design which was painted with bare conductive paint. I drilled holes and connected the touchboard to seven Arduino Nanos, each controlling 13 LEDs. Additionally, I incorporated two potentiometers: one for volume control and another for changing the note setup. Programming The programming task was divided into two parts: Master Program (Touchboard): Reads values from the proximity sensors and sends messages to the slave Arduinos. Sl...
Read more

Counting dice and train wagons using computer vision

Image
Computer vision exercises with preprocessing Before starting my next project, I decided to work on some computer vision exercises. Each example is based on straightforward image preprocessing techniques. No complex data structures or machine learning are involved. Dice Detection I got this idea while browsing the net and became curious about how challenging it would be to write such a script. Here’s a step-by-step breakdown of the algorithm: Movement Detection : By comparing several frames with thresholds, we can determine if there is any movement in the frame. Adding a small time buffer after the movement stops gives us more accurate information. Removing the Background : Thresholding the grayscale frame helps to remove the background, leaving us with a binary image that highlights the objects. Cropping the Objects : Using contours, we can detect and isolate the objects by cropping them. Detecting Dots : By inverting the image, the dots on the dice become more distinguish...
Read more