Skate tricks recognition using gyroscope

In this article we're going to describe how to recognize a skateboard trick using a gyroscope. This sensor is already present in most of the smartphones but in case you are not familiar with it, here is a description. Before we begin, small disclaimer. This project was originally as a part of hackathon that was used with Slido and the purpose was to confirm that it's possible to recognize a trick using a gyroscope. For the sake of simplicity of this project we're not considering skater's stance on a board and we're only trying to recognize 2 simple tricks. So, let's begin by splitting the problem into several smaller ones. We need to record the trick, store it, describe it, understand it, and then of course recognize it.

Recording the trick

Let's start with the trick recording. To have a precise data we need to have a device with a gyroscope that is attached directly to a skateboard. We need to place it on the bottom of the board, otherwise it might interfere with executing of a trick. It also shouldn't be too big or heavy so it does not change the center of a board.

Here are few setups how to achieve this:

a) You can have 2 devices. One simple device that is just recording and sending the data. And a main device that will be storing and processing the data. 

    pros: simple device can be very small and light because of small number of components

    cons: more complicated setup, need to take care of data transmission

b) One device that is doing everything: recording, storing, recognizing, showing output. 

    pros: simple setup, you don't have to take care of a data transmission

    cons: it might be bigger and heavier, need some hardware to show the output

For my first approach, I chose the option with two devices. For the simple device I used Arduino-Nano with MPU-6050 Accelerometer and Gyroscope module, bluetooth HC-06 module and a small battery. For the main device I chose Android smartphone that was receiving data via bluetooth. 

 

The communication is pretty straightforward. After devices are connected via bluetooth, Android sends a message to start recording(1). Simple device starts recording data from the gyroscope for 10 seconds(2) and sends it directly back to Android(3). Data are stored into a file afterwards. 

During this approach I ran into several small issues:

• restart of a device caused by bad connection between components - caused by shock when landing the trick (I was using jumper cables)
• bluetooth distance
• gyro stopped working after some time
• hard to debug

For the lack of time I also tried the second approach. In this setup, I used printed holders to attach the phone onto the bottom part of the board. It was a little bigger and heavier, but for test purposes it was possible to use it. This way I was able to record two tricks (kickflip and shovit) several times.

Understanding tricks

There are several different types of data that you can retrieve from a gyroscope (acceleration, rotation ...). I chose the rotation as the main data source. Each data recording was set to 10 seconds length. Here is how it works. 

1. Here we can see raw data of a rotation. At first, the data don't look very neat. It's because when the rotation reaches 180 or -180 it's flipping to the opposite side of an axis. (e.g value 182 is transferred to -178).
2. Here is a simple attempt to clean the data and keep the function the same. (adding 360 to all negative values).
3. Simple formula helped me to clean the rotation jumps (searching for big differences in two consecutive values). Now we can try to read the data: 
1. In the first 2 seconds I'm putting down the board so we can ignore the beginning part. Then all 3 axis are still.
2. Red axis is showing the vertical change. I did a small ollie (jump).
3. Yellow axis is showing flip rotation. For this trick there was almost none. (I was doing a shovit)
4. Blue axis is showing the turning rotation. We can see the change in the middle of the graph. Rotation changed about 180 degrees. So that's the shovit we've been looking for.
4. Extracted shovit. (around 0.5 second)
5. Every time we change direction on a board we need to normalize the rotation value. (So the graph is always around 0)

Trick recognition

I had several ideas how to do the recognition. Each trick is described as 3 vectors (x, y, z axis).  I started with the Dynamic Time Wrapping algorithm. This algorithm is basically searching for minimal distance between two graphs. I used it with all my recorded samples and found a threshold that was meeting my needs. Because I had only 6 samples, I was able to use the brute force and compare it with all my samples. This wouldn't probably be possible with larger data-set.

Summary

As a prototype I would say it was successful and I had a lot of fun with it. But if I was to dive into the subject deeper, I would definitely choose a different approach. The proper solution would be to create a big data set containing a lot of tricks recorded by multiple skaters. I would also record all data types that are available (e.g. acceleration) from the gyro-sensor. For the hardware setup, I would chose two simple devices to be installed under each skate truck (this way we can distinguish between nollie and ollie).  For the recognition, I would probably use the machine learning and move the processing to a server as I was hitting the time limit on the phone using the brute force approach.

After finishing my prototype, I was searching for similar topics and found a few papers that were describing the same problem with multiple algorithm overview.

• Development of a skateboarding trick classifier using accelerometry and machine learning
• IMU-based Trick Classification in Skateboarding
• The classification of skateboarding tricks via transfer learning pipelines

As I mention at the beginning, this project was originally as a part of hackathon that was used with Slido. Here is short presentation with a demo at the end.