Another lockdown, another project for body and mind. Slackpose allows me to track my slackline walking and review my technique. Spending 5 minutes on the slackline between meetings is a great way to get away from my desk!
I had considered pose estimation for wheelies last year, but decided slackline walking was an easier start, and something the whole family could enjoy.
Setup
I mount my phone on a tripod at one end of the slackline and start recording a video. This gives a good view of side-to-side balance, and is most pixel-efficient in vertical orientation.
Alternatively, duct tape the phone to something handy, or even hand-hold or try other angles. The 2D pose estimation (location of body joints in the video image) will be somewhat invariant to the shooting location, but looking down the slackline (or shooting from another known point) may help reconstruct more 3D pose data using only a single camera view. Luckily, it’s also invariant to dogs wandering into frame!
I use OpenPose from the CMU Perceptual Computing Lab to capture pose data from the videos. See below for details of the keypoint pose data returned and some notes on setting up and running OpenPose.
I then analyse the keypoint pose data in a Jupyter notebook that you can run on Colab. This allows varied post-processing analyses such as the balance analysis below.
Keypoint Pose Data
OpenPose processes the video to return data on 25 body keypoints for each frame, representing the position of head, shoulders, knees, and toes, plus eyes, ears, and nose and other major joints (but mouth only if you explicitly request facial features).
These body keypoints are defined as (x, y) pixel locations in the video image, for one frame of video. We can trace the keypoints over multiple frames to understand the motion of parts of the body.
Keypoints also include a confidence measure [0, 1], which is pretty good for the majority of keypoints from the video above.
Balance Analysis
I wanted to look at balance first, using an estimate of my body’s centre of mass. I calculated this from the proportional mass of body segments (sourced here) with estimates of the location centre of mass for each segment relative to the pose keypoints (I just made these up, and you can see what I made up in the notebook).
This looks pretty good from a quick eyeball, although it’s apparent that it is sensitive to the quality of pose estimation of relatively massive body segments, such as my noggin, estimated at 8.26% of my body mass. When walking away, OpenPose returns very low confidence and a default position of (0, 0) for my nose for many frames, so I exclude it from the centre of mass calculation in those instances. You can see the effect of excluding my head in the video below.
Not much more to report at this point; I’ll have a better look at this soon, now that I’m up and walking with analysis.
OpenPose Notes
Setup
I ran OpenPose on my Mac, following the setup guide at https://github.com/CMU-Perceptual-Computing-Lab/openpose, and also referred to two tutorials. These instructions are collectively pretty good, but note:
3rdparty/osx/install_deps.shdoesn’t exist, you will instead find it atscripts/osx/install_deps.sh- I had to manually pip install numpy and opencv for OpenPose with
pip install 'numpy<1.17'andpip install 'opencv-python<4.3‘, but this is probably due to my neglected Python2 setup. - Homebrew cask installation syntax has changed from the docs; now invoked as
brew install --cask cmake
Running
Shooting iPhone video, I need to convert format for input to OpenPose. Use ffmeg as below (replace slackline.mov/mp4 with the name of your video).
ffmpeg -i slackline.mov -vcodec h264 -acodec mp2 slackline.mp4
I then typically invoke OpenPose to process video and output marked up frames and JSON files with the supplied example executable, as below (again replace input video and output directory with your own):
./build/examples/openpose/openpose.bin --video slack_video/slackline.mp4 --write_images slack_video/output --write_json slack_video/output