Final Project: Motion In One Image

In this project, I took video of a moving subject, and attempted to combine all the frames of motion into one image. I had two motivations for this assignments. First, I hoped that it might be used to correct form in karate or other sports. Secondly, I want to make something that looks cool :)

Here is a video I used as my source for some of the results below. Its a simple video of me using a bo. Next, the overlay of all frames into a single image.

Well, that satisfies my coolness objective, but you can't see much in the way of form. Its mostly just a blur of arm and bo in the middle. Let's try with every fifth frame instead of all of them.

That is a little better. I can see that my shoulder is moving to far forward, my back leg is moving more than it should, and my elbow isn't flush with the bo at the end of the motion. That wasn't as obvious to me in the video, so I've gained some information about karate form.

Now, how did I produce these images? First, I found the median of all the frames.

As you can see, all the moving objects have been removed, while the more stationary parts remain. Now, to produce the output image, for each pixel i,j, we take i,j from the frame where i,j is maximally distant from i,j in the median image (in RGB space). We produce a labeling showing which frame to take each pixel from, and then run an average filter over the labeling to smooth things out a bit. A graphical representation of this labeling for the every-fifth-frame image is shown below. Brighter colors are from later frames.

And voila! That is how we get our result. Well, actually, I wrote some matlab code to make the computer do these steps for me. Here is another image of me twirling the bo, this time shot from the front.

Although these results are (in my opinion) pretty cool, there are still some problems. For one, visible artificats clutter the images. Artifacts can probably be reduced by improving the label generation algorithm. Graph cut optimization is a promising way to approach this. Secondly, we might make it better for diagnosing karate forms-- perhaps by incorporating temporal information. We could, for example, highlight the leading edge of objects, or tint pixels from latter images in different colros. This would allow one to see where in time a pixel came from, and thereby determine direction of movement.