Attack of the Snowbots!! Based on the animated TV commercials that had come out before this campaign went live, the concept involved catching the snowballs thrown by an evil Snowman-bot with your Droid-powered mechanical arms. The arms are directly controlled through the user’s arm gestures. It was designed as a game, with the user receiving power-ups and points on catching snowballs.
The game was implemented in Unity3D.
Since Unity3D’s GUI system wasn’t very good at that time and we needed a really fluid animated visor HUD for this, I ended up building a system which used HUD animations created in Adobe After Effects which were then exported as a png sequence and used inside Unity3D as animated GUI elements.
This was an outdoor experience, hence we couldn’t go with 3D depth cameras. Hence we developed a gesture tracking system using normal webcams, which used the local motion of the user’s arms to determine the catching gesture. We implemented this as a separate module in openFrameworks and messaging happened with the main game using OSC.