Getting TUIO and CCV to work

TUIO and CCV problems and solutions

The first important part of our project was to get the computer vision working. Initially we knew that a previous project had utilized the webcam behind the screen in order to create a touch supported program with Unity. However, we also knew that they had not been able to build their project as a single executable file. They could only run the project with working CCV inside the Unity editor. 

Firstly we set up a particle system in Unity using Unity's built in shuriken particle system. We did not bother with optimizing the particles system through shader programing at first since we wanted to get the CCV multitouch to work first. 

The first system we built only had a 2D particle system on the screen. When a hand touched the screen, nearby particles changed color and/or started falling downwards. We thought this was enough to test whether multitouch would work or not. We also thought we should not develop further than this until we knew if we could build the project as a standalone executable or not.

Figure 1. Each particle within a given radius 

from the touch position changes color.

The multitouch did work. Basically we checked, through TUIO, each hand that was touching the screen and called a method multiTouchUpdate(), for each hand.

However, as the number of hands reached two or more, the framerate suffered immensely. With three or more hands, the project was nearly impossible to interact with as the framerate was extremely low. 

We concluded this would be fixed both by moving the particle system to the GPU as well as by creating a better algorithm for handling the multitouch. Basically, we realized we had to push the hands into a Dictionary or an Array and update for all the hands simultaneously in unity's "Update()" function.

Unfortunately, the project did not build a project that was runnable - touching the screen did not work. In addition to this, we had no idea how to actually solve the problem as a previous project utilizing the CCV/TUIO had not solved this either. However, we came to the agreement that this was the most essential part of the project right now, meaning the next step in development was to find out why the project did not build correctly and how to fix it. That will be talked about in the next blog post.