As we did not have much success neither with compute shaders or VBOs we had a hard time moving forward with altering the particle system in a way which would fit our goals. We therefore decided to put this problem on hold for a while and instead work on other essential parts of the projects.
Something worth noting here is that we most likely have to keep investigating how to save the Particle System data in the VRAM of the GPU. One way to handle this is to keep investigating the VBO and possibly get it up and running to give us the desired effect. An alternative would be to find another solution to handle this problem. We have read a very interesting article where they used a texture to encode particle data. This could be a very quick way for us to store the particle data and easily access it in the next frame. The blog post can be found here: http://nullprogram.com/blog/2014/06/29/
Since we stopped working on the particle system for a while, we decided we should start working a bit more with handling the multitouch in the shader. We realized that this had to be done at one point anyways.
The problem with passing data to the shader in Unity is that we are very limited in the amount and types of uniforms we can send. For example, Unity cannot send an array of data to a GL shader program. So we had to get inventive.
We realized after a while that a texture is basically a big matrix that can store data in a sort of efficient way, if we are smart about it. Basically, we defined a maximum number of hands that we allow for multitouching the screen. For now, we set that number to 20, but it could be increased in the future. The shader needs to know a couple of things about the hands to be able to do calculations. The main thing is of course the current position. But, the initial position of the touch is also relevant - that way we can track how much the user has moved the hand without lifting the hand off the screen.
This meant we needed a 20 by 2 texture to store our data. Since each pixel in the texture consists of a vec3 (or vec4 if using RGBA), we are able to store quite a lot of data in a quite small texture. The first position in the texture, [i , 0], contains the x and y coordinates for the i:th hand. The second position, [i , 1] contains the intial position of the hand. This means that in the red component of the vec3 in the 0:th position contains the x value and the green component contains the y value. Same goes for the 1:st position.
We realize we could have had an even smaller texture (20 by 1) by encoding a RGBA value as x, y, xOld, yOld. However, with the way we have done it now we have reserverd some space in case we want to send some other data at a later stage. Also, this is an optimization that is probably completely unnecessary as the amount of data we are passing is already small. However, we will consider moving to the 20 by 1 texture at a later stage if we find no use for the now zero-value blue position and alfa position in the RGBA-array.
We are now working on decoding the hand position data inside the shader. However, it is difficult as debugging on the shader is nigh impossible. A sudden realization we got while writing this blog is that the reason we can't decode multitouch data at this moment might be that the texture expects values from 0 to 255 and not floating points between 0 to 1... We have to explore this...
måndag 11 april 2016
Investigating other ways to create an efficient particle system
After defining our goals with the particle system, we soon came to the conclusion that simulating space orbit and recreating gravity fields was a very complex task with ordinary vertex and fragment shaders.
Since we did not have a solution where we could update the particles' positions on the GPU, we realized we either had to solve that problem or create a function based on time to simulate gravity. However, this is mathematically very complex and requires a numerical solution. Additionally, we would have had to store the time frames for where hands had been in the past, to be able to "remember" how the particles had been simulated up until the current time.
This meant that we after a couple of attempts using the Vertex and Fragment shader started looking for alternate solutions. First we looked at how to implement and use VBO:s (Vertex Buffer Objects). They are normally used to upload vertex data to a GPU device. Since they use the VRAM on the GPU side instead of regular RAM they offer much better performance when used correctly. Unfortunately, we couldn't find any good examples of how to use them when scripting in C# and Unity, so we continued our search for other solutions.
We then looked at other types of shaders. We found that compute shaders is a very good and quite straight forward solution when creating a system with a lot of particles. Basically, compute shaders is a more modern technology compared to vertex shaders where the shader takes advantage of a large number of the GPU:s parallel processors for computing purposes, which enables high-level graphics. For example, many particle effects and systems in modern games use compute shaders to do graphical calculations. We found a lot of documentation on compute shaders and particle systems, which was promising. In other words, this seemed to be the solution we was looking for.
Due to how the compute shader works, it requires the GPU device to be fairly modern. In particular, one requirement is that the graphics card supports the DirectX 11 API. This was a problem since the GPU on the computer connected to the big 4K screen was a Quadro FX 5800 which didn't support DX11. So as unfortunate as it was, we couldn't continue with this approach and had to look for other solutions.
Since we did not have a solution where we could update the particles' positions on the GPU, we realized we either had to solve that problem or create a function based on time to simulate gravity. However, this is mathematically very complex and requires a numerical solution. Additionally, we would have had to store the time frames for where hands had been in the past, to be able to "remember" how the particles had been simulated up until the current time.
This meant that we after a couple of attempts using the Vertex and Fragment shader started looking for alternate solutions. First we looked at how to implement and use VBO:s (Vertex Buffer Objects). They are normally used to upload vertex data to a GPU device. Since they use the VRAM on the GPU side instead of regular RAM they offer much better performance when used correctly. Unfortunately, we couldn't find any good examples of how to use them when scripting in C# and Unity, so we continued our search for other solutions.
Screenshot of a simple particle system done in Unity with a Compute Shader
Due to how the compute shader works, it requires the GPU device to be fairly modern. In particular, one requirement is that the graphics card supports the DirectX 11 API. This was a problem since the GPU on the computer connected to the big 4K screen was a Quadro FX 5800 which didn't support DX11. So as unfortunate as it was, we couldn't continue with this approach and had to look for other solutions.
fredag 8 april 2016
Defining our goals with the particle system
Now that we had done some computations on the particle system and had gotten our multitouch to work, we were faced by a real problem: We needed to specifically define what we wanted to do and even more specifically how we wanted to affect the particle system based on our input.
We came to the conclusion that we wanted to simulate physics behavior in space. This means we wanted to make the hands touching the screen into big planets with high mass and at the same time give the particles a very low mass, causing them to orbit our "planets". Multiple hands would create more and more interesting behavior from the particles. This meant that we had to create a function based on time that described the orbits around the hands.
At this point we were very tired of discussing what to do with the particle system, so we decided to start implementing things during the next work session.
Moving the particles on the GPU
We started to experiment with moving the particles on the GPU side. Firstly, we wanted to move the particles somehow. After some research and previous experience we decided to try to make a moving star field/vortex simulation.
We figured that we could simulate it depending on the elapsed time by using modulus and sinus operations. The mathematical operations are "static" as such, they will loop forever. We also experimented with user input by sending touch data and moving the system accordingly.
Watch videos of the simulation below.
Moving the particles along the X-axis:
Moving the particles towards the camera with a spin effect:
Moving the particles with touch input:
onsdag 6 april 2016
Optimizing the multitouch
This will be a short post regarding the optimization of the multitouch.
As described in an earlier post, multitouch was handled by calling a function for every hand on the screen. This was a massive load on the CPU - it attemped to update graphics not every frame but as often as it could for every hand.
Instead, we decided that we would push the hands into a Dictionary, where the hands' ids are the keys and the position the values. We created a specific class to keep track of all the information required to handle our hands - here we refer to the source code for the interested reader.
Lastly, instead of using up all our CPU power we now had a Dictionary with our hands in it, ready to be used by either the CPU or the GPU (the goal, however at this point we do not have a solution on how to send the hand data to the GPU - it is a later problem).
As described in an earlier post, multitouch was handled by calling a function for every hand on the screen. This was a massive load on the CPU - it attemped to update graphics not every frame but as often as it could for every hand.
Instead, we decided that we would push the hands into a Dictionary, where the hands' ids are the keys and the position the values. We created a specific class to keep track of all the information required to handle our hands - here we refer to the source code for the interested reader.
Lastly, instead of using up all our CPU power we now had a Dictionary with our hands in it, ready to be used by either the CPU or the GPU (the goal, however at this point we do not have a solution on how to send the hand data to the GPU - it is a later problem).
Problems with TUIO and Unity
For the interaction part, we wanted to use the TUIO/CCV image API with our Unity project. This worked fine until we tried to build the project and run the exe file, and the touch input wouldn't work at all. We really wanted to fix this since we wasn't interested in being forced to run the program through the Unity editor.
We tried several solutions. For exampe, we deactivated the firewall since we initally thought that it would occupy the same port as TUIO. However, this didn't solve anything. We also tried to implement other TUIO-supported API:s. One of these was the TouchScript API, which was supposed to work with TUIO, but it didn't work either.
It was very hard to find any documentation on TUIO and Unity, and it was even harder to find a solution to the Unity build problem. After some extensive Google sessions, we found only one (!) post that even mentioned this problem (read it here). The solution there was to do modifications in the actual TUIO API (or try to implement another modified TUIO protocol). At this point, we almost gave up since it seemed impossible to solve this problem.
However, we discovered that the error message in the log file from the build wasn't telling us everything. We did some googling on this specific error, which lead us to another problem: some Unity DLL-files, that the TUIO API needed in order to work, didn't follow from the Unity editor to the build library. We added these files, and it worked! We could finally use TUIO in a Unity build.
We tried several solutions. For exampe, we deactivated the firewall since we initally thought that it would occupy the same port as TUIO. However, this didn't solve anything. We also tried to implement other TUIO-supported API:s. One of these was the TouchScript API, which was supposed to work with TUIO, but it didn't work either.
It was very hard to find any documentation on TUIO and Unity, and it was even harder to find a solution to the Unity build problem. After some extensive Google sessions, we found only one (!) post that even mentioned this problem (read it here). The solution there was to do modifications in the actual TUIO API (or try to implement another modified TUIO protocol). At this point, we almost gave up since it seemed impossible to solve this problem.
However, we discovered that the error message in the log file from the build wasn't telling us everything. We did some googling on this specific error, which lead us to another problem: some Unity DLL-files, that the TUIO API needed in order to work, didn't follow from the Unity editor to the build library. We added these files, and it worked! We could finally use TUIO in a Unity build.
Getting TUIO and CCV to work
TUIO and CCV problems and solutions
The first important part of our project was to get the computer vision working. Initially we knew that a previous project had utilized the webcam behind the screen in order to create a touch supported program with Unity. However, we also knew that they had not been able to build their project as a single executable file. They could only run the project with working CCV inside the Unity editor.
Firstly we set up a particle system in Unity using Unity's built in shuriken particle system. We did not bother with optimizing the particles system through shader programing at first since we wanted to get the CCV multitouch to work first.
The first system we built only had a 2D particle system on the screen. When a hand touched the screen, nearby particles changed color and/or started falling downwards. We thought this was enough to test whether multitouch would work or not. We also thought we should not develop further than this until we knew if we could build the project as a standalone executable or not.
Figure 1. Each particle within a given radius
from the touch position changes color.
The multitouch did work. Basically we checked, through TUIO, each hand that was touching the screen and called a method multiTouchUpdate(), for each hand.
However, as the number of hands reached two or more, the framerate suffered immensely. With three or more hands, the project was nearly impossible to interact with as the framerate was extremely low.
We concluded this would be fixed both by moving the particle system to the GPU as well as by creating a better algorithm for handling the multitouch. Basically, we realized we had to push the hands into a Dictionary or an Array and update for all the hands simultaneously in unity's "Update()" function.
Unfortunately, the project did not build a project that was runnable - touching the screen did not work. In addition to this, we had no idea how to actually solve the problem as a previous project utilizing the CCV/TUIO had not solved this either. However, we came to the agreement that this was the most essential part of the project right now, meaning the next step in development was to find out why the project did not build correctly and how to fix it. That will be talked about in the next blog post.
CyclooPS' background - where does the project idea come from?
1. Background
Projecting onto a glass screen as in this technological solution cannot deliver touch-functionality without additional technology. Touch functionality on a screen of this size is very interesting for demonstration purposes in several different fields. The main purpose for the visualization studio is to amaze visitors by demonstrating state of the art technology, something which could be achieved with a responsive touch system on the 4K 180 inch screen that creates an impressive visual (or other) feedback.
Therefore, with the addition of an EyeToy camera stationed behind the screen, is it possible to track touches carried out by the users. The so-called TUIO API in turn handles the touch interaction and passes it on to e.g. Unity or some other system that can make use of the input.
The reason that the touch interaction has not been showcased to studio visitors before, is mainly that the applications created for the system have been lackluster. The applications that utilize the TUIO system have not delivered visual (or other) feedback that is, simply put, good enough.
2. Problem
The VIC Visualization studio features an 180 inch 4K back-projected screen. When the studio is showcased, it is custom that we discuss the technical possibility of using the 4K screen as a touch screen. Currently, the studio cannot actually show the touch possibilities as there is no interesting and stable system to showcase it. The idea is to use one or several EyeToy camera(s) to capture the backside of the screen by tracking black blobs that appear when the screen is touched by the user(s).
At the moment, when guests arrive at the studio, they are greeted by a 4K slideshow. We aim to replace this, or add to this by developing an aesthetically pleasing graphical system. The main goal is showcasing the visual brilliance of the screen as well as the interesting idea of a touch-screen of this magnitude.
3. Implementation
There already exists a plugin for Computer Camera Vision and TUIO for Unity, which makes this game engine the obvious choice. Scripting language will be C# as we have previous experience. Graphics will be done in OpenGL with vertex and fragment shaders. The server we are building the project on has an older graphics card which supports up to DX10. Thus, compute shaders are not an option.
Prenumerera på:
Inlägg (Atom)