Setting up first version of particle system on Monstro.

The bad thing about going from OpenGL and vert/frag shaders to Compute shader is that we kind of have a lot of knowledge on OpenGL that we might not be able to use. Of course, graphics programming is going to be sort of similar but it still feels like we have done a lot of work that we just scrapped to try to move over to compute shading.  (Of course, all the interaction programming as well as the theories behind our graphics programming is still intact - which is a lot by itself, but it feels a bit annoying to have to get rid of so much work anyways).

Nevertheless, we did some reading and looked at some examples and used some pieces of code that we could manipulate into our own project. Ultimately, we learned a lot from this step - as we did when setting up the first version of the OpenGL system.

For example, we learned that modern GPU:s process data points in batches of 32, which means that our particle system should be a multiple of 32 for the highest efficiency.

We also decided to move away from the idea of using textures to store the particle data. This is not because it would be impossible to do with the compute shader. It is rather because we could simply use a buffer and achieve a very high particle count anyways.

We decided that the 3.2 million simultaneous particles (at 60FPS) we achieved on the first try by using ComputeBuffer basically was enough and we decided to, at least for the moment, not explore the possibility of using textures as storage. At this point we have already worked so many hours on this project so it is really time to leave the small optimizations and try to achieve something that works.

The next post will show you the particle system and the not-so-well-working hand interaction.

Compute shader basics and how using a compute shader could move our project forward

The advantage of now having a computer shader to work with is that we don’t have to worry about working out a way of identifying vertices. The indices of the vertices are are built in, hence every vertex can be identified. Another important advantage with the compute shader is that we can send more advanced datatypes inbetween the CPU and GPU, we could basically just send an array containing the data needed on the GPU (e.g. not having to encode hand data in textures). This concludes in that we hopefully could perfom the physics calculations and control the behaviour of the particles relatively smoothly.  Most of the work from here on has included learning the syntax of Unity’s computer shader. The debugging is kind of hard and time-consuming since the error-messages from the shader are not very specific or extensive. This forced us to come up with multiple creative ways of debugging.

We will use our newly gained sort of shallow knowledge on compute shaders to attempt creating a particle system as well as moving the particles on Monstro. Hopefully we will get all of these as well as multitouch working within a reasonable time.

Compute shader and migrating to Monstro

Due to the problems mentioned in the previous post with Unity and the lack of support for GL_POINTS, we were desperate to find another solution to keep pushing the project forward. We had earlier thought of compute shaders which we tried out on Niclas laptop and they seemed amazing. Sadly, the Z1 server which is the home for our project has an older GPU which only supports up to DirectX 10.1 and thus not compute shaders.

There is another server available in the studio, Monstro, which has a GTX 780 Ti and therefore has support for DirectX 11 and thus compute shaders. Up until now we wanted to stay away from this server because its desktop is configured by multiple lower resolution desktops and thus leading to problems with fullscreen applications, like the .exe we intend to build.

We decided to give Monstro a shot and halt development on the Z1 server. We also considered moving the project over to WebGL (which we know supports GL_Points as Erik has created a particle system in WebGL before), but this idea died rather quickly since we did not want to implement the Tuio API again for a web based application when we already had it up and running in Unity.

As predicted, Monstro’s 4x desktop configuration does not allow the standalone .exe to run in fullscreen. We tried several different approaches in order to make it fullscreen, but without finding an appropriate solution. Our first idea was to force the application to run in 4096 x 2400 resolution by telling it to do so in the scripted launched on startup. This made it fullscreen, however only on one desktop, leaving the majority of the screen empty.

In our second attempt we tried to configure a multi camera setup in the Unity scene and thus force the application to run two different fullscreen modes where the cameras render one half each, synchronized. Again, we got close to finding a solution but not quite. The two views were misaligned no matter how we tried changing clipping planes and positions for the two cameras. This solutions seems too technical and hard to get working. Probably we will not explore this method further

By the end of the day, we managed to have the application up and running in 4096 x 2400 resolution, however in windowed mode. If we align the window properly, it looks sort of alright but it is not a solution we want to keep by the time we finish the project.

The solution we believe will work in the end is somehow forcing the application to run in 4096x2400px in a borderless window mode, which spawns at 0,0 in the leftmost desktop. That should mean it spawns over desktop 1 and 2, faking a “fullscreen app”.

However, we believe what we had at this point was good enough for the time being as it accomplished the goal. We decided to keep working on the compute shaders instead in order to have a particle system working with multitouch which is what the project is really about - not setting up a fullscreen app on Monstro...

The death of Z1-server and OpenGL: realizing that OpenGL in unity will never support our solution.

This post is extremely important but won't have any images as it is only a long explanation on why we had to move away from OpenGL in Unity on the Z1 server. The solution we had intended, which consisted of storing and encoding particle values in two 2D textures (using the color values as 1) coordinate values and 2) velocity/direction values) was a very good idea.

The theory was that we could either 1)

use the built-in gl_VertexID (integer) to decode that value from an int to a vec2 value which would correlate to a position in the texture. For example, if the texture is a 5x5 texture (25 particles) and you have gl_VertexID = 6 (first vertex has index 0), it would mean that that particle’s correlated position within the texture is (1,1).

or 2)

Give each vertex a custom attribute which was a vec2 that correlated to that particle’s index in the texture - so basically the same as method 1) except that we instantiated for each vertex before our shader code even started exectuing so to speak.

Unfortunately, because of our newly gained knowledge on how particle systems and OpenGL works in Unity, both of these solutions are impossible due to two reasons:

1) Unity does not allow custom attributes for vertices (at least in the OpenGL versions available to us) - which means we could still solve our problems with gl_VertexID as described above

2) Unity does not support GL_Points mode. This ruins everything and requires a longer explanation.

When doing per-vertex operations, which we require since we need to calculate new directions and velocities for each particle, the shader has to run in GL_POINTS mode. This means that, since Unity by default runs the shader in GL_Quads mode (changable to triangles or lines), each particle consists of four vertices.

The way these are arranged are unknown to us. This means that if we correlate one of the vertex points, which we hoped was one particle, to our texture, it makes no sense for the GPU - it wants four vertices per quad.

There might be a solution where we find out how the vertex IDs are stored in gl_VertexID - for example the first quad perhaps has the vertex indexes 0,1,2 and 3. However, it seems like we are moving farther and farther away from our idea of optimizing the particle system (for example this obviously requires four times the operations => potentially four times fewer particles).

When I (Erik) realized these problems late at night in the VIC studio, I started to think that I would have explain this in detail to the others ASAP so we could figure out a way to move on from this point. Probably, I assumed, we would have to reconsider the idea of using the “Monstro” server with the 780Ti GPU:s which have dx11 and thereby can utilize compute shaders. In other words, scrap all of our texture scripts and shader code. Woah.

Changing priorities in order to keep moving forward

As we did not have much success neither with compute shaders or VBOs we had a hard time moving forward with altering the particle system in a way which would fit our goals. We therefore decided to put this problem on hold for a while and instead work on other essential parts of the projects.

Something worth noting here is that we most likely have to keep investigating how to save the Particle System data in the VRAM of the GPU. One way to handle this is to keep investigating the VBO and possibly get it up and running to give us the desired effect. An alternative would be to find another solution to handle this problem. We have read a very interesting article where they used a texture to encode particle data. This could be a very quick way for us to store the particle data and easily access it in the next frame. The blog post can be found here: http://nullprogram.com/blog/2014/06/29/

Since we stopped working on the particle system for a while, we decided we should start working a bit more with handling the multitouch in the shader. We realized that this had to be done at one point anyways.

The problem with passing data to the shader in Unity is that we are very limited in the amount and types of uniforms we can send. For example, Unity cannot send an array of data to a GL shader program. So we had to get inventive.

We realized after a while that a texture is basically a big matrix that can store data in a sort of efficient way, if we are smart about it. Basically, we defined a maximum number of hands that we allow for multitouching the screen. For now, we set that number to 20, but it could be increased in the future. The shader needs to know a couple of things about the hands to be able to do calculations. The main thing is of course the current position. But, the initial position of the touch is also relevant - that way we can track how much the user has moved the hand without lifting the hand off the screen.

This meant we needed a 20 by 2 texture to store our data. Since each pixel in the texture consists of a vec3 (or vec4 if using RGBA), we are able to store quite a lot of data in a quite small texture. The first position in the texture, [i , 0], contains the x and y coordinates for the i:th hand. The second position, [i , 1] contains the intial position of the hand. This means that in the red component of the vec3 in the 0:th position contains the x value and the green component contains the y value. Same goes for the 1:st position.

We realize we could have had an even smaller texture (20 by 1) by encoding a RGBA value as x, y, xOld, yOld. However, with the way we have done it now we have reserverd some space in case we want to send some other data at a later stage. Also, this is an optimization that is probably completely unnecessary as the amount of data we are passing is already small. However, we will consider moving to the 20 by 1 texture at a later stage if we find no use for the now zero-value blue position and alfa position in the RGBA-array.

We are now working on decoding the hand position data inside the shader. However, it is difficult as debugging on the shader is nigh impossible. A sudden realization we got while writing this blog is that the reason we can't decode multitouch data at this moment might be that the texture expects values from 0 to 255 and not floating points between 0 to 1... We have to explore this...

Investigating other ways to create an efficient particle system

After defining our goals with the particle system, we soon came to the conclusion that simulating space orbit and recreating gravity fields was a very complex task with ordinary vertex and fragment shaders.

Since we did not have a solution where we could update the particles' positions on the GPU, we realized we either had to solve that problem or create a function based on time to simulate gravity. However, this is mathematically very complex and requires a numerical solution. Additionally, we would have had to store the time frames for where hands had been in the past, to be able to "remember" how the particles had been simulated up until the current time.

This meant that we after a couple of attempts using the Vertex and Fragment shader started looking for alternate solutions. First we looked at how to implement and use VBO:s (Vertex Buffer Objects). They are normally used to upload vertex data to a GPU device. Since they use the VRAM on the GPU side instead of regular RAM they offer much better performance when used correctly. Unfortunately, we couldn't find any good examples of how to use them when scripting in C# and Unity, so we continued our search for other solutions.

Screenshot of a simple particle system done in Unity with a Compute Shader

We then looked at other types of shaders. We found that compute shaders is a very good and quite straight forward solution when creating a system with a lot of particles. Basically, compute shaders is a more modern technology compared to vertex shaders where the shader takes advantage of a large number of the GPU:s parallel processors for computing purposes, which enables high-level graphics. For example, many particle effects and systems in modern games use compute shaders to do graphical calculations. We found a lot of documentation on compute shaders and particle systems, which was promising. In other words, this seemed to be the solution we was looking for.

Due to how the compute shader works, it requires the GPU device to be fairly modern. In particular, one requirement is that the graphics card supports the DirectX 11 API. This was a problem since the GPU on the computer connected to the big 4K screen was a Quadro FX 5800 which didn't support DX11. So as unfortunate as it was, we couldn't continue with this approach and had to look for other solutions.

Defining our goals with the particle system

Now that we had done some computations on the particle system and had gotten our multitouch to work, we were faced by a real problem: We needed to specifically define what we wanted to do and even more specifically how we wanted to affect the particle system based on our input.

We came to the conclusion that we wanted to simulate physics behavior in space. This means we wanted to make the hands touching the screen into big planets with high mass and at the same time give the particles a very low mass, causing them to orbit our "planets". Multiple hands would create more and more interesting behavior from the particles. This meant that we had to create a function based on time that described the orbits around the hands.

At this point we were very tired of discussing what to do with the particle system, so we decided to start implementing things during the next work session.