Development (Diagmato's Blog)

At long last, the final assignment of the third year was handed in. The forseeable future seems open to all sorts of development, in my time, for my purposes, without any looming deadlines for some write up, or small application in another programming language to ’solve’ a task I am not interested in.

The dissertation was ok - it had a lot of potential, but I cannot say I am happy with the outcome so far. The demo crashed during presentation to both supervisors, and the results of the terrain appeared far poorer than when I was writing up the conclusion documentary. Overall not too good, but the end of the dissertation does not mean the end of that project - obviously I will not be writing up pages and pages of research, implementation planning and conclusions - no. It is time for the fun stuff - uninterrupted development. The project has a lot of potential, at least personally, for creating resources procedurally. During research, the book - “Texturing & Modeling - a Procedural Approach”(David S. Ebert, Ken Perlin et al.) offered the temptation to explore more of what noise-based techniques can do, other than produce rather powerful featureful terrain.

Accidentally, my Fractal Brownian Motion implementation produced a heightmap which resembled a concrete texture from Duke3D, and also a rather blurry grass texture in place of the alpha map. There was not much more that needs to be done to start tweaking sliders to adjust colour ranges to noise values in order to produce textures procedurally.The same toolset could easily be used for generating the terrain, but hopefully much better than the dissertation’s snapshot of the application.

In honesty, when I first started the project, my interest laid solely with producing terrain. During the first few meetings, I had not heard of Perlin Noise, or Multifractals, and thus, did not start off with an interest in this area. However now, I leave university hopefully with the degree (in case it ever comes in useful, but more on this perhaps another day), and the desire to carry on with procedurals, at least to develop them to be an invaluble tool. Being graphics-based, it is a good approach to take this to shaders, and to three dimensional textures.

If you guessed a nice, Voxel terrain engine from the last paragraph, then you would be correct. GPU Gems 3 has a nice article on Voxel Terrains - I have been hard pushed to find much more on the topic, so maybe I should change that, and also demonstrate it with OpenGL - also doubling up as a nice Geometry Shader tutorial.

The future seems bright and free at the moment - just, with a loom of whether or not missing two assignments will still be sufficient to get the degree, if only to leave university trully behind.

Started: 11th October 2008


I had been thinking of how to go about this for a while – it is something I was never inclined to look into using tutorials. Previous 'experience' with this subject has only gone as far as a brief "it needs to render less detail the further away the terrain is from the camera". The quadtree algorithm gave inspiration, but I will get onto that later – for now I just wanted to see if the idea worked, THEN quad tree's can come into it when i'm satisfied. As it write this, so far it seems to have worked almsot spot on how I imagined it to.

The idea was this – divide the heightmap into a series of 'nodes', each of which would be responsible for the height values they contain.The further each respective node is from the camera, the less detail it needs to worry about.

If that sounds familiar to concepts in books that is because it is. Seriously, this is a situation where anyone with some form of experience will come up with similar, if not the same idea. I am by no means claiming I invented this idea – all I am trying to say is that this is the idea to go about it that came to mind.

1st attempt

The first attempt involved a lot of numbers and offsets into the grid. I spent some time in front of a white board drawing out what I had pictured in my mind – a large grid of 6x6 nodes with different resolutions of triangle strips depending on their distance from the camera. Each node was given a coord at each corner, and each patch was given an X and Z coord at each corner. A lot to draw, but it probably saved off all sorts of time trying to re-imagine the numbers and offsets in my mind.

The loop to fill the node data was too much for what it should be. I tried adding each height value to each corresponding node, so that rendering would require looping over each node, then looping over its patch coords with no possibility for a node to overlap its neighbour. Again, too much for what it is, and an added frustration was that collision detection would have had to loop over all nodes to find the height values.

2nd attempt

Much better. A struct full of the height values, texture coordinates, colour values, etc. This was filled up as before with the bruteforce terrain. The nodes would simply reference this array of structs with an index into the height values struct. Filling up the node data was just a case of dividing the terrain size by the (configurable) patches-per-node value, so that we are left with a count of nodes (for both X and Y). Then, each node is given an id, its centre coord calculated, and an x and z start (which is used in the rendering code to offset into the height value array to get the correct coords).


With that done, rendering the terrain was straight forward. Loop over the nodes, and use their x/z start values to offset into the height values, to plot each height value.

Amongst rendering, we check to see how far away the camera is from the current node – if it is greater than a (configurable) threshold value, we reduce the detail level (by a multiple of 2) so that the rendering loop skips over some of the height values. The further the node, the more we reduce the detail level.
What we are left with is a 1024x104 terrain which displays suprisingly similar to a bruteforced version, but with much more FPS. We are also left with 'popping' terrain, and cracks. Damn.

This is where the id's come in. Each node is supposed to be able to refer to its neighbouring nodes. The node struct contains an up, dn, lf, rt value which links to its respective neighbours. This will (hopefully) allow us to easily check neighbouring nodes for their level of detail, and (somehow) blend the triangle strips so that the crack is eliminated.

Anyhow, progress shots:


(Stage 1 - Nodes and indexed vertices loading fine)


(Stage 2 - Heights added)


(Stage 3 - For test purposes, rendered a vertical line at each node's centre coordinate)


(Stage 4 - Errornous attempt at adding detail levels to nodes)


(Stage 5 - detail levels fixed - before... (see next image))


(Stage 5 - detail levels fixed - after (yes, I am aware its only applying to the one column of nodes at the moment))

Proposed Improvements
-Fix distance checking - only first row seems to be changing detail
-Implement neighbour checking, and avoid rendering height value which the lesser detailed patch does not render
-Investigate detail 'popping'

Started: 7th October 2008

(Note - if this looks familiar, it is probably because we both have the same book - "Focus on 3D models". I used it's MD2 model for developing this loader for. I have indicated where I have used the book as I go along.)

I wanted more...character to the engine. The MD3 rendering seems just a bit too out of reach at the moment, so the idea is to get more understanding with models, using simpler formats first. MD2 was a good candidate for this.

Loading was straightforward except for the vertices. Ill expand on this in a moment.

The model file itself is simple – the header, which contains the ID, version, offsets, and count of each corresponding data sections. I simply read in the entire file, then used an unsigned char pointer to start at each offset and work its way to the count of each type. For triangles, this seems to be a simple memcpy of iNumTriangles * sizeof(SMD2Triangle).

Frames were the tricky ones, and ended up with me looking up a solution in "Focus on 3D Models". There are 'iNumFrames' frames in the model, and each has 'iNumVertices' vertices each. I had trouble thinking out the best approach to memcpy this data – the solution was to do it in small chunks – read in the scale, then the translation, then the char array for the name. Then its onto looping over iNumVertices, and reading in the vertices like so:



...Which is inside a for() loop, inside the for() loop which iterates over each frame. The vertex values are essentially compressed – we multiply the raw values by the scale + translation to get the real coords (converting from short to float). Finally, we increment the read buffer by the number of bytes just read in (plus 1 byte for a reserved value), so that the next loop iteration ends up with the necessary data, and not the data we just used.


Rendering was simple for now – I just used the first frame for testing purposes, and looped over the count of vertices:





Of course, this doesnt render the model – just its vertices (which was done in GL_POINTS mode). Rendering a more physically solid model was done by implementing the triangle data, which holds indexes into the vertex struct array, within the current frame. It is not as complicated as it sounds:



As it stands though, it seems errornous. The feet are connecting to the tail of the model, which should not be happening:




Improved 8th October 2008

I fixed the problem with the triangles connecting the tail to the toes, and so forth. I was offsetting the read pointer by the number of triangles, instead of the offset value:


(The second line used to be ucpReadBuffer += header.iNumTriangles).

The model now renders fine – just, it needs texturing, and animating.


(MD2 rendering successfully, but without textures)


Initially, texturing came as a suprise. The model I am using apparently has a skin count of 0, so the loop I was using to read in each skin (texture), wasnt looping. I added an entry to the skins for it to account for the default skin. This seems a cheap, nasty workaround, but I havent read much into it yet.

Initially, mapping the texture to the model was full of errors. The coordinates are calculated as follows:



The coordinates follow ID's little compression idea, of converting the vertices to short values which take up less memory than a float. They have provided the values to divide by to get the float, which is the skin width and skin height from the file's header.

We are incrementing the 'read buffer' by 4 bytes, which is the size of two short variables. We deal with both the s and t (u and v) texcoords one pair at a time.

The texcoords are mapped out as follows:


In amongst the vertex calls in the same render loop.

The problems I had were:

-Not referring to the correct TexIndicies[]
-Not putting the TexIndicies as the index of the pTexCoord
-Texture image was upside down

All were fairly quick to fix – the last problem took the longest – Initially I thought I had calculated the coordinates wrong, or placed them wrongly in the render loop. I noticed that the model's tail underside was where it's mouth was, which in the texture itself, is the wrong way around. The texture was originally a bitmap, which is stored upside down, so I flipped the image vertically, and finally, the model is showing as it should be. As a note, I saved the image as a tga, as at this point, I dont have a bitmap reader.


(Not referring to the correct TexIndicies[])



(Texture image was upside down)




Now, it just has to be animated!


Improved 10th October 2008

I have added a bounding box to the model. As the vertex data is loading in, I did a number of checks to find out which was the lowest and highest xyz values. Those values are then passed to a 'cube' struct which is then used to calulate a vertex array for drawing the actual bounding box. I did it this way as I seem to keep coming across the need to draw a cube, so I made a function to calculate and draw it.


(Gun turret model with bounding box)

Started: (Roughly before 19th September)
Improved: 3rd October 2008

I have had a mix of results from this. The cubemap successfully applies to the water, but rotates the wrong way, even when trying to apply an inverted texture matrix. It also squashes when the camera is close to the water, no matter which direction it looks. The only success I have had with sorting this is to render the water as a grid, which seems to stop the cubemap from bending over the large, single quad.

The improvement last night made the result much better – all I did was enable blending before both textures (the normal map is texture0, the cubemap is texture1). The blending uses the alpha of the colour we apply, but the cubemap is rendered onto the water surface and is still visible.




(Blended cubemap)

I am unsure what will happen with regards to the normal map, when it comes to per-vertex lighting. Hopefully it will still work regardless of its relation with the cubemap, and the blending.

Proposed Tests
-Render the water as two triangles instead of a quad (an attempt to fix the bending cubemap)

3rd October update
Fixed cubemap distortion by rendering water as a 16x16 triangle strip (16 triangle strips with 16 patches). The idea is that, when it comes to making water wavy, it will likely do this by manipulating the heights of each vertex that makes up the water, so two birds with one stone, hopefully!


(Cubemap displaying without warping)

Improved: 3rd October 2008

Previously, terrain was textured by applying a tiny texture to each square created by a series of 4 vertexes in a triangle strip. This was improved so that a larger texture is tiled across the terrain as a whole, in that we calculate each vertex's texture coord as such:

vertex[x][z].s = x * (textureTileCount / terrainWidth);
vertex[x][z].t = z * (textureTileCount / terrainLength);

We only need to calculate the bracketed equation once, so it was placed outside the loop for generating the coords, and its value assigned to a variable. No point in the computer doing the same homework twice, aye? :)

To test this, I adjusted the tileCount. The results were successful – a tileCount of 1 ended up with extremely thick grass, a tileCount of 20 ended up with much thinner grass (the texture was a rather detailed grass texture).

Here it is in action:


(Before)


(After)


Proposed Improvements
-Multitexturing for added detail (DONE)
-Normal-oriented texturing (for cliff faces, etc)
-Lighting support




Multitexturing

The next improvement for this concept was with multitexturing support - getting surfaces below the water level to show a muddy texture as opposed to a grass texture. This came somewhat close, but there was no grass texture. What I tried was to only render the coords for the second texture (the mud) and not the grass. This caused anything above the water level to have a seriously stretched version of the mud texture.

The solution I believe is to plot out the textures as usual, but adjust their blending, so that the mud texture shows through the grass texture.

What I have left this concept with (for now) is a terrain that is blending between rather clean looking grass, and mud, as seen in the below screenshot (going forward in time a little here):

Started: 3rd October 2008

Last night, I worked on the TGA writing function. It only supports image type 2 at the moment (raw, uncompressed). To write to this format, we just need to setup the header with the necessary values (most are 0, except for image type, and the width and height, which are the values of the viewport (iViewport[2], iViewport [3]). Then after the header, we just write the image data, with the size obtained through the width * height * bytes per pixel (3 in this case). The image is read from the OpenGL Front buffer, which will gather the viewport rendering including both perspective and orthographic.

To take a screenshot, currently the controls are ctrl + right mouse click. But this will keep taking screenshots as long as they are both held down, and unless the person controlling it has the reflexes of an agent in The Matrix, they will get a number of files.

The save tga function does the work of checking for a unique file name. It iterates an integer, prefixes it to the filename, and tries to open it. If it successfully opened a file, then there is already a file with the name it is trying to use, so it increments the integer, and tries again until it fails to open a file, and thus creates and writes the new file. This makes it handy for dumping screenshots for exporting to a movie clip (with videomach for example). This needs to be improved – currently, the more screenshots there are, the longer it takes for it to search for the next filename iteration, and this quickly becomes noticable when the computer is trying the same process on every frame.


(first ever working screenshot using in-engine code)

Proposed Improvements

-Keep a record of the last successful iteration so that the computer does not need to test potentially 50 files just to save the next one, each frame.
-Add support for saving RLE-encoded tga's
-Add support for taking one screenshot, or a series of screenshots



Improved - 3rd October 2008

Texture class now remembers the last file number saved, so it only needs to check once if a file exists before writing it.

I really need to keep updating as I go along. The last entry was 13th September, which, checking a log file, all sorts of concepts have been worked on since then.

So for the next few blog entries, I have started with a manually written date to indicate when they were worked on.

Various concepts were toyed with but didnt get far. Others are actually worth mentioning. Here's some (in no particular order):

- Implemented extension loading and GLSL support
- Implemented cubemapping, tested on a spherical object. Successful, but scrapped.
- Implemented sound using SDL_Mixer
- Implemented runlength encoded TGA loading
- Implemented screenshot function and TGA writer
- Implemented font functions (for displaying text in-scene)
- Experimented with half-life .mdl files using source code from their SDK (scrapped - will write my own version when needed)


GLSL Shader attempt at lighting



GLSL shader attempt using specular and diffuse lighting (emphasis on attempt)

I was growing tired of the OpenGL scene looking so...lifeless. Just a floating terrain with a red background. It was time to make it begin to feel like a game engine.

I declined to rush off in search of tutorials - the first idea was to get a huge cube surrounding the 'world bounds', and texture them with a skybox texture. So I did, and, the results were what feels like a giant wall with a texture, statically in place where you could walk up to it, as if its some backdrop in a movie set:






Not good. Tutorial time.

After a read around, I learnt that the skybox is actually tiny, and just wraps around the 'camera'. When the camera moves, so does the skybox, giving the feeling that it is infinitely far away:





So, as the camera is moved, it passes its XYZ position to the skybox class, which updates the position of the skybox. Depth testing is disabled before the skybox is drawn, then enabled straight after - if we didnt do this, then all we would see is the skybox, and everything else would be hidden, as in reality, the skybox is infront of (mostly) everything else in the scene.

That iceberg looking heightmap rendering from before was wrong - its not supposed to look nearly like that. Problem was that I was setting each height into a signed variable, so anything above a certain height became a negative height.

Uni draws closer, and my project is procedural terrain generation. I decided to improve on the heightmap rendering:



In a sense, it accidentally resembles mars, with its darker area's believed to have one held water - compared to the next image, where the water level is risen:



It has so many improvements needed - when water rises, all puddles in the area dont miraculously rise at the same time - it flows around, rather than raises from the ground in sync with other puddles (unless its a very wet field, or sand). But working out the physics to get water to flow is well beyond me.


There is one other advance in this 'snapshot' - the GUI. I have been working on a GUI framework in openGL - you instantate buttons, panels, etc, and, say, for buttons, you assign them a function pointer for them to run when onClick, onMouseUp, onMouseDown, etc. Its designed to replicate features from other GUI's, and the actual code to instantiate and set it all up is similar to Java. It was suprisingly easy to setup how a button calls a function - just need to adapt it to be able to work with methods of other classes, which would make the GUI so much more flexible to work with.

An added gem is that, when the button is clicked, it changes its texture id to use a 'select' version of the texture. Just need a good artist to draw up some buttons properly.

Spent a weekend over Terry's - amongst a good bash of gaming, was a very good coding spree. When the people working on the same project are in the same room sharing ideas and a fresh pair of eyes on each others errors, things get done VERY quickly.

The biggest problem seemed to be the Tilegrid. At the moment, I have a quick testing function which pumps a struct array of vertexes full of x,y,z coords in a grid fashion. The individual tiles then point to each vertex which represents their corners. This way, when it comes to the map editor - if a tile is raised, then its adjacent tiles will automatically slope up towards the newly raised tile as they share the same vertices as the raised tile's closest edge to the said tile.

If you're trying to picture what I mean by a Tilegrid - think of Red Alert 2, Tiberian Sun, and various Isometric games like the SimCity series.

Arran did a good slice of work on the building models - some nice stuff there.

Terry did a good job with unit interaction - so far a marine moves where the mouse was clicked (on the terrain). It's model animates appropriately too. It just needs to implement 'picking' in order to select the marine, and tell it to move, rather than the marine always responding to the mouse.

Ive made major progress with the GUI system - ive tried to make it as straightforward as I can, following the GUI setup rules of other languages (it has a close resemblance to Java). So far it supports panels, buttons, and buttongrids. The latter simply makes it easy to add a grid of buttons, and itl work out the sizes and coordinates automatically.

Here's the all-important progress shot :). The texture sucks, but im no graphics artist. I couldnt find a martian ground texture, so I copied a tiny square from an image of the surface of mars. The GUI has a nuisance white placeholder texture, but it is working fine - I just havent made a sidebar texture. The buttons in the bottom left are there temporarily for testing the raise/lower functions of the tiles.

For a change of scene from PHP for a bit, I had a huge coding spree with C++. Ill tell you now, the more I use this language, the more I like it.

I was getting bored of rendering a quad or two (or the un-finished Quake BSP classes pumping out a bunch of coloured brushes), so I decided it was time to give the graphics card a bit more work, and render something a bit more interesting, like a natural landscape.

So, I did!



I wrote a reader for a .raw image, which is nothing but the actual 'raw' image data. Just fread() the pointer to the file, for the size of its width * height, and you have the image data.

The heightmap is somewhat interesting - the colour value for each pixel denotes a vertex's height. So, if the pixel at 37,48 had a colour value (as a byte) of 127 (greyish), then the height value (as a float) would be 127/256.0 = 0.49609375. So, when rendering the heightmap - we glVertex3f() the x position (from the for() loop), the z position (from the sub-for() loop), and use the height value we calculated earlier for that individual vertex's height. So in this case, it would be:

glVertex3f(37, 0.49609375, 48); //X, Y, Z

For the ONE vertex. But bare in mind that, for terrain, were using GL_TRIANGLE_STRIP, which wants 4 vertices specified in a 'Z' pattern:


//Top left
pointHeight = this->pointHeights[x][z];
glVertex3f(x, pointHeight, z);

//Top Right
pointHeight = this->pointHeights[x+1][z];
glVertex3f(x+1, pointHeight, z);

//Bottom Left
pointHeight = this->pointHeights[x][z+1];
glVertex3f(x, pointHeight, z+1);

//Bottom Right
pointHeight = this->pointHeights[x+1][z+1];
glVertex3f(x+1, pointHeight, z+1);


Not too bad when it comes to rendering, but the bit that had me stumped was actually filling up the pointHeights[][] array. For ages I was referencing the vertex in the texture data as texture->textureData[x*z], which was wrong. It should be:

texture->textureData[(x*terrainWidth) + z]


So to summarise, filling the values and rendering the values are the same for() and sub-for(). The worst part was getting the maths right when loading up the pointHeights array.


This is not meant to be a tutorial on it - maybe one day ill write one though. At the moment im certainly not convinced that ive done it as efficiently as it could be.

Now the question is - why on earth cant I get this .raw texture to map to a quad?!?

This category is rather...lonely. It has so far had rants when I was working on globetron, yet that was a couple years ago!

I have been dabbling in C++ for quite a bit - specifically with OpenGL. A friend got me into it, and helped with loads of teething problems. Ive gotten used to the language quite nicely now, but i'm certainly no John Carmack. Either way, i'm going to use this blog category to focus on creating a log of what ive done, when, etc.

As a primer, so far:

-Used win32 and OpenGL, plowed through some of Nehe's tutorials.
-Used SDL and OpenGL, created a basic map loader (without textures) for Quake 2, 3, and half life bsp files.
-Written functions to load in a .tga (datatype 2), and map the texture to a quad. Supports a number of textures, so each quad can have a different texture.

Not exactly the most amazing progress so far - well, pretty much not amazing at all especially if you are seasoned in OpenGL development. But yeah - its progress, its been fun, and it's been rewarding.

So now the next challenge - ill hopefully have a bsp loader that also loads textures, applies them correctly, and may even support half life's .map format as well. That is of course, after/as well as tidying up the actual project code.

The release candidate for the new phpbb software has been released, and this prompted a rush to the site to download the software, and install on a test server.

Im amazed just how well it works! The forums were converted without errors, from phpbb2, even though it had a few homemade hacks for various site features.

As for the code - it sure is a darn site bigger than phpbb2, but far more thought about. It has features which ive been meaning to plonk into the site for years, but "never got around to it". Now its just a little adjusting here and there, and it can be used site-wise.

I'm already in the process of implementing it site-wise, ready for the final release itself. This really seems a lot easier to do than getting phpbb2 seamlessly part of the site.

Stay tuned!

The bugs have been splattered, and the theme finally done. It looks a bit too blue for my complete liking, and parts resemble lego blocks, but its not bad, I guess.

Now to sit here and wait for customers/get advertising and attract customers! And go through University, and work on black0ps...

If there is one thing I hate about programming, it's finding bugs which seemed to be overlooked, at the last minute. This one isnt a bug, its more, something I completely overlooked. So, thats another few hours of development required before release.

University starts soon, and I dont want to spend free time trying to get this, which was supposed to be released weeks ago, finished. Being a lone developer assigned to a project really gets dull quickly. Well, never again. Globetron is the last script I make solely, as for two, I dont have the free time to manage more than two sites, mainly the bug fixing.

The continuous editing of other files to make way for the new changes, especially the database ones (although not the most difficult, as the script will compain there and then), is the biggest downer right now. In this case, its editing email templates and getting the email function to replace the variables as needed. Not difficult, but again, monotonous. If this had been a team effort, it wouldnt be so bad, as some of the other teething problems could be solved at the same time.

Also the script is complaining about having two email functions on the same page. Darn. Shouldnt take long to fix though, I certainly, absolutely hope.

This project has taken well over the expected time to create, and I have vowed never to start a new PHP project unless it's with a team of people. It does get fairly lonesome doing an entire piece of scripting by yourself. Would certainly help if someone else was here to do some of the monotonous functions.

Anyhow, I have just uploaded it all to the server, and guess what. Those darn emails dont seem to be working again, and sessions is tripping over itself when setting cookies. Note that this problem didn't occur on the development servers, else I would never have uploaded it...

It really takes the cake when one server acts different than the other - I dont have the settings which the live server has, but I cannot see why the setcookie problem is happening on the live server, and on neither of the development servers.

I'll get to the bottom of it hopefully. Tomorrow, really hopefully. At least, before University starts next month.