Library Woes

My least favourite part of game development (so far) is finding, building and linking libraries into my projects, especially when the language I’m using isn’t industry standard (D). The other day, I ran into an issue where I tried expanding the list of libraries I was using in denj and instead broke all of them by accidentally upgrading them and rebuilding them all. It would pay to mention that all the libraries I’m using so far are a part of a collection of libraries called DerelictOrg, and as such all rely on one utility library called DerelictUtil. This, I found after hours of searching, was the problem. All the libraries I was using (DerelictGL3, DerelictSDL2, and DerelictASSIMP3) depended upon DerelictUtil version 1.9.something, but the version in the master branch of the git repository was 1.0.3. This meant that everytime I tried to link I’d get a convoluted linker error about an undefined symbol, _D8derelict4util6loader15SharedLibLoader23configureMinimumVersionMFS8derelict4util6loader16SharedLibVersionZv
which demangled is
void derelict.util.loader.SharedLibLoader.configureMinimumVersion(derelict.util.loader.SharedLibVersion)
(This is here so google can find it when other people have the same issue, sorry)

Eventually I figured out that the proper version of DerelictUtil was in a branch ‘2.0’, built it with dub build, linked it, and continued on with my life. This wouldn’t have been a problem if I were using the project manager for D, ‘dub,’ because it fetches the correctly versioned libraries from a central repository and makes life easier, but it imposes a file structure and requires extra files to describe the project so it’s not for me.

I recommend anyone building derelict from the git repos to keep this in mind.

Read More

Two Thirds Through

The end of trimester four of six of uni is nigh. Probably time for some reflection. This trimester has involved a lot of AI and rendering things. It started with the Killbot tournaments, in which we all developed AI driven bots that were put into an arena to fight. We were given the arena and an API and told to make bots. The API was in C++ and I, having been obsessed with D for at least a few months, decided to port the API. This was a huge lesson in binary compatibility (I wrote a blog about it) as I quickly learned that the binary format of classes in the standard library are far from standard. So I spent a fair chunk of time reverse engineering std::string and std::vector and writing D interfaces to them. That was the first time I’d ever reverse engineered something so I was pretty happy with that. On top of that was actually developing the bot. Developing the bot taught me a lot about implementing AI given limited information. I hadn’t used A* much before so this was a great exercise and really helped me figure just how it and it’s heuristics worked. That said, my bot didn’t do fantastically, but it navigated the maze (one of the arenas) pretty well and looked great doing it (Part off the API exposed was for line drawing and I abused that a bit).

Next major thing we did was a collaboration with a bunch of other disciplines (mainly games design) to make a game based on trust. The game I worked on was birdlife. This was quite challenging as I’d never had to work with a brief centred around an emotion (or whatever trust is). I’d also never had a game put in an art gallery before, so there was a fair amount of pressure to make this game good. After the first week of doing things we were informed that our game didn’t quite match the brief so we had to redesign most of it. Luckily, half of the programming we’d done was kept (mainly the flying stuff), but we did lose the other half because it became irrelevant which taught me that you should totally get your game design stuff concrete before starting things, especially so when working with a relatively large team as we lost some models as well. Birdlife had a pretty strong æsthetic from the get go, low-poly and flat colours. This meant that prototyping assets that looked at least a little good was really easy. In fact, some of the prototype assets I made made it into the two playtests we did before the gallery. As well as prototyping assets, I also spent a good deal of time figuring out Unity’s shaders so we could use post-processing effects. The two playtests and the session at the gallery really highlighted *just* how important communication and feedback are. People didn’t know how to play our game despite our efforts to communicate things. They did however show some of the reactions we were hoping for, so some of our efforts *did* pay off, but we didn’t try to teach the mechanics as well as we could have and that was our game’s biggest downfall.

Throughout the trimester I worked on a few small programs on the side just to try and get my head around some things and as such I learnt quite a bit about rendering and about some general programming things. One of the projects is a realtime GPU raytracer (see previous blog) which involved drawing animated, reflective spheres and planes using raycasting and intersection functions. Another is a small project that just drew some simple geometry, which I used to learn about using framebuffers and picking with opengl. I also wrote a simple program that drew two ‘scenes’ through portals on a moving cube, which I made with the intent of learning how to use the stencil buffer. Since working through these small things I’ve become a lot more comfortable with OpenGL and have begun working on an engine to put the knowledge I’ve gathered to use. All code is available on github.

Read More

GC Bit My Finger

I was working on my engine the other day (I’m working on an engine called denj, by the way) and I was getting ready to start a graphics module. My first task was to separate the opengl context creation code from the window creation code as much as I could (SDL requires the SDL_WINDOW_OPENGL flag to be passed to windows upon creation if you want to use OpenGL but that’s not so bad). So I started a new Renderer class that was going to hold the context and do render-y things. I moved the context creation code from the Window class, tidied a few things up and added declared a new renderer in my graphics test function (pretty much main wrapped in a try-catch). It built. Awesome. Then I go to run it aaaaand… seg fault. What?

After some messing around and inserting logs into semi-random bits of code, I discovered that it was happening on an opengl function call. Not just any function call, but the first function call. Obviously, it was some kind of problem with opengl. On a hunch, I added some log calls into the constructor of my new renderer class in case for some weird reason it wasn’t being called or was failing or something. It was fine. Then I added a log call in the destructor of my new renderer class and found my problem. My renderer and the context it managed were being destructed immediately after being constructed.

As it turns out, D’s GC is a lot fluffier in terms of behaviour than what I expected. I expected that if you created a class, it would stay active until end of scope. Which is what D’s structs do, but apparently D’s classes are different. If you have a single reference to a class and never use it, the GC will pick up on that and remove it. In a regular every day program this mightn’t be as much an issue, but my Renderer class was interfacing with a C library, OpenGL. OpenGL, being a C library, isn’t aware of the GC nor is the GC aware of the global state contained in OpenGL. Therefore, my using of the global state managed by my renderer was ignored as use of my renderer.

So I knew what the problem was, so what was my solution? Gross.
removedasapObviously this is super temporary but it fixed the problem enough that I could begin working on the actual API of my renderer. I could probably remove the reference as of last night considering that I’m actually using the class for rendering things in my main loop now. But I’d still like to find some way to keep it alive just incase I want to use raw opengl at some point. Although, my API is already nicer to use (not that hard) so that probably won’t happen soon.

Closing thought. If you plan on using a garbage collected language, make sure you read up on what the hell the garbage collector actually does and when it does it. Otherwise you’ll have weird bugs for (seemingly) no reason.

Read More

Stencil Buffers Are Rad

I started playing with the stencil buffer the other day and apparently it isn’t as mystical as I once thought. A couple of years ago, back when I was still learning how to use openGL and couldn’t upload data to a vertex buffer without failing miserably and copypasting code from everywhere, I tried to use the stencil buffer, and I failed miserably. So I gave up and tried learning about textures instead. A few days ago I got distracted and decided to try again and I totally figured them out. Turns out it’s actually really simple and I was just being bad. After a few hours of work, I had these rendering.

stencil7 stencil5 stencil3

Pretty simple to do but I think they look pretty cool. Next day I decided to take it up a notch. So I tried portal rendering in 3D. And it works. Not crazily complex, but cool for the time I put in.

And I managed that with (almost) no tutorials. At one stage things were drawing weird, e.g., the scenes in the portals were drawing outside the portals sometimes, and I just couldn’t figure out why, so I looked up some tutorials to see where I went wrong. Turns out glClear doesn’t work on buffers that have write masks set to zero (which I was doing while rendering). Protip: reset write masks after messing with them. It seems super obvious but it wasn’t at the time so keep that in mind.
But yeah, I’ve determined that stencil buffers can make things look really cool relatively easily. 10/10 recommend looking into them if you haven’t already. I will most certainly be messing with them more.

Read More

Birdlife Update

Birdlife (the game I was working on last post) is coming along nicely. Since the last post I’ve done a lot of messing around with the flight dynamics to get them feeling right, and I’ve attached said flight dynamics script to some new entities that can fly around of their own accord. In the game now are crows, cars, a chick, a buttload of particles, decorations, and gameplay. The crows and cars fly/drive around on waypoints; the chick follows you if you tell it to by chirping at it; and everything can die. There are end states now. So many things exist now. Here, have a video.

This video is a little outdated now, but I haven’t gotten around to making another. In the current version, everything (except the map) has a model – the bird, the chick, the crows, the cars. There are postprocessing effects now, and the map is bigger. I hope to have a new video soon.

Also, last week we did a playtest of the game. What we discovered was, no one knew how to play our game. But to be honest, we kind of knew that was going to happen. We put a bunch of effort into making it look prettier, making the flying feel better, making tools and shaders, and making sound work (which is also pretty good, if I do say so myself) but ran out of time and didn’t put any feedback into it. So there were people not knowing how to feed the chick, what the chirp did, that the chick existed or that the game was able to be won. We have another playtest on wednesday and hopefully we’ll have most of that fixed. I’m pretty sure we’ve dealt with at least some of it. But there’s still work to do. That said, I’m pretty happy with how it’s going so far and I look forward to seeing it done.

Read More

Flight Dynamics are Hard

One of the projects I’m working on at the moment is a game in which you play as a bird, and do normal bird things. One of those normal bird things is flying. Flight is hard.

When I started I figured that I could just go to wikipedia and there’d be two or three equations that I could semi copy-paste into unity that would just kind of work. Wikipedia certainly had some equations for me, but they weren’t the nice n’ simple ones I was hoping for.

AerodynamicForce

This is an equation to calculate aerodynamic force. It’s a surface integral involving tangential stresses and pressure differences. While I’d normally be all about figuring out how to solve surface integrals and learning about pressure and what not, time isn’t exactly on our side. So I kept looking. Alas, I found a more inviting equation.

lifteq

It looked more like something that I could easily drop into unity, so I used it. Found here and on wikipedia. This did require a gas density, but that was easy enough to google. Somewhere on google mentioned 1.229 as the air density at sea level, so I just used that. Also provided was an approximation for the lift coefficient for thin aerofoils, which thankfully was just Cl = 2πα where α was angle of attack. Despite how simple it was, I still managed to get it wrong and for at least two days I was flying with a negative angle of attack, which meant that when I pulled back I’d fall faster and when I leant forward I’d float away. But I WAS flying and that was the main thing. I also had an issue with NaN’s whereby if something was negative (I think forward velocity) you would be flung so far that floats would stop working. I managed to fix that when I figured out the angle of attack though. At that point (and after multiplying air density by 50) I had this.

Kinda dodgy, but also kinda working.
After about a day of tweaking (and playing with particle effects, materials, and adding controller support) I had this.

This felt much better. The gliding worked (pretty sure air density is still ~50x that at sea level) and leaning forward didn’t send you skyward. Admittedly there are some hacks afoot here (see last blog). One hack is a small correction that takes the difference between the forward vector and the velocity and applies that back to the bird. This (broke physics and) stopped the bird from maintaining momentum perpendicular to the bird, in other words, if you made a sharp turn you wouldn’t keep going sideways until you hit something. Another is the fact that I’m multiplying air density somewhere between 50 and 100. This made lift a lot easier to generate with small wings and stopped the bird from just falling out of the air at 9.8 m/s². Considering that these hacks are totally necessary to make the game not feel like a bad bird simulator, I’m completely ok with having suffocatingly thick air.

To finish, here’s something else I worked on other than making bird flight not bad.

Read More

Hacking Is Totally OK In Moderation

I used to have this preconception that games (that people actually bought) were meticulously planned and hand crafted masterpieces that were produced by programmers that knew their API’s inside and out, and that never made mistakes and never hacked out features, or at least never shipped them. Now I know better.

This article by Gamasutra talks about some of the nasty hacks that game developers have used to get a game out the door. Admittedly, some of the hacks were necessary because of hardware limitations and whatnot, but one of the hacks involved mutating the EULA to cause a buffer overflow that overwrote code with new code that allowed the developers to patch their game. That’s impressive and all, but kind of scary as well.

Then there’s the talk by That Game Company about their sand rendering in Journey (available here). Most of the talk is dedicated to the hacks they employed to get the sand to look ‘right’. Their method for deriving the lighting model they used was quite literally “Keep trying random things until it looks good.” Which they did. Their lighting model is as follows.
journeylightingmodel

Finally there’s this. A testimony by someone whom worked on graphics drivers about how some AAA games ship with some not-so API compliant code. Which to be honest isn’t entirely surprising.assassins-creed-glitch
My point is obviously not that using API’s wrong is okay. My point is that if industry professionals use dirty hacks to get games out the door, and do random things until things work instead of sitting down and planning out algorithms and what not, I’m not going to feel as guilty when I do the same. Obviously I’ll try to do things the right way, but if I have to finish something quickly and doing things the right way isn’t working, hacks will be had.

Read More

Raytracing is a Thing

Yesterday I started a new project (not a proper project, just a new folder really) in which to test how quickly I could get a window open and an opengl context ready to mess around with. Turns out that that doesn’t take very long, so I spent the rest of the day just learning about neat D stuff (and being sick). For example I learned that I didn’t have to specify ridiculous command line options in order to get libraries to link. Instead I can just use pragma(lib, "Library"), which is akin to a visual studio vendor extension except standardised and actually part of the language. I also learnt a bit about tuples in D. D tuples are kind of nuts to think about coming from c++ land. They could be very loosely tied to lua’s multiple return types/tuples except that they’re entirely compile time structures. D tuples allow some pretty neat things, like for instance, packing function arguments and repeating arguments. Here is the code that sets up said neat thing:
tupletimesWhich allows me to do this:
argtimesWhich expands to glClearColor(0.7f, 0.7f, 0.7f, 1f). Probably not the most useful example but it’s still cool.

Today I was even more sick than I was yesterday, so I wasted most of my time blowing my nose, drinking orange juice and watching strange and interesting videos on youtube. Then it came to eight o’clock and I thought ‘Hey, I haven’t done anything productive today. To justify my existence I’m going to write a raytracer.’  And then I did.

raytracing

I didn’t do this alone of course. I’ve never touched raytracing before in my life. I HAVE however watched videos on youtube about raytracing while I was supposed to be doing other things. One video I remembered and based most of my code on. This was a video by Iñigo Quilez, a graphics wizard by any standard. Anyway, I modified yesterday’s hacked together ‘project’ to render a quad and compile a shader. For about half an hour after, I watched and copied Iñigo’s code until I got here:
raytraceThe main reason for straight up copying being that my face felt like it was going to explode and sphere intersections looked too hard at the time.

From then on I started getting the jist of things and managed to figure out reflections all on my own! Turns out raytracing makes this really easy. It’s literally just point and shoot. The hardest part was figuring out how to make a recursive function, not recursive. Once that was figured out, this was the result:
acidtraceThe colours are a bit off but the reflections are totally there. One bug I noticed was that the plane appeared noisier as the incidence angle increased, but I ignored it and messed with the colour code instead of trying to figure it out. Adding a multiplier to the colours as they accumulated for each fragment made a big difference in making it look less like an acid trip. After some fiddling I got this:

wormreflectIt’s all animated and running in realtime on the gpu so I’m pretty happy with what I’ve done given how much my face hurts.

Read More

A General Update

Pre- the last post, it’d been a while since I’d last posted. So this is pretty much a recap of what I did and what I learned over the past few months.

The most exciting/interesting thing I did was the Global Game Jam. The game I helped with was named Battle Brothers Circus and was essentially Warioware + sportsfriends but somewhat buggy and done in 48hrs. The jam was a hell of a lot of fun to be a part of. Met some new people, saw some interesting games, and reunited about a month later to show off our (slightly) updated games at a game themed bar thing. (note: the slightly updated game is sitting on the desktop of my windows partition and I’ve not gone through the effort of uploading it yet. The version on the GGJ page is the original version with bugs + no sound during the game. One day). I will definitely be doing this again.

At the end of last year, literally two days after uni finished I went to NZ to visit family and what not. While I was over there started a couple of projects (which have been a little slow since). The first one was just experimenting with hardware tessellation and controller input. It was pretty much just, there were planets (spheres) that changed their tessellation level depending on how far away they were. You could move around in a “ship” and accelerate indefinitely. It was pretty cool but I haven’t touched it in a while.

The other project was a programming language + compiler which I named Bear. I spent a little bit more time on Bear than I did on the other one. So far I have a working parser that spits out an AST. I’ve started looking at llvm which is pretty cool, but before I get that into Bear I have to get semantic analysis going – type inference/checking, scope stuff, etc. Though I’m thinking of rewriting the parser to make it easier to do later stages as at the moment, it’s mostly functional and all the AST nodes are the same struct with an enum set. I think making it more OO will simplify things down the line.

So yeah, those are things. It might be worth mentioning that I spent most of the three weeks in NZ without internet. The only reason I was able to do things was because on the rare occasion that I had access to the internet, I’d download references and save useful webpages so that I could continue when I was in the dark. I learned recursive descent from a couple of papers on parsing.

I’ve also set myself up a TO-DO list and made it public so as to force myself to try and do things on it. It’s quite lengthy, but hopefully I’ll have made some really awesome things by the end of it. The list is actually a text file on my laptop, which I’ve linked to my dropbox, so the list updates in realtime so long as I have dropbox running.

Oh yeah, and I discovered that I could do this in D:

rubyD

 

So there’s that.

Read More

Accidentally Reverse Engineering Things

I started back at uni this week. In studio (my favourite class by far) we’re starting to get into the nitty gritty engine stuff that I love so much. Stuff like graphics, and AI. In fact, in a few weeks time we will be setting our own AI bots against each others bots in a virtual ring to fight to death. Our facilitator handed out said arenas so we could write our bots and actually test them.

The Problem

There was no source; the program was a windows executable compiled in VS2103; and the bots had to come in the form of a dll. Usually my solution to this sort of dilemma would be using a cross-compiler in linux but that wouldn’t suffice in this situation for a few reasons.

  1. I needed to run the program to test my bot. Wine might have been a possibility except for the fact that –
  2. gcc based compilers (to my knowledge) can’t generate actual dlls. This also meant that mingw wasn’t a solution.

Fortunately I DO have windows dual-booted, but my strong opinions about not taking 10 minutes to start an IDE and my great fondness of c++11 features (of which, despite what I’ve heard, some are STILL missing from VS2013), meant that visual studio was never an option for me. I figured, if I couldn’t write AI in my favourite OS, the next best thing would be to write it in my current favourite language.

The Solution

That’s right. I decided to write D bindings to the arena program’s bot interface. Not exactly the most straightforward and easy approach, but it’s the one I’d be happiest using. The reason I chose to use D, other than the fact that it’s a breeze to use, is because of it’s limited capacity to interface directly with C++, which includes being able to pass some objects between them and call their methods as if they were native classes. This is important because the arena requires that the bots are classes. For the time being I’m using an intermediate C++ binding layer because some of the parameter types and structures aren’t directly supported in D. For example, I have to map std::vectors to dynamic arrays and get std::strings internals. I’m compiling the intermediate binding layer with the Digital Mars C++ compiler, which is NOT very good. It’s missing a lot of c++11 features and stalls whenever I use nullptr. But it came in a zip, ready to go so it’s staying for now.

The Discoveries

Now this is the first time I’ve ever written or compiled a DLL, or had to interface with anything that I didn’t write. One of the first problems to arise was the fact that the arena program passed data about the world to the bots through either references or const references to structs (which for the most part were easy to duplicate in D). D doesn’t support references. So the solution was taking addresses in the intermediate stage and passing them to D instead. Next, some structs had std::vectors and std::strings. For these, I created new structs in which the (important) std::vectors where transformed into pointer-length pairs, and the (important) std::strings where just transformed into const char*s. Then I hit my first problem.

Every time I tried printing strings, either in the intermediate layer or in the D part, everything would crash. So I stopped trying to print them for the time being and tried doing other things. Then I discovered that everything that followed a string was corrupted. I spent a good few hours trying different things but nothing seemed to work. But then I had an epiphany and after a quick google search discovered the cause of these two strange problems. See, the arena program was passing structs that contained std::strings, std::vectors, floats, some other structs, pointers, so on and so forth, to my intermediate C++ layer. For the most part that was okay. What wasn’t okay was the fact that those structs were using the Visual Studio 2013 standard template library, and I was using some other STL provided by Digital Mars. My epiphany was that the only part of the c++ STL that’s standardized is the interface. It’s completely up to vendors (to my knowledge) how they implement the classes therein. My google search came up with this, which confirmed all my suspicions. What was happening was the difference between what the arena provided and what my binding layer expected in terms of data layout was large enough that I was getting bits of string in my floats (literally).

My initial solution to this was to replace all std::strings in the struct definitions in my binding layer with 24 byte arrays. And this fixed it. My floats were floats again. Then I started experimenting a bit and found that treating the byte array as a character array and printing it actually worked. I found this puzzling as this meant that the string was being stored in the struct itself, where I assumed that it would have been stored behind a pointer. To confirm this, I dumped the byte array to console to figure out what was going on. The string was indeed being flattened. Note: I also discovered how good printf is for making tables of things.

std::string dumpThere in the first 5 bytes was the name of the map that the bot was in. Then at the 16th byte offset, the length of the string. Then at the 20th byte offset, what seems to be the capacity of the buffer excluding the null terminator. To see what would happen with a larger string, I renamed the map. And this was the result.

std::string dump 2

Unlike the last dump, the string didn’t appear to be flattened. This time, the first four bytes seemed to contain a pointer instead. The 16th byte was the same as the size of the string so that was definitely where the length is stored. And the 20th byte has doubled (if you account for the zero terminator), which is behaviour that one might expect from an STL container. This was the behaviour that I expected initially. I did one more test because why not.

std::string size 15 dumpI made the string one byte shorter and there it was again. The string was flattened. So since I’d figured out the behaviour of std::string (enough to not make it crash when I use it), I thought I’d put that knowledge into practice and wrote a struct that matched the layout of std::string and wrote a function that returned a D string constructed from the write data.

Not exactly the most complicated code in the world but I’m pretty happy that I managed to figure out why my float was wrong. Maybe I’ll try vector next. Or maybe I’ll try writing a bot. Who knows.

 

 

 

Read More