A while ago, I wrote a post about package sorting in order to minimize diffs when patching on Steam.
I ended up running a straightforward test where we would change the game minimally and build consecutive patches. The patches turned out to be quite large, but upon investigation, it was clear there usually wasn't a good reason for this. Our compiled shaders would contain small differences, as would many meshes in the game.
These looked like uninitialized memory errors and, upon further investigation, that's indeed what they were. They are fixed now, so patches are a lot smaller.
However, patches may never be as small as we would like, and I find there is a pretty deep philosophical issue where.
Right now we have 17MB of files that change in response to minimal changes in the game (say, moving a few entities in the world by small distances). These are mostly cube maps for Light Probe entities, which are used to do lighting on nearby objects. Here's what a Light Probe looks like in the game:
(Well, you don't see that sphere the game, only in the editor!) To generate one of these cube maps, we render the scene in all directions, from the point at the center of that sphere, then apply a blur to the result.
Imagine that we move a couple of entities in the world, even slightly. All Light Probes that can see those entities are going to change, by at least a pixel or two. Because we are applying a big fat blur kernel, a 1-2 pixel change becomes small changes spread across the entire image.
We have an automated script that preps the game for release, clearing out data that we only use for development, and ensuring that automatically-computed things get recomputed (like LODs for faraway objects). Recently we added a step to this process that re-generates all the Light Probes, because it just makes sense to update those automatically. If we have to keep track of them manually somehow we are just going to make mistakes.
What this means is that you can pretty much guarantee that a lot of light probes are going to change if you move some entities and patch the game. If we change anything basic about any of the shaders, which seems very likely, then all of the light probes are going to change (because the output of the shaders feeds back into the light probes).
That said, maybe it's the case now that more light probes are changing than I would expect. I am wondering if other things in the game are causing them to change. For example, when you are playing the game, foliage waves in the wind. I wonder if the animation timer for plants is ticking during the process when we generate light probes, causing them to move over time? This would cause most light probes to be different every time we bake them, regardless of other factors.
Now, there's a bigger issue. In addition to light probes, we have pre-baked lightmaps all over all geometry in the game:
In a way that is reminiscent of the light probes (but a bit different), we generate these by placing a camera at many points along each surface, rendering the scene from each camera, and interpolating those into a final result.
So, just like light probes change when entities move slightly, lightmaps are going to change also. Lightmaps will also change if shaders change, but furthermore, lightmaps will change if light probes change (because light probes determine the look of entities that shine on the surfaces we are lightmapping), and light probes will change if lightmaps change (because the lightmaps influence the objects rendered in the light probes). So we have a circular dependency, which is not so bad in practice, except it means that if you change one entity in the game, then repeat this process iteratively, you may well end up changing most lightmaps and light probes in the game.
The light probes are not so big, but last I checked there are between 500MB and 700MB of lightmaps. However, re-baking the lightmaps is not part of the automated build process; we have to trigger a bake manually. But it seems like good policy to do a full rebake any time we change the game significantly. So, we are probably looking at a minimum patch size in the 500MB range.
I guess that is not such a big deal, since we are not an online game that would be patching all the time. Hopefully once we release the initial version of the game, the "content" won't change very much (sorry, but the word "content" as used in video games is kind of absurd, so it's hard for me to use it without sarcasm quotes; it's almost as bad as "title", which is a word I refuse to use, except to make fun of its use)... the "content" won't change very much, so patches would be rare.
We'll see. We have a while to figure these things out.
Interesting!
On an unrelated note, I suspect that you will like this game, Jonathan:
http://www.kongregate.com/games/wanderlands/impasse
Wow, that first interior looks noice!
Thinking out loud here but with respect to patching the light maps, one approach might be to bundle the lightmap diffs in patches, reconstructing the new lightmaps on users’ machines. If the changes to the lightmaps are moderate, the diff should yield higher compression ratios then the lightmaps themselves (via something like runlength encoding or a dictionary based compression algorithm like http://en.wikipedia.org/wiki/LZW).
Ehh, no, this is just a level of complexity that we would be unwilling to engage in. Complexity management is how you ship quality games in a reasonable period of time.
If it comes down to doing something like this, or having people download 500MB patches, then it is going to be the latter.
(The situation would be different if it were an online game that patches all the time; in that case such a technology investment would be focused on the typical user experience. But that is not the case here.)
Fair enough. I also value simplicity, not only from standpoint of implementing features but even moreso for the programmers who will have to maintain those features in the future. As someone working on an online game, issues like this are of higher importance and thus earn a larger portion of the time/complexity/person power budget.
How about just generating the lightmaps and/or probes on the users machine when a new patch is downloaded? Shouldn’t add too much complexity since you would be using the same tools, just on the users computer instead.
Or is the processing time too long for this to be viable?
Best regards
Even if the processing time were low enough, you would essentially be turning an in-house tool into a shipping product. Thus, the test burden goes up.
Are the light probes significantly different than just being what you’d get by rendering the live scene into an FBO and then blurring that into another FBO via Gaussian blur shader? If not, any reason you can’t just generate the light probes on-the-fly when their region is about to come into view?
In principle there is nothing preventing this, but at present generating light probes is a little bit slow. I think it mostly has to do with the blur step, which I would guess is done on the CPU (if Ignacio is reading these comments then maybe he can say something about it).
My understanding is, if you want to do a perfect blur across a cube map, it is kind of a pain in the butt to do that in hardware. (If anyone out there knows how to do it very well, let me know!) Right now our shadow-casting point lights are limited to one hemisphere for a similar reason (that way we can project the shadow onto a plane, rather than 5-6 cube map faces).
Although now that I think about it a little more, it doesn’t seem hard at all to do a good blur on a cube map. Maybe the problem is something else. I didn’t program it so I am not sure!
Are you rendering the light probes and light maps on the GPU? In my experience GPUs don’t always produce deterministic output.
We have a similar problem in Path of Exile. We render sheets of tiles in an isometric perspective that are used at run time for our mini-map. This is done during our build process automatically. Unfortunately we have found that there are very slight differences in the output so when doing patches we would get an extra 50-80 or so meg of data.
Our solution so far has just been to run a script that strips out minimap tile changes in patches that no real changes have occurred in. It’s not a great solution, but it’s working out okay for us so far.
Interesting post. 500MB patches seem a bit hefty so I hope the biggest problem can be solved, but as you said there probably won’t need to be patched too often.
By the way, I can sort of understand why you wouldn’t like using the word “content” for games, but what’s wrong with “title”?
Pars pro toto. The Witness is the title of the game, but The Witness is not a title! Unless you are publisher and the game is just a row in a spreadsheet. It’s a sad evolution of the language…
How long does it take to regenerate the light probes and lightmaps? If it is smaller than the time it will take your average end user to download nearly a gigabyte of data, maybe you could run the generation scripts client-side. Either at some kind of Steam triggered “post-update” hook-in, or if that’s not possible, just on first run after an update.
I apologise for adding my voice to all these suggestions – you have likely thought of all this already.
Lightmaps take a long time (more than 24 hours right now). Light probes are much faster, just because there are many fewer sample points.
Users SHOULD have to generate their own lightmaps. If they can’t wait 24 hours more to play the game. They don’t deserve it. Think of it as the first puzzle in the game.
Sorry for being random and dumb. I just felt it really strongly.
In a humorous way. Of course. :P
Would you mind going into a little more detail on your view of the words “title” and “content” in the context of video games? “Content” seems to be a fair enough term to vaguely describe the substance of the game as the player encounters it, whether that be the length of the game, its size on a hard drive, or some other measurement of how much “game” there is. It’s understandable that quantifying a game often defeats its purpose (e.g. Dear Esther), but so long as the term is used only vaguely, it should be reasonable enough when coupled with other descriptors.
This of course raises the question: “What is your stance on DLC?” but I understand that’s a difficult subject to detail in a comment.
Your development principles are fascinating!
“Content” means “something goes here but it doesn’t really matter what”. It’s something that’s created with the goal of filling space. As for why it’s a irritating term, people who create things for their own sake aren’t in the same line of work as people who create things to fill space, and they don’t like being grouped in with the latter.
The light probe filtering is not a simple blur. We convolve the environment maps with a cosine weighted radial filter raised to different powers in each of the mipmaps. It’s possible to parallelize it and run it on the GPU (we currently compute it in multiple CPU threads), but even with a very fast implementation I think it would take too much time.
Right now we only have 72 light probes in the world and we process them in 5 minutes. However, I expect to have around 300 by the time we ship the game. Even with GPU optimizations, I think it will be hard to process them in less than that.
Could you elaborate on what you mean by “content” and “title”? I honestly don’t know what your getting at.
Yeah, “title” and “content” strike me as generic jargon meant to make things seem other than they are, or to cover up the fact that the speaker isn’t saying anything substantial, or (in the case of “DLC”) to shift the conversation from something that serves the user into something that serves the seller.
“Title” was just put in place by suits who didn’t want to go around saying “game” all the time because they wanted to be taken more seriously, and it is hard to put up a serious image when you are saying “game” all the time. (“Title” does have some technical usage in the retail game world, where “title” refers to the game in all its incarnations and “SKU” refers to one specific over-the-counter release e.g. for one platform, but this is not how “title” is generally used these days. It is generally used to mean “game”.)
“Content” is just a generic word that lets people say nothings about nothing. It comes from the software application world where you had “code” that makes things happen and “content” that the code moves around without knowing what it is. I have always taken it as a bad sign when people use the word “content” very much because it means they aren’t thinking about the actual things being operated on by the code, which probably means the code is not doing a good job.
(It then took a more sinister meaning in the web world, where “content” is what draws users to a web site and “ads” are the real purpose of the site.)
In general, calling things “content” is ugly. Go watch the movie Jiro Dreams of Sushi. Do you think that Jiro refers to the food he serves as “content”? If not, then why is it better for us to do so?
In the case of “DLC”, what does “DLC” mean? It just means “something generic you can get for yoru game”. As the player, don’t you care what it is — whether it is an expansion pack, some new character skins, or horse armor, or whatever? More informative terms would help, there. But in reality the term DLC was made up by people who sell software, and its meaning is: “Things we can sell to players for money after the original purchase, or else give away for free in order to maintain engagement with our game / get a news boost / etc”. This is what I mean about shifting the conversation. If the point were really to give players something valuable, then it would not be called DLC, it would be called something that tells you what you are actually getting.
Content is the same thing as Lorem Ipsum. If you’re looking for a generic term for things that have a reason for being, try “works”.
I think it’s interesting that often developers who are creating substantial and valuable changes to their games tend to refer to the DLC as “Expansions” just to avoid the negative connotation of frivolity that the word has come to mean. A good example being Bethesda with the Skyrim expansion, even though they were the original proponents of frivolous DLC. (The horse armor)
I’m curious if you think the use of these words harms game development in a way that wouldn’t happen anyway, or if it’s just something that points back to a bad way of thinking. I just ask because it seems like arguments about words on there own are kind of weird. I guess Jiro could refer to his Sushi as content if to him, content means something positive (yeah it would be a weird word to use for food). I’m really curious what the practical purpose of pointing this out is. I can’t word that in a non rude sounding way, sorry.
I do understand what you’re trying to say – but I believe there’s some semantic philosophical issues with your definitions and their use. Surely there’s a point where we move beyond the connotational limits of ‘games’, but cannot broadly define every such work as art – ‘title’ and ‘content’ are less jargon and more adoptions for concepts that people are having a hard time interpreting – Marshall McLuhan was concerned that “we cannot free ourselves of the delusion that it is how a medium is used that counts, rather than what it does to us and with us” and such terminology is surely an attempt to wean perceptions off of the later attitude toward a more progressive one. Perhaps I have misunderstood your usage though.
Thank you very much for clarifying (and also for the great movie recommendation)!
The 500 MB for light maps seems really high, is that for the entire game? It feels like the world would have to significally change for so much of the light maps to be affected.
I suggest re-reading the posting; I say exactly why.
Looks great!
This discussion brings up a running concern with game reviews, that they often judge a game by how much stuff is in it or by how much of a ‘full package’ it is. While there may be some positive intention to this choice of words (for a game to explore it’s full potential) the consequence is that the reviews treat games as consumer goods.
Not all sites are guilty of this, but it gives me a shiver whenever I hear it.
I think I understand what you’re saying, but perhaps the idea should be to simply be careful about the use of the word content – and be aware that it can be/and is often used as sleazy marketing speak – as opposed to forever marking it as a signal that someone is devaluing their own work. Ignoring the marketing side altogether, developers have definitely come to use the word as a pretty legitimate shorthand. You used the word yourself in your post (disdain-filled quotation marks or not), because I assume that you figured most people who read this blog would be able to successfully fill in the blanks. Really, the best solution is for everyone to do their best to say specifically what they mean (art, encounters, level layouts) and not use a catch-all word like “content” – but sometimes it’s just faster when your audience is mostly on the same page.
Jiro struck me as pretty practical – so I don’t know if he’d really care very much about what he specifically called his sushi, as long as he was able to clearly communicate with the chefs that work under him. ;)
When these light probes and lightmaps change due to miniscule changes in the world, such as leaves moving in the wind, couldn’t that be handled by having the build process compare the old and new probe/map and substitute the old version if it was similar enough according to some simple metric?
Not so simply, because you could have two lightmaps on neighboring objects, then rebake, and decide one is “similar enough” but the other isn’t, and you then start getting discontinuities between objects. Unless your criterion for “similar enough” is super-small. I dunno.
I thought super-small differences might indeed occur, especially in cases where you really didn’t expect any difference at all, but I guess it depends on the cause of the difference.
My first foray into computer art and more importantly, computer programming, was working with the Source engine, and I can’t help but notice the similarities in the engine design and programming. I was wondering, are you using the Source engine as a model for this engine? Also, why didn’t you guys use the Source Engine? It’s flexibility always amazed me, and the things a performance beast.
I don’t have any idea how the Source engine works. We are just making decisions that we think are reasonable.
… and the Source engine is pretty old these days. If we licensed it we’d have to rewrite a lot of things, it’d be hard to get it onto iOS, etc. Also, we just don’t want to license an engine.
Ahh I was just curious haha. Can’t wait to see the final game! I loved Braid and I just saw Indie Game and was browsing around looking for info on your new game. I ‘got’ Braid. There’s a lot of us out there that got it. We just don’t have blogs because we don’t think anyone gives a shit about how much a video game affected us.
Do really lightmaps need light probes for being calculated? I am studying the topic and i’d like to understand :)
The cubemaps of light probes are calculated from the rendering of the scene at that point. Every object is shaded with its own material + precalculated lightmaps (if any, maybe some objects have no lightmaps?). Lightmaps are calculated from radiosity using diffuse & emit properties from the objects materials.
For example i’ve a white room lit by an emitting square (like the Cornell box). Also there’s a red sphere inside the room. I’d lightmap every surface of the room, also the sphere, so the lighmaps have that red shade over the walls. Then i calculate a probe inside the room rendering the scene with their materials + lightmaps. When i’d place, in realtime, something inside the room i’ll shade this something using the probe. The object would not have a lightmap because it’s a dynamic object in the scene. Is this right?
Is it possible with light probes to capture shadows inside the volume of the scene? For example i’ve a room with sun light from a window. I calculate the shadows inside the room with global illumination or other methods, then in some way i store that in the XYZ position inside the room is shadowed, and in this other point is under the sun and so on. When i’ve a character inside the room between the shadow and the light, for example, his back is half shaded in shadow and half in light. This done using the grid of light probes calculated inside the enviroment. Is this possible or these kind of things can only be done using realtime shadows?
About “content” sadly also inside other kinds of software development there are companies who aren’t really interested in their products but just need to have something to sell. I’m working on a software for about 3 years and in this ammout of time people just don’t want to help me when i ask, nor they want to know how much is difficult to create a CAD-like 3D enviroment from scratch. They pressure me just to finish, paying very little (the same as working on not graphics software). Then, sometimes, they say they want to implement “this” or “that” because chatting with someone they heard about “this’ and “that” that bigger companies do.
It’s not surprising i’m rallying to finish a minimum ammount of things they need and quit to start doing better things myself ;)