Advanced Ambient occlusion, terrain & Blender 3D

Baking Ambient Occlusion maps

In the first part of the tutorial the basic procedure for baking a terrains ambient occlusion map was discussed, how the mesh should be prepared and materials applied. Whilst this is not that different to baking ambient occlusion for other types of objects, there are a number of important factors that should be considered or kept in mind because they directly influence the resulting maps.

The following "Advanced Ambient Occlusion for Terrain" is broken into two general topics of discussion; "Materials" and "UVW's", and how they can be utilised to gain better baking results on meshes that are, more-often-than-not, extremely large.

Modelling the Terrain Mesh ^

The physical size of a terrain mesh has a direct affect on the outcome of texture baking Ambient Occlusion maps, it's important to understand this critical concern as most 'errors' or 'bad' bakes can be attributed to this single fact. Because Blender expects content to be made relative to a scenes default scale and objects contained there-in (the 'camera', 'lamp' and 'cube' items), any significant deviation from this tends to cause problems that cannot, generally, be fixed by changing Blenders 'Unit' properties; which do not have a direct influence on the texture baking sub-system. In other words, if objects are overly large they fall outside the capabilities of the renderer and its ability to read mesh data as that relates to the structure of a mesh, the locations of individual elements (faces and objects) and the lighting system used specifically for Ambient Occlusion bakes. The 'fix' for this problem tends to require that terrain meshes be reduced down to a more manageable size, relative to Blenders default scale so the baking system can produce better results; this should be done before continuing on with any texture baking.

Design note: much of what follows is best done to a 'duplicate/copy' of the original terrain being baked, which can remain intact - when scaling meshes use "Transform" properties, available in "View Properties" ("N"), this helps with regards to keeping track of Objects size and scale. This can also be aided holding "Ctrl" when scaling to snap to the grid.
Original size of terrain mesh relative to NPC helper

Original size of the terrain mesh to be baked relative to an NPC helper (the relatively small red entity marker highlighted). Extremely large mesh objects tend to cause proportionally poor texture bakes - the larger the mesh size, the greater the potential for problems to occur

Rescaling a mesh to produce better texture bakes

Terrain mesh rescaled to a much smaller size relative to an NPC helper - used as a fixed reference point, the NPC helper is a game related indicator of game character sizes, ideally these should be used as part of the design process when making terrain

Ambient Occlusion from a relatively small scaled mesh

Re-scaling the terrain mesh to a smaller size relative to Blenders scene default, baking ambient occlusion will produce 'maps' with a more natural and evenly distributed spread between 'lights' and 'darks' based on the objects physical characteristics

Terrain mesh starts to 'bleech' due to it's size

The same object scaled up 100x starts to exhibit problems. Relative to the scenes lighting, because the renderer can't determine shapes and distance properly at this scale - it is literally seeing distance as being either too far, or too close with nothing in between - the results of baking tend to 'flatten' out...

Extremely large mesh causes problems for ambient occlusion

... until making the mesh 1000x larger results in only the very obvious of features being baked to ambient occlusion because the render engine is only seeing the mesh in terms of everything being either too far or too close. Currently there isn't a workable way around this problem except to temporarily rescale the mesh (a duplicate) too a smaller size before baking

Terrain & Texture Density ^

Aside from the scale of a terrain mesh effecting how well shading is represented as ambient occlusion, generally speaking the primary concern with baking such an object is its relative "Texture Density". Put simply this relates to too small a texture being mapped to too large an area (or object), resulting in said image pixellating when assigned. There isn't really a way around this when using standard, 'static' meshes for terrain; the inherent nature of baking textures requires the use of 'unique' assets generally mapped on a 1:1 basis (the entire object is mapped within the bounds of available "Texture Space"). Invariably then, this means there will be pixelation to some degree when using this technique on terrain (or any type of model for that matter).

In the images below for example, if the terrain mesh is UVW mapped and textured on a 1:1 scale (assuming for the sake of argument that 1 "pixel" equates to 1 "unit" of measurement), it will result is pixelation because a 1024x1024 image is being used to cover an object that's perhaps four or five time larger than the image itself - a 1:4+ ratio difference between 'terrain', "1:", and 'texture', ":4" (1024:4096+). Conversely, mapping an image to too small a ratio, 4+:1 perhaps, means that whilst a much better texture density can be had, the mesh can no longer be baked; because the image repeats (tiles) over the object it ceases to be unique, and for all intents and purposes, is thus unbakeable.

This presents something of a problem that can be expressed by a simple question; "how can a terrain be baked so as to use unique assets, without risking the decrease in clarity due to low pixel density?". Whilst the answer is always one of compromise, and based firmly in the capabilities of a given technology (game or interactive engine), the general solution is to either split the UVW map into sections, use more textures/materials, and/or do a combination of the two.

Texture mapped with'low' density leading to pixelation

The initial and basic UVW map has the texture applied to the terrain mesh on a 1:1 ratio - UVW's are mapped within the 'bounds' of available "Texture Space" - making it appear relatively low resolution

Texture 'tiled' over mesh with 'high' density

On the other hand, increasing the apparent resolution of the UVW map/texture relationship now means that, whilst the image has much better 'density', it cannot be baked because the texture mapping is no longer 'unique'

Terrain, Splitting UVW & Single Textures ^

Splitting a UVW map into sections is determined by how many, and how large a given texture asset is. However, it's not as simple-a-matter as mapping the UVW's to whatever assets are available and 'job's done', to use an colloquialism; terrains are typically very awkward shapes, which means it's often difficult to capitalise on their respective layout such that UVW seam placement is unobtrusive and logically advantageous at the same time. As there isn't a shortcut to this process a certain degree of trial-and-error will be necessitated with respect to marking edges for the best layout of the map relative to the object and texture assigned to it.

Being mindful of the two main points raised above then - 1) hiding seams and 2) taking advantage of any natural shapes for their placement - use "Mark Seam" to split the layout and then "Unwrap" everything again. This time Blender will do its best to redistribute the sections more efficiently relative to the texture and the size of each UVW segment. Some minor editing may be necessary but the end result should be a map with better clarity (to varying degrees) than previous attempts, when the UVW was a single 'lump'.

Design note: when using this approach to mapping large terrains it's important that, had the mesh been physically split into separate objects as an aid to the process, any loose or separated parts be reconnected (select each object and press "J" to "Join") and "Remove Doubles" used, else texture bake will interpret splits as 'hard' edges (effectively 'mesh smoothing'), baking them as such (hard lines) into the resulting ambient occlusion texture.
Using "Mark Seam" to split UVW map

Using "Mark Seam", split the terrain UVW into a series of smaller isolated 'islands' as a means to 'optimise the overall layout and better the texture density and distribution of UVW's over large object

Wire view of the terrain with 'Seams' marked

"Wire" view of the same mesh showing the layout of 'seams' marked around the mesh, each one will form an separate area within the overall UVW map

Re-mapping the UVW's after 'seam' splitting

Re-mapping the terrains UVW's after using seams to split the map. Note that Blender will try a 'best fit' approach to distributing the newly created islands into available texture space whilst keeping UVW's relative to each other (ensures texture density is approximately equal for each segment)

Re-baked ambient occlusion based on split UVW maps

Depending on the distribution of the UVW's after being split, it may not be immediately apparent any improvements have been made, as in the example above, when compared to the original, single UVW map baked below

Original AO bake based on Blender default UVW unwrap

The original default UVW unwrap

Terrain, Split UVW's & Multiple Textures ^

Although splitting a terrain UVWs and mapping them to a single texture is relatively straightforward, for very large or complex objects that particular approach generally doesn't provide the kind of quality typically needed for terrain. In these situations the alternative is to map the overall UVW to a number of separate textures, the advantage being that the general distribution ratio of UVW texture space to texture density can be better controlled and equalised to ensure each segment of the terrain uses proportionately the same or similar density as other sections. To make using this approach easier to manage it's best to use a number of separate Materials, each assigned to a specific section of the mesh - depending on texture availability this may also means each having its own texture image assignment.

Design note: breaking the UVW into sections and using separate textures also means being able to control density better from the point of view of what the player can see, increasing it where they directly interact with their surroundings, i.e. the ground underfoot, versus less detail for distant elements, i.e. hills along the horizon.

In Edit mode create a number of additional Materials and assign each to a specific part of the mesh. Then using the "UV/Image Editor", 'map' the UVW associated with each section to their respective texture image. Repeat the entire process per separate material/image/mesh section. The end result should be a terrain with several Materials and UVW's mapped to separate textures.

Design note: before creating and assigning any additional materials, select the entire mesh and click the "Assign" button under the Material List, this ensures the active entry is set as 'default', else any new material upon assignment may inadvertantly over-ride what's already present on the mesh. To create a new material entry click the "+" button to the right of the Material List and then the "+ New" button below once the new data becomes available. Select a group of faces around the mesh, make sure to select the new Material from the Material List and then click the "Assign" button to make the material/mesh association. For clarity change the "Diffuse" property where necessary to distinguish the additional Material entries from each other. Finish up by adding a new set of "Texture" and "Image" properties per each subsequent Material, then finally add either an available bitmap image or a generated texture.
Using multiple textures on a terrain

Using additional materials the terrain mesh is broken down into a number of sub-sections, each remappedto its own texture - note that re-mapping the UVW's and using the default "Unwrap" produces disproportionate mapping; the pixel density differs between each as Blender attempts to use as much texture space as possible per UVW

Default UVW mapping assigned to Material '1'

Ground plain section separated into it's own UVW/texture assignment

Default UVW mapping assigned to Material '2'

Hillocks separated and mapped

Default UVW mapping assigned to Material '3'

Surrounding cliff section separated and UVW mapped

Default mapping assigned to Material '4'

Plateaux separated and UVW mapped

Breaking Large UVW's into Smaller Sections ^

In splitting the UVW's up as discussed above, a problem occurs which relates to the way Blenders mapping defaults to a 'best-fit' assignment which causes texture density to differ across the objects as a group. Usually the fix for this is to simply rescale the UVW's. However, this can further result in smaller areas having a disproportionately lower texture density (assuming all textures are 1024x1024, the larger UVW segments are being mapped to the same sized image as smaller ones). To address this problem the large UVW maps are broken down further still into smaller units that can then be better re-mapped and redistributed within the allotted texture space (whilst still mapped to the same image).

Shown below for example, the default mapping for the 'ground' plain means it's disproportionate to other areas of the terrain, leading to uneven texture density issues across the set of elements (or objects if each mesh area were detached). To get around this problem the area is split into a number of smaller sections that, when remapped, will have a better distribution which should increasing texture density.

Single large UVW has low texture density

The default mapping of the ground plain (shown with other sections) results in different texture densities for each part of the mesh due to Blenders 'best fit' scaling of UVWs within the bounds of a texture

Segmenting the UVW map of a section further for better texture density

Larger UVW map sections will need to be further broken down into smaller parts for better unwrapping - splitting the UVW like this usually allows each section to be repositioned using a larger area of texture than otherwise would be had if still attached to the larger mapping

Do this to all the sections with the aim of evenly distributing the UVW's and texture density so each is approximate to the other, or in other words making sure to produce an overall mapping that's more uniform across each component of the terrain as that relates to the group of materials/sections. Once the UVW mapping has been corrected, ambient occlusion can then be baked as normal, this time producing a series of images each of which belongs to one section of the mesh. These will then need to be saved.

Design note: when using this approach it's important the mesh remain intact and whole else the bake process will interpret splits and division as 'hard' edges, baking them to any subsequent maps. In addition, attention needs to be paid, where-ever possible, the the 'joints' between UVW maps; it's more-or-less inevitable that due to the way UVW mapping works, there will be a slight differences in texture density between sections - because of UVW distortions - which will manifest in bakes as lines or differences in pixelation.
Terrain UVW map after being properly prepared

The end result of using several textures each with their own re-mapped area of the terrain is a much more evenly distributed density and/or one that can be better optimised by changing density depending on how or where the player will be or what they can see

Final rebaked ambient occlusion after adjustments

An ambient occlusion map baked from this type of set-up is usually of much better quality because of the increased texture density attributed to each segment of the overall terrain UVW map

Conclusion ^

Terrain is generally quite tricky to work with because its often composed of awkward shapes and forms. The challenge here is to unwrap these shapes whilst at the same time as keeping the pixelation associated with such large models down to a minimum. However, other tricks and techniques can be used to obscure this fact; breaking terrains into blocks (1024x1024 'unit' models for example); adding a 'detail' stage to an in-game material or shader to mask relatively low resolution images; using additional UVW layers which specifically hold light or ambient occlusion maps, freeing up the main UVW layer for tiled texturing, and so on. Each approach, having it's own 'Pros and Cons', needs to be considered alongside any limitations or criteria set in place by a given game engine or real-time technology. With a bit of forethought and/or pre-planning, baking ambient occlusion shouldn't present too many problems, the key being to accept that there will be some level of "compromise" necessary; it's never going to be possible to rely solely on texture baking of any description to hide the fact that a terrain is an extremely large modelled object.

In the final following page, the process of baking a Terrain mesh Ambient Occlusion is discussed in Video form.

^