Baking ambient occlusion for terrain, Advanced Options & information
Ambient Occlusion maps - or AO bakes - as they more commonly known, are a useful way to generate 'shading' (they differ from 'shadows') for an object based on 'ambient' light; this is the background and indirect lighting that illuminates objects in the world around us. For game and 3D media content, using ambient occlusion adds 'depth' and 'substance' to an object that may not otherwise be possible using other methods of in game lighting.
In 3D this can be represented by rendering scene based 'ambient occlusion' and 'baking' that to an image; the process tries to approximate the same non-directional lighting on objects in 3D space that happens in the real world by 'writing' (baking) a gray scale image to memory (file) based on the physical characteristics of the model being processed.
What are ambient occlusion maps? ^
There two things that need to be kept in mind when baking AO maps for terrain meshes (or any extremely large mesh object);
Although both of these effect the results of an AO bake, they do it in slightly different ways; image size has more of an effect on the quality of the renders themselves ('pixelation'), whereas the size of the mesh tends to effect the fidelity and accuracy of the results.
The physical size of the image ^
When rendering an ambient occlusion map, it's being done relative to the amount of pixels space available on the UVW mapped image applied to the mesh, so, the more pixel space available the better the results - a ray traced over 10 pixels will have greater clarity than one traced over 1 pixel
So with regards to terrain objects, it's always best to use at least a 1024x1024 (uncompressed) image as the basis over which the AO bake occurs as this ensures there is a reasonable amount of texture space available to be used for rendering - smaller sized images lead to pixelation and 'stepped' shadows - subject to the density of the UVW map after unwrapping the polygons of the mesh.
Which leads on to the following.
As much as possible, make sure the outer dimensions of the mesh are square. In other words looking down on a mesh reveals a square area in to which the terrain has been carved. Whilst not absolutely necessary is does ensure a 'best fit' scenario for the UVW map over the texture (which should also be square to follow the "power of 2" rule as much as possible) which should sit completely within the bounds of the texture; making the mesh square means that on a square texture, the UVW map will also be square and using as much of the available space.
Design note: none square meshes mean none square UVW maps, which, although not a problem in of themselves, become difficult to optimise the UVW map space for; trying to stretch the UVW map to use more texture space results in severe distortions of the image applied over the mesh which should be avoided as much as possible. Pinning the outer vertexes of the UVW map to the outer edge of the texture map as it appears in the UV/Image Edit window will force the the "Unwrap" calculations to use every bit of available space; making the mesh, texture and UVW map square (the same dimensions) means as little distortion as possible.
Physical size of the mesh and it's effects on AO bakes ^
As mentioned above, the physical size of the terrain mesh has an effect on the end results of the AO render bake. It doesn't effect the image quality so much as it's accuracy and fidelity in relation to the terrain itself. What does this mean?
The larger the physical mesh size, the poorer the accuracy of the results from the AO bake. Because rendering an ambient occlusion map relies on the physical distances used by ray tracing, the larger the mesh the less accurate the results and the greater the compensation needed to achieve the desired results (based on using the default settings).
There's only one parameter that matter in the case of large mesh objects and that's the "Distance" value ("0" - "5000.00"); the larger the mesh, the higher this value has to be - very large meshes (above and beyond 25,000 units in size) will more often-than-not require the Distance value be maxed out. To illustrate this issue, see below;
The results of baking an AO map from a mesh that approx. 2.0 x 2.0 x 1.5 units
In the image above, the AO bake was rendered to a 1024x1024 pixel image applied to a mesh that was 2.0 x 2.0 x 1.5 Blender units in size - that's approximately the same size of the default cube that's in a scene when Blender is opened. The settings used to render out this AO bake are the defaults used above in the tutorial.
The same mesh and AO settings now re-sized to 60,000 x 60,000 x 40,000
Using the same AO bake settings as above but this time applied to a mesh that has now been massively re-sized to a 'game ready' size of 60,000 x 60,000 x 40,0000 units in size for the one section (yes, you can create truly massive terrain meshes in Blender, Ed.), the image above shows the results of a bake using those initial settings.
Compensating for the extreme size by increasing the "Distance" used for ray trace
To compensate for this problem the "Distance" needs to be increased; setting it to "500" results in the image above.
Maxing out the "Distance" settings but still using the AO bake render settings
But because this mesh is excessively large (relative to the original size) the "Distance" settings need to be maxed out to its highest value of 5000.00. Even doing this doesn't result in the same fidelity or accuracy as the original render, so that leaves two options;
Use the results 'as is' relative to the very large mesh.
Re-size the mesh smaller and then bake or work on the mesh at a more reasonable size to get the AO bakes done and then re-size to actual size afterwards.
Ambient Occlusion in relation to real time game environments:
Ambient occlusion rendering can be changed to suit requirements in many ways and as each iteration of Blender comes out there will no doubt be changes and improvements to AO baking overall that will have a direct effect on the quality and accuracy of the results. Having said that, keep in mind that AO baking in this manner is relative to the results being used in real time 3D environments where the golden rule applies;
Design note: "Just because you can, does not mean you should".
So, spending an hour rendering out a AO bake to a 1024x1024 image using 32 samples won't necessarily 'gain' anything to what's being done when a render at 16 samples is more than enough - as texture size effects quality of image results it's questionable whether the difference between the two settings is noticeable and the extra time spend rendering out 32 samples worth it, especially when AO bakes are used to 'enhance' art work rather than being used on their own.
Having said that, using the largest image size possible is a good idea, mainly to avoid the pixelation issues associated with smaller images - making use of anti-aliasing when re-sizing images in photo editing software is advantageous in this instance - render huge and re-size to actual needed size.
UVW map problems:
Because of the way AO baking works, it does rely on a good quality and clear UVW mapping, as anyone that's tried to unwrap a terrain will tell you, the awkward shapes present in one often mean that the UVW map has either or both uneven UVW map polygon distribution - leading to texture stretching, or, uneven UVW triangle sizing - also leading to texture stretching.
This means that a good deal of time may need to be spent 'fixing' those distortions as they appear in the mesh, the accuracy of doing that is dependant, again, on the texture size of the UVW mapped image (snapping to pixels is advised to ensure that there's no overlapping the edges of the texture.
When using a terrain mesh in this way it's always a good idea to 'pin' the outer edges to the outside edge of the image in the UV/Image Edit window; there are a couple of reasons for this
but mainly it ensures that all the UVW triangles stay within the bounds of the texture space as well as forcing a more even distribution of the UVW map itself - which is why it's often a 'best case scenario' to make the outer edges of the mesh itself square because there is then a direct 1:1:1 ratio between mesh:texture:UVW map.
Ambient Occlusion renders black:
If you try to bake a ambient occlusion map for an object or set of objects and they all render as black with nothing apparently on them, then you need to go back into the material settings for each material, texture and/or object and check to make sure that the"Shaders" panel "Amb [value]" in the EDIT buttons window isn't set to "0.000"; ambient occlusion requires the "ambient 'light'" in Blender be set for it to work correctly so if this is turn 'off' or 'zeroed' then that's the likely cause of ambient occlusion bakes being black.
To fix black AO renders simply set the "Amb" slider to the default value of "0.500" (or above) and re-bake the AO; you should now find the AO bake process rendering out a proper AO map.