Blender 3D, baking Ambient Occlusion & terrain
Producing an Ambient Occlusion map for a terrain mesh, part or whole, through the use of Blenders Texture Bake system is usually a relatively straightforward procedure that differs very little from processing other types of object, a simple chair for example. Where there are differences, they typically relate to the physical size of both the mesh being baked and the texture being baked to, and whether or not the results are to be used within the context of an interactive environment where various optimisation's and texture restrictions, in place to encourage efficient and speedy real-time usage, typically influence the outcome.
The following tutorial, split into two sections - the first covering the mechanics of the process, the second, understanding some of the broader issues associated with the baking terrain meshes in this way - assumes the resulting Ambient Occlusion maps are to be used in some form of 'real-time' environment or a game development or content production process therein, although the techniques and understanding can be used to wider effect. A basic understanding of Blender 3D is required to get the most from the below material.
The basics of Texture Baking ^
As briefly mentioned in the introduction, through the use of Blenders "Texture Bake" rendering sub-system, particular aspects of a terrain mesh can be 'translated' into one or more 'types' of image that can be used to perform different functions within the context of real-time environments; a procedural material could be 'made real' and then used to replace the very same data it was baked from. Similarly, the way structures, surfaces and objects affect lighting, how their interplay occludes it to create Object 'shading' can also be rendered. This in essence, is what "Ambient Occlusion" is and why it's useful to bake it.
Terrain & basic Material assignments ^
For "Texture Bake" to work correctly all Objects to be processed need at least one Material assignment, which in turn needs at least one 'image' associated with it - this is what 'bake' data is written to. In other words it's important to make sure that by default, Materials are set up so they contain the standard "Material", "Texture" and "Image" slots otherwise baking may not work correctly. So, if there are no assignments yet or nothing has yet been created, select an object and click the "Material" Properties button, click "New +" to add an entry, which will then appear in the Material List. Change the "Diffuse" colour if necessary by clicking the colour sample area, selecting a colour from the pop-up - this helps distinguish one mesh or Material from another if several are being used or worked on at the same time.
Design Note: if a Material already exists simply edit the name and Diffuse colour as needed - click the input field, "Unique datablock ID name" appears when hovering the mouse, and type a new name to distinguish the material from others that might be present. And/or click the colour sample field to change the materials solid shading appearance.
Making and/or editing and assigning a basic Material to the terrain mesh. Select "Material" properties and either add a "New" entry or change the basic settings of any already available datablock as required - change the "ID" name and/or "Diffuse" colour
Next, click "Texture" Properties then "New +" to create a new datablock and associated properties if one doesn't already exist. Change "Type:" to "Image or Movie" and scroll down to the "Image" sub-section, click "Open" to add an image; once loaded into "Image" properties it will be referenced and appear as a "Source:" data location. Finally in "Mapping", change "Coordinates" to "UV".
Design Note: Blender defaults to using UVW's as the basis upon which textures are baked, ignoring other "Coordinates" types. However, to avoid problems elsewhere it's generally a good idea to set this Mapping property to "UV" regardless.
Adding a "Texture" slot with an associated bitmap. Select "Texture" properties and either add a "New" entry or edit an existing one. Add a new Image slot and load in a bitmap, then change the "Coordinates" data to "UV" (not shown in the above - scroll down the properties to the "Mapping" sub-section)
Using Blender generated textures ^
Typically images assigned to Materials already exist as 'bitmap' assets, "*.tga" for example. For the purposes of texture baking this is not an absolute necessity because Blender has the ability to generate 'procedural' image data that can be assigned and used in much the same way as bitmaps.
To make a 'generated image', after the initial "Image" slot has been created and appropriate options made available, click "+ New" instead of "Open" to access the "New Image" properties dialogue. Change the "Name" (LMB click and type, in this instance "TerrainTex") and change it's size to something usable - remember that textures need to follow the general conditions associated with the "Power of Two" rule. Activate "UV Test Grid", and for the purposes of baking, de-activate "Alpha" (not generally needed for this type of texture baking). Click "OK" to be returned back to the previous options with the new data generated.
Design Note: image size can be made reasonably large this way but bear in mind that extremely large files will take much linger to render - making a texture twice as big actually means rendering four times the initial amount of data; changing a 1024x1024 image to 2048x2048 means the texture now occupies an area equivalent to 4x 1024.
Finally, to the left of "+ New" click the "Sphere" icon - "Browse Image to be linked" - and select the entry just created from the list, in this instance "TerrainTex", this assigns the datablock to the "Image" slot and to the Material overall. If everything was done correctly the preview pane will update itself showing a 'checker' image, indicating a successful creation and associated of the generated image data.
An alternative way to assign an "Image" to a Material is to use Blender to generate one. Click the "+ New" button and change the properties associated with the "New Image" slot, click "OK" and be returned back to "Texture" properties with an 'image' waiting to be assigned
With the "New Image" data created, make sure the "Texture" slot is selected in the "TextureList" and then click the "Browse Image to be linked" button, select the entry just created to assign it to the Material and watch for the 'preview' pane to update showing successful application
UVW unwrapping & textures ^
Before unwrapping the terrain mesh use "Alt+Z" to first switch "Viewport Shading" in the main 3DView to "Textured", this displays the Materials image so it can be seen during the following process. Once done change one of the available viewport's so the "UV/Image Editor" is visible - this can be done using "Shift+F10" (changes the view over which the cursor is currently active) or by selecting the "UV/Image Editor" option from the "Editor Type" list for a given viewport (click the button on the left of a 'Header'). Doing this simply means being able to see and edit various aspects of the process without unnecessarily switching views and/or editors back-and-forth.
Design Note: if using the "Shift+F10" shortcut switching between the main 3DView and UV/Image Editor, use "Shift+F5" to switch back (make sure the cursor is over the area when doing this else the shortcut will change the wrong viewport).
If not done already, changing one of the available viewport's so the "UV/Image Editor" is visible, this will help the process of texture baking as both the UVW map and resulting bake will be seen during and immediately after completion
With Blenders layout changed, select the terrain mesh to be unwrapped and press "Tab" to enter Edit mode, use "A" to "Select All" - UVW's are mapped and edited based on selection - then press "U" to access the "UV Mapping" menu. From the list of available options select "Unwrap" to create a basic UVW map of the entire mesh. On doing this both the "3DView" and "UV/Image Editor" windows will update, the former displaying the terrain as a flat lit 'white' mesh (shown below) indicating it has been UVW unwrapped but not yet assigned an image; the latter to displaying the new UVW 'map' atop a grid representing the Objects "Texture Space" (top-right below).
Design Note: in the absence of both a UVW map and/or a texture assignments, the mesh will be coloured initially based on the 'diffuse' property of any applied Materials - so long as viewport shading is set to "Textured", the objects appearance will change once it's been unwrapped, else it will remain diffuse colour tinted.
The area shown as the active "Texture Space" will be a 'square' or 'rectangle' depending on the ratios and physical dimension of the assets to be loaded in - the default is 'square'. Any none-square assets will be stretched and resized to fit.
Press "U" to unwrap the mesh and generate a basic UVW Map - the mesh updates to display white in the 3DView because an image has yet to be assigned to the map. Note also the "Texture Space" grid visible in the "UV/Image Editor" top right
With the mesh unwrapped, next the image needs to be assigned to the UVW map. From the "UV/Image Editor" click the "Browse image to be linked" icon and select from the list of available entries the material image previously set-up, this will associate the texture with the highlighted UVW map which should then appear assigned to the model in the 3DView. The purpose at this point is to produce a basic UVW/texture assignment, but if need be the map can be edited in the UV/Image Editor using the normal tools associated with manipulating Objects and elements in the main 3DView, so "G", "R" and "S" can be used to "Move", "Rotate" or "Scale" selections to change or amend the UVW layout, gaining better placement and/or more texture space where necessary.
Design Note: terrain meshes usually result in rather awkward UVW map layouts that, whilst they can be edited, don't or won't really proffer up much better utilisation of space than originally calculated by Blender, so tweaking the layout without doing some major editing may not be of much use.
Terrain mesh UVW unwrapped and assigned a texture image ready for baking the Ambient Occlusion map. Note the above uses Blenders standard "Unwrap", it's generally typical that some minor adjustment can be made but there won't necessarily be much more gained from doing so
UVW maps & generated textures ^
UVW mapping generated images works in much the same way as using standard bitmap images, the only difference is their needing to be saved once applied. So, as per the above and using bitmaps, assign the texture to the terrains UVW's in Edit mode by selecting the entry from the "Browse Image to be linked" list in the "UV/Image Editor" (in this instance "TerrainTex"), this will change the "Image" menu option to "Image*" (the addition of an "*") indicating the datablock has now been made real and needs to be saved. Click "Image*" and select "Save As Image" from the available options.
Once the Blender generated 'image' is assigned to both the Material and mesh UVW map, making it real in the process, it's best to save the image before continuing so there is a 'fixed' reference point for the asset prior to baking
In the "Browse" view that opens select the location to which the file is to be saved, change the name of the new asset as appropriate (this defaults to the images "ID", 'TerrainTex' in this instance). Select the image format to be used, making sure to use a 'loss-less' format at this stage - uncompressed *.tga for example[3,4]. And then finally click the "Save as Image" button; Blender will write the data to file and return to the previous view with the asset now 'real' and ready for further work.
Design Note: once a generated image is saved it become real and can be used and manipulated in exactly the same way as if it were a pre-existing asset loaded into Blender and assigned to the mesh and Material.
Select a "loss-less" format to avoid compression artifacts that may cause problems later on when editing or using the image elsewhere in the process (when combined with the diffuse layer for example)
Baking the terrains ambient occlusion ^
Before any baking is done it's best to move, "M", or delete, "Del", any "Lamp" Objects and/or other illumination the scene might contain. Whilst these types of item do not typically have a direct influence on texture baking itself, they do tend to interfere with the management of the process - because lamps and other types of lighting 'shade' meshes in the 3DView, their presence can cause confusion when trying to determine how the mesh is being 'lit' versus what is being displayed as 'baked' occlusion. On moving or deleting, the mesh will appear 'flat lit' (without shading or illumination - shown below) in the 3DView and be ready for processing.
Design Note: moving lights and other objects can of course be done at any point in time. It's preferable however, that it should be done before baking anything.
Moving "Lamps" and unnecessary Objects to a different layer so they don't interfere with the process and so the resulting ambient occlusion texture can be seen clearly over the mesh once its baked
With the scene now flat shaded but "Textured" ("Alt+Z"), click the "Render" properties button and scroll down to the very bottom of the panel to the sub-section titled "Bake". Click the arrow to the left of the heading to display the options where necessary. From here, first change "Bake Mode:" to "Ambient Occlusion" by selecting the option from the drop-down list, a new set of properties appear, click the checkbox to activate "Normalized" and leave the remaining settings as-is. The selected Object can then be processed by clicking the "Bake" button - progress will be displayed and real-time updated in the UV/Image Editor, or on completion the resulting texture will temporarily replace the image originally mapped to the object, confirming the process as well as providing an opportunity to see the results in-situ before saving.
Click "Render" properties and scroll down to the very bottom of the list to the "Bake" sub-section (expand the options if not visible). Select "Ambient Occlusion" to access the appropriate settings...
...then make sure to set "Normalized" - everything else can be left 'as-is'. Click the "Bake" button to render ambient occlusion to the UVW mapped texture
Saving the baked Ambient Occlusion map ^
After bake is completed the resulting image needs to be saved, done from the "UV/Image Editor". Simply click the "Image*" menu, selecting "Save As Image" from the available options - this is preferable as it allows the texture to be saved separately to any previously existing asset/s whilst replacing the originally mapped image (so it can be seen applied over the mesh). This will open the "Browser" where the file can be given a name ("terraingrid_AO.tga" in this instance), a location to be saved can be found and the "File Type:" can be set - here it's preferable to use an uncompressed or loss-less image format so as to ensure the source for further editing is 'clean' and without the artifacts typically associated with JPG's or similarly compressed files - select "Targa Raw", "BMP" ("Bitmap"), "Tiff" or similar format. Once the necessary options are set, click the "Save As Image" button; Blender will save the file and return to the previous screen with the baked Ambient Occlusion map saved to file and displayed on the mesh.
Design Note: When choosing a 'save' option, note that "Save Image" will write the new texture data over the top of the original image mapped to the mesh (the original 'grid' texture used in this tutorial for example, 'grid.tga', would be physically overwritten by the new baked image - still called 'grid.tga'); "Save As Copy" saves a 'copy' of the texture without replacing the originally mapped image - to see this applied to the mesh would mean replacing the currently mapped asset with the newly created baked AO map. In addition, aside from the presence of the new texture, the only indication that anything has been done is the appending of a "*" to the "Image" menu text, "Image*", this means that if a bake session included more than one image and/or they were baked in addition to other assets, the "*" symbol would be the only indicator there was a difference between them and/or that an image has yet to be saved.
It's important to point out here that the baked image is being held in memory so if the *.blend file were reloaded (deliberately or inadvertently) the bake data would be lost, requiring the process to be redone.
From the "UV/Image Editor" select "Save As Image" to save the recently baked Ambient Occlusion map to separate file from the one originally mapped to the object - note this replaces said original in the Material
In the "Browser" be sure to use a "loss-less" image format where-ever possible, even if the final output is to be a compressed JPG it is better to have an uncompressed source file to keep image 'artifacts' to a minimum
Texture baking & Mesh Smoothing ^
Before finalising any saved Ambient Occlusion bakes, be sure to check that "Mesh Smoothing" is active via the "Shading:" property in the "ToolShelf" ("T"). For terrain this usually means just setting the entire mesh to be "Smooth" shaded so that surfaces are 'averaged' - although setting "Flat" won't prevent an AO bake being done, it will likely produce undesirable results (certainly in relation to what the object being baked is). If the mesh is to have 'hard' edges, keep in mind that when these types of surfaces or edges are baked to ambient occlusion they will generally form hard breaks between light and dark areas depending on the surface relationships - the result will be a hard line in the resulting texture which isn't usually appropriate for terrain Objects which are generally not lit in a way that would encourage that type of 'feature'.
Setting the terrains "Shading:" to "Flat" results in hard edges and features that may not be desirable or appropriate for an Object lit by ambient lighting - hard edges and surfaces are usually the domain of directional lighting
Using "Smooth" over the entire terrain produces ambient occlusion bakes that are more typical of this type of OBject, lit ostensibly through indirect background illumination
Side by side comparison of Ambient Occlusion maps baked with "Flat" (top) and "Smooth" (bottom) shading assigned to the terrain mesh
If the now saved Ambient Occlusion map is taken into an image editing application such as Corel Photo-Paint or GIMP, using one the the available image 'blend mode' filters, the shading attributed to the ambient occlusion image can be blended with an underlying terrain texture so that together, they produce a final 'diffuse' (or 'colour map) image with better 'depth' than can be had separately. This new version can be reassigned to the terrain mesh as a 'cheap' alternative to more modern and costly techniques to shade a game asset (shown below).
A basic 'terrain' texture shown with and without Ambient Occlusion merged into the 'diffuse' image which is then assigned to the models UVW map - the AO shading emphasises features and physical attributes of the mesh giving it more depth than otherwise can be had using diffuse (colour map) on its own