Topic: RtCW SP Shader overflow bug

[kat]

Problem
If you've done any single player level design for Return to Castle Wolfenstein you may have come across this bug where the textures appear to get messed up in game. It's usually the associated with weapons flashes and explosions, or are the most noticeable offenders. The reason for this bug is a 'shader overflow'; but it's commonly called the 'weapons flash bug'.

RtCW weapons flash shader overflow bug
Basically this happens when you've got a map that's generating too many lightmap/shader combinations and has gone over the limit.

Design note: it's to do with how many unique lightmaps the brushwork creates not how much lightmap data is being generated as part of the overall KB/MB file size of the BSP. Each time you chop a brush up the textured surfaces create an associated lightmap, the more faces you have the more unique lightmaps that get created.

Next time you run a compile take a look at the last few lines in the BSP monitor during the LIGHT stage of the process (or file dump if you do them from BAT files) and note the number that's listed next to 'shaders / lightmap combos' (or words to that effect - I've not done a BSP for a while so can't recall off the top of my head what is says exactly), if that number is greater than 390 then you're stuffed.

To fix the problem you have to go back through the map and be mercenary about what *needs* to be there and what doesn't, you've essentially got to limit or reduce the number of shaders you've used.

It's also worth pointing out here that the bug isn't necessarily related to the number of surfaces you have but more the number of different shaders you've used and the resulting lightmap data that then generates (although it's been a while, I think that's correct).

RtCW shader overflow quick fix
As a quick fix use this switch in the LIGHT stage -nocollaspe. If you still get the error after than then I'm afraid to say you have a lot of optimizing to do and there isn't a short cut to fixing this issue either.

Some things that might work to fix the problem;

• Create new single textures by combining two or more together; like a step texture and a floor trim used on the edge of the step - instead of using two separate textures (and hence shaders) you could combine them into a single image for use in game.
• Complex geometry should be converted to ASE models which is then player or weapon clipped - as they already have heavily optimised lightmaps..
• Any non-axial brushwork (like cracks, rocks, broken walls etc..) should be converted to ASE models and clipped appropriately as above.
• Try caulking every hidden face, esp. on detail flagged brushwork as they're not always culled by the compiler.
• Try converting patch mesh objects into either ASE models or brushwork - the crypt vaulting in my last SP was brushwork and not patch meshes.
• Try not to break brushes into too small or unnecessary sizes.

By the way, depending on what version of GTK you have you're more than likely to already be using q3map2

Some additional shader overflow fix information

There are ways to fix the problem depending on how bad the weapons_flash bug is - you may noticed that cutting back and simplifying brushwork will result in the weapons_flash bug decreasing in severity, the flamethrower for example would gradually become partly functional, then a bit more until it was a complete flame.

Some known things to try to 'fix' the bug (may be repeats of above info);

• Reduce the number of faces in the map, if you've got tonnes of 3pt clipped or vertex manipulated brushwork remove as much of it as possible and leave only the bare minimum for the effect.
• Try func_group brushes that have more than one face in 'coincidence'.
• Try 'merging' as many brushes together as possible.
• Allied to func_grouping above... 'convert' any big 'terrain' sections into just that, a terrain mesh by adding the following key/value settings into the entities world spawn;
  • key = "terrain", value = "1"
  • key = "layers", value="1"
  • key = "shader", value = "textures/yourfolder/yourtexture"
    [note: you don't need to add an actual file extension here .tga]
  • key = "alphamap", value ="maps/yourfile.pcx"
    [note: this is basically a white texture that's been saved as a paletted image (256 colour palette), I've also not had any problems with placing it in various locations so long as the worldspawn path is correct.
• use q3map2 with the -nocollapse.
  What this does is force q3m2 to create unique lightmaps for everything instead of 're-using' similar lightmaps on other objects.
• q3map2 with -meta or -patchmeta.
  Results for me with this were unreliable, it does 'odd' things to the lightmaps and created some heavy shadows where it should have done...!
• gridsize xxx xxx xxx does work but it really makes the lighting go noticably grainey. The default setting are similar to the following;
  
  • key = "_gridsize", value = "128 128 256"
    Changing the 'value' reference to "256 256 256" or above will increase the size of the lightmaps (by decreasing their resolution - same sized images are stretched over a larger area) and should reduce the number of unique lightmaps that are generated.