Hammer won't load custom shaders

[GeorgiaPeanuts]

If you have custom shaders hammer will not load them. Models using them will be invisible, and brush faces using them will be wireframe.

edit: this is fixed by calling hammer.exe with parameter -game "path_to_your_mod"

[AniCator]

What kind of a custom shader is this?

[GeorgiaPeanuts]

Any custom shader I assume.

Specifically in my case I have a custom water shader where I implemented the white paper from Portal 2/L4D2. It shows in-game, but is invisible in hammer

On models, I have a version of VertexLitGeneric where I added masked color tinting and a low-tech translucency/subsurface scatter effect (uses precomputed thickness map).

I don't think the specifics of the shaders matter though, I suspect you could make any shader and it wouldn't show. I know in source2k7 you couldn't use LightMappedGeneric custom shader far as I could tell... My custom shader wouldn't get lightmaps so I would have to swap shaders in vmt during compile and then swap back to my custom one and it worked, but I suspect I could solve that looking in the compile tools.

[AniCator]

I'm currently having issues rendering my shaders in Hammer as well but there is a good chance it is to do with my coding. I've been trying to port this custom SDK_LightmappedGeneric for a while now from 2007 to 2013 but it hasn't been going too well.

I'm assuming you're seeing something like this in Hammer when using custom shaders. http://i.imgur.com/kv26TS7.png

[GeorgiaPeanuts]

If you mean the invisible floor, then yes. My shaders work fine in-game, it is only in hammer they just don't show up. I can show a picture when I'm home.

[AniCator]

Just tested it with a simple shader that outputs only a red colour. Works fine in-game but is a no-show in Hammer.
In Hammer: http://i.imgur.com/MllaVUx.png | In Game: http://i.imgur.com/98AmJqs.png

[alanedwardes]

I've done some investigation as we just switched our mod over to use the SDK_ shaders as we did with Source 2007, and it seems that Hammer exclusively loads stdshader_dx*.dll, it doesn't try to load game_shader_dx9.dll at all.

The fix for this in VBSP (when it doesn't mark faces for a lightmap compile) is to add a call to g_pMaterialSystem->ModInit(); in utilmatlib.cpp, the method InitMaterialSystem();.

I'd assume Hammer needs the same treatment. This is quite urgent, as there appears to be no workaround.

[k0091601]

If only we had hammer's source code! this really needs fixing. does anyone know if hammer src is coming in a future update?

[AniCator]

Having Hammer's source code would be rather pointless I think. We just need it to load the game_shader_dx... files so it knows how to deal with our shaders in the viewport.

[AniCator]

So I've found a fix. It isn't the best fix probably but it works some magic.
It turns out in our case it has to do with the VPROJECT settings I believe.

So our fix is rather simple.
"hammer.exe -game "path to mod directory"
It looks really silly doesn't it?

Image of the DLL loaded in Hammer. Attached to Hammer via Visual Studio and looked at the loaded modules.

[alanedwardes]

Just to add to @AniCator's comments - you can tell it to look at the sourcemods directory too for the -game parameter.

The model viewer, faceposer and other tools in the SDK do need that -game parameter to function, but I removed it from our Hammer startup script because it showed this warning when starting if you supply it:

Looking back at our Source 2007 script for starting that version of Hammer we did supply the -game parameter there, so this is consistent with 2007.

So this may not be an issue, however perhaps you might consider revising that error message displayed in Hammer, as it is misleading.

Edit: Just found out the warning message shows because one of the slashes in the mod path is the wrong way around. /cries

So @AniCator's fix above is all's you need.

[GeorgiaPeanuts]

Awesome this worked. It was odd though as I called directly hammer.exe which would bring up a ui allowing me to select from game configurations, so I assumd it would pull the game value from there. Guess not :P