Add custom shader template support to rendering server (core)

[BastiaanOlij]

After discussing during a render meeting some weeks back, we discussed reviving #94427 but breaking it up into parts.

This PR purely adds in the core of the implementation. E.G it introduces the shader template framework and allows us to set a shader template for a .gdshader file. It does not touch the BaseMaterial class nor does it add the breakup of shaders.

I'm still basing this on the .gdtemplate file extension but the logic has been expanded that it can include files though there is still a pathing limitation here. If built-in include files are specified, the include directive remains in the source, but files on disc are loaded immediately and embedded in the imported resource.
For now I've removed the ability to specify multiple renderer shader code, the files just become unwieldy long and its easy for the developer to just switch out shaders.

Currently only Mobile and Forward+ work, still contemplating to add support for compatibility but that may become a separate PR.

I'm still unsure if we've got a few things in the right place, too much of the previous system assumed there would be only one ShaderRD instance centralized and this is no longer the case. Definitely the grouping system and possibly the mutexes applied may need a bit more thought.

Finally we discussed that instead of trying to expose more of the shader code through build-in includes, we build the shader with the complete default template code. In my sample project I've extracted what I need, but the idea was to add a feature to save the built-in shader template so the user can edit it to their hearts content. They will need to be made aware that the code will likely only work with that version of Godot.
This still needs to be added.

[BastiaanOlij]

Update test project for now is in this branch: https://github.com/BastiaanOlij/test-custom-shader-templates/tree/ldotn_template

[BastiaanOlij]

@clayjohn @DarioSamo I think this is pretty much in working condition and as far as I want to take the foundation. I am tempted to add template support to the compatibility renderer as well but might do that in a follow up.

A few things I'd like to discuss with you before taking this out of draft.

• Dario, for you specifically, I ended up not touching the mutex around shader compilation, so that works as before. I'm still tempted on moving it into the ShaderTemplate class but we'd need to move part of the pipeline creation in there then as well.
• I'm also not 100% sure if we've got caching right, and if there is any further impact on the shader baking logic. I think it works correctly, but could use an extra set of eyes on this.
• There is a bit of confusion with naming atm. User shaders now refer to the template they are using, but inside our template the object that contains the template code is also called shader for historic reasons (since we only ever had one). I wonder if we should choose some better names here (e.g. MaterialStorage::ShaderTemplate::shader), same with all the shader_template_* functions that now refer to the shader, we may want to come up with better naming.
• Even though the mobile renderer now uses shader groups, we still have some left over functions around versions.
• And on the subject of shader groups, I wonder if we should store these differently. Because we only ever had one template, we store the groups on that level, but now that we have multiple templates, they are enabled in unison. I think we can come up with something better for this.
• Currently you can't edit .gdtemplate files from the editor and we don't have functionality to save the built-in template(s) to a .gdtemplate file to function as a starting point. I think we should do this as a separate PR (same as with eventually deciding to have additional built-in includes). For now this sample project can be used but it includes only a bare bones implementation.

[HydrogenC]

Can we have more uniforms than the standard include files specify? It would be common to bind some extra uniform buffers for custom shading models.

[BastiaanOlij]

Can we have more uniforms than the standard include files specify? It would be common to bind some extra uniform buffers for custom shading models.

At this point the only way to get extra uniforms in, is to add those in through the user shader that is embedded.

There was someone who did a PR on my old implementation to add in the ability to define additional uniforms in the template that would be exposed on the material. I think its worth revisiting that but as a follow up to the initial implementation.

One of the difficulties of getting this merged initially was that we were trying to do too much and reviewers don't see the wood for the trees anymore.

[Calinou]

Tested locally, it works as expected.

shader_template_reload.mp4

Some feedback:

• Class reference XML files need to be updated with --doctool, and descriptions for new items must be filled in.

• It's possible to have multiple shader_template declarations in the same shader, without any errors:

It's not clear which one "wins" in this situation, so I'd say this should be an error, just like duplicate shader_type or render_mode declarations.

• Shader template reload on changes only affects the editor, not the running project. That said, we don't have live reloading for .gdshader in the first place.

• I'm not sure if the .gdtemplate extension is ideal; it could be referring to any kind of template such as a script or export template, not necessarily a shader template. I can't think of any better alternatives right now, still.

[blueskythlikesclouds]

Maybe I missed something, but it doesn't seem like new templates inherit previously enabled shader groups? Or does that not matter?

[BastiaanOlij]

Maybe I missed something, but it doesn't seem like new templates inherit previously enabled shader groups? Or does that not matter?

This is one that could probably do with a rethink and moving the group logic into the templates at a later date. However for now this logic remains implemented on the ShaderRD level. You can see that in both the forward+ and mobile renderers, when groups are enabled this is applied to all templates.

[BastiaanOlij]

@Calinou got all of your comments sorted.

Docs should now be updated, and it should now check for double inclusion of "shader_template" in the user shader.

Live updating seems like a whole different thing to tackle, especially as this would be the same workload as making this work for gdshader, I think we should keep that out of scope for this PR.

Finally, agree that gdtemplate is a very ambiguous name. But I haven't heard an alternative yet that works better without it becoming very verbose. Totally up for suggestions though...

[BastiaanOlij]

Marked this as ready for review, I'm sure there will be more requests for changes.

[paddy-exe]

Finally, agree that gdtemplate is a very ambiguous name. But I haven't heard an alternative yet that works better without it becoming very verbose. Totally up for suggestions though...

Some ideas that would come to mind for me would be:

• gdshader_template or template_gdshader
• gdshader_templ or templ_gdshader
• gdshader_type
• gdshader_model

I would be in favor of having the connection with the gdshader file extension be clear from the template file extension. That might be verbose but I am not sure if being more verbose would be such a big concern. I mean for those that need to edit their paths for the .gdextension file to load the correct binary files they aren't complaining that much either about this specifically (and those string paths are LONG). You aren't doing it that regularly anyways as I imagine.

[AThousandShips]

Suggested change

	shader_name = shader_name.replace("/", "_");
	shader_name = shader_name.replace_char('/', '_');