Using GLSL shaders in cocos3d

Why are you here??
 
Visit the wiki article that covers this topic on the new, shinier, and vastly improved Cocos3D website, instead!!

---

cocos3d is a sophisticated 3D application development framework for the iOS and OSX platforms. With the release of version 2.0, cocos3d now supports cocos2d 2.x and OpenGL ES 2.0 under iOS, and OpenGL under OSX, including the use of GLSL vertex and fragment shaders. This document provides an introduction to writing and using GLSL shaders within cocos3d.

[toc=”2,3,4″ title=”In this Document:”]

Introduction

cocos3d is a companion to cocos2d, a popular framework for building iOS and OSX games and applications that play out in 2D (or 2.5D isometric projection). Although on the one hand it is possible to start with the cocos3d Application template and develop a 3D application without knowing too much about the workings of cocos2d, to get the most out of using cocos3d, you should also familiarize yourself with cocos2d. You can learn more about cocos2d at the cocos2d Wiki.

cocos3d 2.x supports both the programmable pipeline of OpenGL and OpenGL ES 2.0, plus the fixed pipeline of OpenGL ES 1.1, with the same codebase. In fact, when developing for iOS, you can easily flip back and forth between testing your app against OpenGL ES 2.0 and OpenGL ES 1.1, allowing you to explore the trade-off between the ease of use of the configurable pipeline of OpenGL ES 1.1, and the additional flexibility and creativity available through the programmable pipeline of OpenGL ES 2.0.

Although cocos3d does support OpenGL, OpenGL ES 2.0, and OpenGL ES 1.1 from the same codebase, cocos2d does not. To use OpenGL or OpenGL ES 2.0, you must link with the cocos2d 2.x libraries, and to use OpenGL ES 1.1, you must link with the cocos2d 1.x libraries. In an iOS cocos3d project, you can easily flip back and forth between using OpenGL ES 2.0/cocos2d 2.x and OpenGL ES 1.1/cocos2d 1.x by simply replacing the cocos2d group in the Xcode Project Navigator panel. The README.md file in the cocos3d distribution and the README file in the iOS cocos3d Application project templates provide instructions on how to move back and forth between versions of OpenGL ES & cocos2d.

While running under OpenGL ES 1.1, any shaders defined or loaded by your application will simply be ignored.

Although you will certainly want to write your own GLSL shaders, cocos3d comes with a set of standard default shaders that are automatically selected and applied to nodes that do not explicitly specify a shader. This facility allows you to add a model to your scene without first having to develop the GLSL shader code to render it, and it allows you to take an app that you built under OpenGL ES 1.1 (in pre-2.0 cocos3d), and run it out-of-the-box under OpenGL ES 2.0.

Loading and Compiling a Shader Program

In OpenGL, a GLSL shader program is comprised of two parts: a vertex shader which is run for each vertex in the mesh and determines the location and style of each vertex, and a fragment shader, which is run for each fragment (pixel) that is rendered to the screen, and determines the final look of each pixel. These two shaders must work together and be linked together into a single GLSL shader program.

In cocos3d, each shader program is represented with an instance of the CC3ShaderProgram class. Reflecting the structure of a shader program in OpenGL, each instance of the CC3ShaderProgram contains two instances of the CC3Shader class, one representing the vertex shader, and the other the fragment shader.

You can load, configure, and link a CC3ShaderProgram instance using the class-side programWithSemanticDelegate:fromVertexShaderFile:andFragmentShaderFile: method, which takes the file names for the vertex and fragment shaders, loads and compiles each of the two shader files, and links them to create the shader program.

Shader programs are cached. The first time this method is invoked, it automatically adds the shader program to the shader program cache. Subsequent invocations of this method with the same shader file name arguments will retrieve the existing shader program instance from the cache, instead of loading, compiling and linking another version of the same shader program. You can therefore use this method whenever you wish to attach a shader program to a mesh node, without having to worry about incurring the overhead of loading, compiling, and storing multiple copies of the same shader program.

The following example shows how an application can use this method to add a shader program to a mesh node:

[objc]
CC3MeshNode* aMeshNode = …;
aMeshNode.shaderProgram = [CC3ShaderProgram programWithSemanticDelegate: self.mySemanticDelegate
fromVertexShaderFile: @"MyVertexShader.vsh"
andFragmentShaderFile: @"MyFragmentShader.fsh"];
[/objc]

The mySemanticDelegate property method is maintained by the application and returns an instance of an implementation of the CC3ShaderProgramSemanticsDelegate protocol, which semantically resolves the GLSL attribute and uniform variables in the shaders into content within the 3D scene. The shader program automatically populates the attribute and uniform variables, from content automatically extracted from the mesh node and the scene, each time the shader program is used to render a mesh node during each frame. By using semantic mappings, the application never needs to figure out how to set the attributes and uniform variables within the shader program. It will happen automatically.

The following is an example of an application implementation of such a mySemanticDelegate property method:

[objc]
/** Returns the semantic delegate to use with my shader programs. */
-(id<CC3ShaderProgramSemanticsDelegate>) mySemanticDelegate {
if ( !_mySemanticDelegate ) {
CC3ShaderProgramSemanticsByVarName* sd = [CC3ShaderProgramSemanticsByVarName new];
[sd populateWithDefaultVariableNameMappings];
_mySemanticDelegate = sd; // retained by new above
}
return _mySemanticDelegate;
}
[/objc]

In this example, the application is making use of the same GLSL variable naming conventions used by the standard shader programs provided as part of the cocos3d distribution. However, if you already have a wealth of GLSL shader code from other sources that uses a different naming convention, instead of having to rename all the attribute and uniform variables in that GLSL code, you can define a different semantic mapping for those shaders. The Shader Program Semantics section below describes the semantic mapping process in more detail.

Program Matching

The CC3ShaderProgram class retains an instance of an implementation of the CC3ShaderProgramMatcher protocol in the programMatcher class-side property. The purpose of this program matcher singleton is to automatically match an appropriate default shader program to a mesh node, if you do not explicitly specify a program for that mesh node. This mechanism allows you to add mesh nodes to your scene without having to worry about defining exactly which program shader is required to render it appropriately.

The first time a mesh node is rendered, if it does not have a shader program defined, it is passed to the program matcher in the class-side programMatcher property to match the mesh node to an appropriate shader program. Since the loading and compiling of a new shader program can take time, you can also use the same mechanism to establish the shader program for the mesh node at initialization time, using the selectShaderPrograms method on a single mesh node, or a node assembly, or even the entire scene.

The default CC3ShaderProgramMatcher implementation is CC3ShaderProgramMatcherBase, which defines, loads and caches a standard set of shader programs to apply to mesh nodes based on certain matching criteria, such as whether the mesh node has one or more textures, or whether it is using bump-mapping, etc. You are free to replace this default implementation with your own program matcher implementation and use this mechanism as your primary means of assigning shader programs to mesh nodes.

GLSL Variables: Vertex Attributes and Uniforms

The compiled GLSL vertex and fragment shader code executes on the GPU. Like any program, the shader code needs access to external content in order to know how to position vertices and render visual content. In OpenGL, this external content is passed to the shader program through vertex attribute variables and uniform variables.

Vertex Attributes

As the name implies, a vertex attribute is a shader program variable that describes an attribute of a vertex. Examples of vertex attributes are the location, color, or texture coordinates of a vertex. Each vertex attribute is passed as an array to the shader program, one element per vertex, but within the shader program, each vertex attribute appears as a single element, since the vertex shader is run for each vertex independently.

Vertex attributes can only be passed to the shader program semantically. This is handled automatically for you, by the CC3VertexArray subclass instances contained within each CC3Mesh.

Uniforms

A uniform variable is a general environmental or contextual parameter passed to the shader program by the app to help the shader program determine the location and visual style of each vertex and fragment. Examples of uniforms might be vertex transformational matrices, light positions and characteristics, material color, and texture identifiers. The app is also free to define and pass any other parameters that might be of interest to the shader program.

Uniforms can be passed to the shader program manually by the app, or more commonly, may be passed semantically, as described in the section below on shader program semantics.

Manually Setting Uniform Variables

The most direct way to set the value of the uniform variables in a shader program is to set them manually from the app, during the update of any particular mesh node. The mesh node’s material contains an instance of CC3ShaderProgramContext, which manages the context for the shader program when it renders nodes covered with that material. To set the value of a uniform variable for the shader program used by that material, you can request the variable from the program context using the uniformOverrideNamed method, and then set the value of that variable to the value you want to use when rendering nodes with that material. You can do this as follows:

[objc]
CC3ShaderProgramContext* progCtx = meshNode.shaderContext;
CC3GLSLUniform* uniform = [progCtx uniformOverrideNamed: @"u_cc3MaterialDiffuseColor";];
uniform.color4F = kCCC4FMagenta;
[/objc]

In this code, u_cc3MaterialDiffuseColor is the name of a GLSL uniform variable, as declared in the shader source code. Each uniform variable defined in a shader program is represented by an instance of CC3GLSLUniform. That class has a number of different setter methods for setting the value of the uniform, depending on how the uniform variable has been declared within the GLSL source code. The value of the variable can be set as an integer, float, vector, color, or matrix type, either as a single value, or as an element in an array. See the cocos3d API documentation for more information on CC3GLSLUniform and the methods available for setting uniform values.

Although uniforms can be set directly, relying on doing so for all uniforms would burden the application with an enormous amount of glue code. Instead, the recommended mechanism for setting vertex attribute and uniform variable values is to define program semantics, which provide automatic mapping between declared GLSL variables, and contextual content. You can then use the manual uniform setting techniques described in this section to override the values derived semantically, if needed, or to augment and populate uniform variable values that cannot be derived automatically by semantic. Program semantics are described in the next section.

Shader Program Semantics

Shader program semantics is the declaration of a consistent semantic mapping between the vertex attributes and uniforms declared in a GLSL shader program, and the contextual environmental content of the scene. This means, in essence, declaring that a particular uniform, for example the u_cc3MaterialDiffuseColor uniform variable identified above, semantically means the diffuse color of the material covering the node currently being rendered. By building up a mapping between the attribute and uniform variables and their semantic meanings, we allow the framework to automatically extract and apply the appropriate content (in this example, the current diffuse material color) as the shader program renders each node. This occurs automatically, and the application need take no action during the updating or rendering of any nodes.

An extensive default and standard set of content semantics is defined in the CC3Semantic enumeration. This enumeration is intended to be open-ended, and you are free to define further content semantics that may be of use in your specific vertex and fragment shaders. See the documentation and declaration of the CC3Semantic enumeration for more information on these standard program content semantics, and how to add your own custom content semantics.

As explained previously, each CC3ShaderProgram holds onto a special delegate object in its semanticDelegate property. This delegate is an implementation of the CC3ShaderProgramSemanticsDelegate protocol. During program linking, the shader program automatically invokes the configureVariable: method on this delegate to map each variable to a program semantic. Then, during runtime node rendering, whenever the shader program needs to populate the value of a variable, it uses this semantic relationship to automatically populate the value of the vertex attribute or uniform variable.

For vertex attribute variables, the semantic is automatically resolved to one of the CC3VertexArrays held by the CC3Mesh. For uniform variables the shader program invokes the populateUniform:withVisitor: method on the delegate to retrieve the appropriate uniform value from the environment. An implementation of this method may retrieve the uniform value from the mesh node itself (as in the example above of the u_cc3MaterialDiffuseColor uniform given above), or it may look further into the scene or runtime environment for the uniform content (for example, to retrieve lighting or camera environmental content).

You are free to create any implementation you wish of this protocol, but cocos3d provides the CC3ShaderProgramSemanticsByVarName class as a standard default implementation. If you want to define your own semantic mappings, one very convenient way for you to extend the standard variable semantics, or define new semantics, is to add a method to the CC3ShaderProgramSemanticsByVarName class, in an Objective-C Category, and have that method invoke the mapVarName:toSemantic: method for each of your own uniform variable names, to map each to a standard, or custom, CC3Semantic.

CC3ShaderProgramSemanticsByVarName

CC3ShaderProgramSemanticsByVarName is an implementation of the CC3ShaderProgramSemanticsDelegate protocol that maps variable names to semantics, and provides several methods to allow you to instantiate an instance and add mappings between each variable name and the corresponding semantic.

The populateUniform:withVisitor: implementation of CC3ShaderProgramSemanticsByVarName automatically resolves the standard semantics defined in the CC3Semantic enumeration (for both vertex attributes and uniforms) into environmental content.

You can invoke the populateWithDefaultVariableNameMappings method on an instance of CC3ShaderProgramSemanticsByVarName to populate the mappings for a comprehensive set of standard variable names. You can then use these standard variable names in the GLSL source code of your vertex and fragment shaders to automatically extract contextual content for your shader variables. All of the default shaders provided in the cocos3d distribution use the standard variable naming convention defined by this populateWithDefaultVariableNameMappings method.

Please review the source code implementation of the populateWithDefaultVariableNameMappings method for a full list of these standard variable names. If you don’t like these standard variable names, you can also write your own variations of a method like populateWithDefaultVariableNameMappings to establish your own set of standard variable names and relate them to the appropriate CC3Semantic semantic values.

Loading Shader Programs and Semantics using Effects Files

If you are using existing vertex and fragment shaders, or shaders that have been developed for running in other environments, you may be in a position of having GLSL source code that does not follow a standard variable naming convention that can be easily mapped to a standard set of semantics. In this case, cocos3d allows you to define the mapping between GLSL program variable names and program semantics using effects files, which are loaded at runtime when you load the shader programs.

As the name implies, an effects file contains the definitions for a set of effects, where each effect collects together the vertex and fragment shaders, declarations of the vertex attribute and uniform variables expected by the shaders, the semantics of these variables, and optionally, the textures that the shaders are expecting to apply to a material.

cocos3d uses the PFX effects file format from the PowerVR SDK by Imagination Technologies, the supplier of the GPUs used in the iOS devices. The PowerVR SDK is the same toolkit that is the source of the POD model file exporters. You can download this SDK and toolset here. The structure and content of the PFX effects files is described in the document PFX Language Format Specification within the PowerVR SDK download.

PFX files provide the opportunity to declare each vertex attribute and uniform, and to define a semantic for each that cocos3d can then automatically map to one of the semantics in the CC3Semantic enumeration.

You can apply an effect to a mesh node as follows:

[objc]
[meshNode applyEffectNamed: @"EtchedEffect" inPFXResourceFile: @"MaskEffects.pfx"];
[/objc]

where EtchedEffect is a named effect contained in the PFX effects file MaskEffects.pfx.

If your mesh node is loaded from a POD file, you can also identify which effect and effect file to load in the definition of each material within the POD file. This provides a completely automated resolution. You simply load the POD file from your app, and the POD file will then automatically identify and load any PFX files needed to provide effects and semantics for the materials described within the POD file. You can use the PVRShaman tool from the PowerVR SDK toolset to edit your POD file to set the effect to use for each material in the POD file.

Under the covers, the PFX file is loaded using the CC3PFXResource class. Like other resources, loading a CC3PFXResource automatically places it in a cache, so it can be referenced by a number of different nodes. You do not need to manage this loading and caching, as it occurs automatically when you attempt to access an effect using methods such as the applyEffectNamed:inPFXResourceFile: method above.

A specialized semantic delegate, CC3PFXGLProgramSemantics (a subclass of CC3ShaderProgramSemanticsByVarName) provides the mapping between the textual semantics contained in the PFX file and the enumerated semantics of the CC3Semantic enumeration.

cocos3d provides a default PFX textual semantic mapping in the CC3PVRShamanGLProgramSemantics class, which is applied by default when you load PFX files, or access effects using methods such as applyEffectNamed:inPFXResourceFile: in CC3MeshNode. This default mapping is the same mapping used internally by the PVRShaman tool from the PowerVR toolkit. The PVRShaman tool can be used to view POD files and develop and test shaders and PFX files. By using this PVRShaman semantic mapping within cocos3d, POD models and shaders can be developed and tested using PVRShaman, and then run within cocos3d without changes.

A list of PVRShaman semantic definitions can be found in Appendix A of the PVRShaman User Manual, which is included in the PVRShaman distribution as part of the PowerVR SDK download.

If you wish to define your own semantics in PFX files, instead of using the default PVRShaman semantics, you can subclass CC3PFXGLProgramSemantics to provide this mapping.

For flexibility, you can also mix the standard uniform variable names, as described above, into the GLSL shader code loaded from a PFX file, side-by-side with the uniform variables that are mapped to semantics within the PFX file itself. However, if you are trying to develop shaders that can run both in cocos3d and PVRShaman, be aware that those standard variables will not be usable within PVRShaman (unless you map them within the PFX file).

The CC3DemoMashUpScene addPostProcessing method in the CC3DemoMashUp demo app, included in the cocos3d distribution, provides examples of loading a PFX file (PostProc.pfx), via the applyEffectNamed:inPFXResourceFile: method described above.