April 10, 2012 / Daniel Freeman

Drawing MadComponents with Stage3D

A few users have asked if it might be beneficial to render the MadComponents UI using Stage3D, or to incorporate a 2D framework such as Starling. Rather than using the display list.

Everything is possible – “It’s only software” – so maybe the question should be – would the performance gains be worth the development effort?

Provided that you use GPU render mode, the performance of MadComponents is adequate. But if for any reason we were unable to use GPU mode, this would certainly tip the scales in favour of Stage3D rendered components.

Stage3D graphics, and direct render mode, is impressive. Yet nothing I’ve seen is impressive in any way that would benefit MadComponents or enterprise mobile app development.

Frameworks such as Starling are excellent for games. Games are Adobe’s main focus now. They’re great for pushing sprites around the screen, and particle effects. But could frameworks like this be applied to the lists, buttons, and forms that make up a mobile enterprise app UI?

Not really. Frameworks such as Starling are good at bitmaps. But a MadComponents UI is “elastic”. The components adapt their size depending on the size of the screen, and its orientation. Even if you could use nine patch images, MadComponents allows a developer the flexibility to customise the colours and shapes of the components – but bitmaps are fixed.

So none of the current frameworks that have been built on top of Stage3D would be any use to us. If we were to exploit the power of Stage3D, for MadComponents, then we would have to get deeply into AGAL and shaders, and build our own dedicated graphics rendering layer.

MadComponents builds up the appearance of the UI components using vector graphics. Shapes, and gradients combined in such a way to render the appearance of buttons, switches, sliders, list rows, etc.

Let’s start with a simple shape. Consider a circle. How might we develop a shader to render a circle?

We’re going to build a quad out of two triangles, and write a shader to render the circle inside the quad.


protected static const INDICES:Vector.<uint> = Vector.<uint> ([ 0, 1, 2,	0, 2, 3 ]);

protected static const VERTICES:Vector.<Number> =

   Vector.<Number> ([
//	X,  Y,  Z,		r, g, b,		u, v,
	-1.0,	-1.0,	0.0,	1, 1, 0,		0, 0,
	 1.0,	-1.0,	0.0,	1, 0, 1,		1, 0,
	1.0,	1.0,	0.0,	0, 1, 1,		1, 1,
	-1.0,	1.0,	0.0,	1, 1, 1,		0, 1
]);

This is a pretty standard approach. Notice that I’ve assigned a colour to each corner of the quad, and the U,V values can be used to tell the fragment shader exactly where it is inside the quad. The interpolated values of UV are like the coordinates of the particular pixel that we’re processing.

[ If you’re not clued-up on Stage3D, you’re going to need to find some tutorials to understand the rest of this. Alternatively, you can just look at the pictures – and see the results I’ve been able to achieve so far with my experiments. I had to write this in a hurry, as I’m catching a flight to Australia later.]

To draw a circle, we would measure the distance between the centre of the quad (0.5, 0.5) and if this distance is less than or equal to the radius of the circle, then we turn the pixel “on”, otherwise we turn it “off”.

Here is the vertex shader:

“m44 op, va0, vc0\n” + // 4×4 matrix multiply

“mov v0, va1\n” + // uv

“mov v1, va2\n” // rgb

And here is the fragment shader:-

“sub ft0.xy, v0.xy, fc4.xx\n” + // vector from centre

“mul ft0.xy, ft0.xy, ft0.xy\n” + // square

“add ft0.x, ft0.x, ft0.y\n” + // add x and y

“slt ft0.y, ft0.x, fc4.y\n” + // < radius

“mul oc, v1, ft0.yyyy\n”

fc4 is set up like this:

_context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 4, Vector.<Number>([ 0.5, 0.25, 0.0, 0.0 ])); // radius, radius squared

And here’s the result:

That’s a circle, but the edges are aliased. Let’s modify the fragment shader to correct for this. This time, we’re going to set the alpha/transparency value of each fragment, and we’re going to employ a little feathering to anti-alias the shape.

Here’s a new fragment shader:

“mov ft1, v1\n” +

“sub ft0.xy, v0.xy, fc4.xx\n” + // vector from centre

“mul ft0.xy, ft0.xy, ft0.xy\n” + // square

“add ft0.x, ft0.x, ft0.y\n” + // add x and y

“sub ft0.y, fc4.y, ft0.x\n” + // (1 + radius squared) – sum

“pow ft1.w, ft0.y, fc4.z\n” + // feathering to anti-alias circle

“mov oc, ft1\n” // output colour

Let’s set up our constants:

_context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 4, Vector.<Number>([ 0.5, 1.245, 200, 1.0 ])); // va4 centre, 1 + radius squared – frig, feathering, 1.0

Don’t forget to allow transparency:-

_context3D.setBlendFactors(Context3DBlendFactor.SOURCE_ALPHA, Context3DBlendFactor.ONE_MINUS_SOURCE_ALPHA);

_context3D.setDepthTest(false, Context3DCompareMode.LESS);

And now our result looks like this:-

Much better.

Now consider a MadComponents UI. What kind of shapes would be need to write shaders for? Well, fortunately, we can consider most components as consisting of rounded-rectangles. As circles, and rectangles are special cases of rounded rectangles. Actually, there are a lot of semi rounded rectangles. That is only the top corners, or the bottom corners are rounded. Look at MadComponents grouped-lists, for example, and you’ll see semi rounded rectangles.

So how do we write a shader to render a rounded rectangle. Let’s start with a fragment shader to draw the corner curves. We can accomplish this by modifying the circle shader.

“mov ft3, v1\n” +

“sub ft0.xy, v0.xy, fc4.xx\n” + // vector from centre

“sub ft0.z, fc4.x, v2.x\n” + // 0.5 – curve radius

“abs ft1.xy, ft0.xy\n” + // absolute

“sub ft1.xy, ft1.xy, ft0.zz\n” + // relative to the centre of a corner circle

“mul ft2.xy, ft1.xy, ft1.xy\n” + // square

“add ft2.x, ft2.x, ft2.y\n” + // add x and y

“sqt ft2.x, ft2.x\n” + // square root

“add ft2.z, fc4.w, v2.x\n” + // 1 + radius

“sub ft2.z, ft2.z, ft2.x\n” + //

“pow ft3.w, ft2.z, fc4.z\n” + // feathering to anti-alias circle

“sat ft3.w, ft3.w\n” + // 0 <= alpha <=1 (may not be necessary)

“mov oc, ft3\n” // output colour

Now, we can incorporate a test to detect if we’re in a corner or not. And if we’re not in a corner, we can ensure that the fragment alpha is >=1.

Ok, let’s cut to the chase now. The above renderers only work for “square” quads. We want to modify the curvature of the corners to compensate for other aspect ratios. Also, we’d like to introduce a parameter with only curves at the top, or only curves at the bottom. Or curves in all four corners.

Here’s the complete program:-

package
{

	import com.adobe.utils.AGALMiniAssembler;
	import flash.display.Sprite;
	import flash.display.Stage3D;
	import flash.display.StageAlign;
	import flash.display.StageScaleMode;
	import flash.display3D.Context3D;
	import flash.display3D.Context3DProgramType;
	import flash.display3D.Context3DVertexBufferFormat;
	import flash.display3D.IndexBuffer3D;
	import flash.display3D.Program3D;
	import flash.display3D.VertexBuffer3D;
	import flash.display3D.Context3DCompareMode;
	import flash.display3D.Context3DBlendFactor;
	import flash.events.Event;
	import flash.geom.Matrix3D;

	public class Main6 extends Sprite {

		protected static const INDICES:Vector.<uint> = Vector.<uint> ([ 0, 1, 2,	0, 2, 3 ]);

		protected static const VERTICES:Vector.<Number> =

			Vector.<Number> ([
			//	X,		Y,		Z,		r,	g,	b,		u,	v,		cx, cy, flag
				-1.0,	-0.5,	0.0,	1,	1,	0,		0,	0,		0.2, 0.4, 1.0,
				1.0,	-0.5,	0.0,	1,	0,	1,		1,	0,		0.2, 0.4, 1.0,
				1.0,	0.5,	0.0,	0,	1,	1,		1,	1,		0.2, 0.4, 1.0,
				-1.0,	0.5,	0.0,	1,	1,	1,		0,  1,		0.2, 0.4, 1.0
			]);

		protected static const N:int = 11;

		protected var _context3D:Context3D;						// 3d graphics window on the stage
		protected var _shaderProgram:Program3D;					// compiled shaders
		protected var _vertexBuffer:VertexBuffer3D;				// vertexes used by our mesh
		protected var _indexBuffer:IndexBuffer3D;				// indexes of each vertex of the mesh
		protected var _modelMatrix:Matrix3D = new Matrix3D();	// transformation matrix

		public function Main6(screen:Sprite = null) {
			if (screen)
				screen.addChild(this);

			stage.scaleMode = StageScaleMode.NO_SCALE;
			stage.align = StageAlign.TOP_LEFT;

			stage.stage3Ds[0].addEventListener(Event.CONTEXT3D_CREATE, onContext3DCreate);
			stage.stage3Ds[0].requestContext3D();
		}

		protected function onContext3DCreate(event:Event):void {
			_context3D = Stage3D(event.currentTarget).context3D;

			if (!_context3D)
				return;

			_context3D.enableErrorChecking = true;
			_context3D.configureBackBuffer(stage.stageWidth, stage.stageHeight, 0, false);

			_context3D.setBlendFactors(Context3DBlendFactor.SOURCE_ALPHA, Context3DBlendFactor.ONE_MINUS_SOURCE_ALPHA);
			_context3D.setDepthTest(false, Context3DCompareMode.LESS);

			_vertexBuffer = _context3D.createVertexBuffer( VERTICES.length/N, N);
			_vertexBuffer.uploadFromVector(VERTICES, 0, VERTICES.length/N);
			_context3D.setVertexBufferAt( 0, _vertexBuffer,  0, Context3DVertexBufferFormat.FLOAT_3 );
			_context3D.setVertexBufferAt( 1, _vertexBuffer,  6, Context3DVertexBufferFormat.FLOAT_2 );
			_context3D.setVertexBufferAt( 2, _vertexBuffer,  3, Context3DVertexBufferFormat.FLOAT_3 );
			_context3D.setVertexBufferAt( 3, _vertexBuffer,  8, Context3DVertexBufferFormat.FLOAT_3 );

			_context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 4, Vector.<Number>([ 0.5, 0.0, 200, 1.0 ]));	// va4  centre, radius squared, 0.0, 0.0

			_indexBuffer = _context3D.createIndexBuffer( INDICES.length );
			_indexBuffer.uploadFromVector(INDICES, 0, INDICES.length );

			_modelMatrix.identity();
			_modelMatrix.appendScale(0.5, 0.5*stage.stageWidth/stage.stageHeight, 0.5);
			_context3D.setProgramConstantsFromMatrix(Context3DProgramType.VERTEX, 0, _modelMatrix);

			var vertexShaderAssembler:AGALMiniAssembler = new AGALMiniAssembler();
			vertexShaderAssembler.assemble( Context3DProgramType.VERTEX,
				"m44 op, va0, vc0\n" +	// 4x4 matrix multiply
				"mov v0, va1\n" +		// uv
				"mov v1, va2\n"	+		// rgb
				"mov v2, va3\n"			// curve
			);

			var fragmentShaderAssembler:AGALMiniAssembler = new AGALMiniAssembler();
			fragmentShaderAssembler.assemble( Context3DProgramType.FRAGMENT,
				"mov ft3, v1\n" +
				"sub ft0.xy, v0.xy, fc4.xx\n" +		// vector from centre
				"sub ft4.xy, fc4.xx, v2.xy\n" +		// 0.5 - curve radius

				"abs ft1.xy, ft0.xy\n" +			// absolute
				"sub ft1.xy, ft1.xy, ft4.xy\n" +	// relative to the centre of a corner circle

 				"div ft4.z, v2.x, v2.y\n" +			// aspect ratio of the curved corners
				"mul ft1.y, ft1.y, ft4.z\n" +		// modify circles by aspect ratio

				"mul ft2.xy, ft1.xy, ft1.xy\n" +	// square
				"add ft2.x, ft2.x, ft2.y\n" +		// add x and y
				"sqt ft2.x, ft2.x\n" +				// square root to give distance from centre of circle

				"add ft2.z, fc4.w, v2.x\n" +		// 1 + radius
				"sub ft2.z, ft2.z, ft2.x\n" +		//
				"pow ft3.w, ft2.z, fc4.z\n" +		// feathering to anti-alias circle

				"slt ft4.xy, ft1.xy, fc4.yy\n" +	// Are we in a corner?
				"add ft4.z, ft4.x, ft4.y\n" +		//
			 	"add ft3.w, ft3.w, ft4.z\n" +		// If not, alpha>1.0

				"mul ft0.z, ft0.y, v2.z\n" +		// Option to completely fill top or bottom section
				"slt ft0.w, ft0.z, fc4.y\n" +		//
				"add ft3.w, ft3.w, ft0.w\n" +		// If fill, alpha>1.0

				"sat ft3.w, ft3.w\n" +				// 0 <= alpha <=1 (may not be necessary)
				"mov oc, ft3\n"						// output colour
			);

			_shaderProgram = _context3D.createProgram();
			_shaderProgram.upload( vertexShaderAssembler.agalcode, fragmentShaderAssembler.agalcode);
			_context3D.setProgram ( _shaderProgram );

			_context3D.clear(0,0,0);
			_context3D.drawTriangles( _indexBuffer);
			_context3D.present();
		}
	}
}

And here’s the result:-

Here, we’ve set a flag parameter to 1.0 to specify curves at the top. We can set the flag to -1.0 for curves at the bottom, or 0.0 for curves on all four corners.

Note that this shader is also capable of drawing rectangles, squares, circles, and ellipses, which are all special cases of rounded rectangles.

As an example, I’ve applied my shader to build a composition of shapes to make up a button. Rendered using Stage3D:-

If buttons, are possible, so are all the other components. This is all still in the experimental phase. But it certainly WOULD be possible to do the graphics UI rendering of MadComponents, using Stage 3D graphics. I’d anticipate that performance could benefit as a result. But it will might take quite a bit of development effort to move from these experiments, to a new UI renderer engine within MadComponents.

Any volunteers?