How to Get Started with OpenGL ES 2.0

OpenGL ES 2.0 is coming to DTV. Like Java before it, learning OpenGL ES will become a necessary part of programming in the DTV world. However, its not easy, even for old OpenGL programmers like me. As already noted in a previous article, it is very different in philosophy from even OpenGL ES 1.x. Secondly, its not easy to write code for an embedded environment without having that device. Fortunately AMD have solved that problem. Here then are the steps to setting up and beginning to learn OpenGL ES 2.x.

Prerequisites

1. A windows based PC, with an AMD/ATi graphics card. Nvidia or Intel platforms may work but are untested.
2. Knowing how to program in C or C++
3. Experience with projects and IDEs
   

Installation

1. Install the latest catalyst drivers or equivalent for your card
   
2. Install Visual Studio Express C++ from Microsoft
3. Install the AMD OpenGL ES 2.0 Emulator If you are not already an AMD registered developer, you will need to do so but its free and painless.

First OpenGL ES 2.0 Program
The emulator from AMD comes with a single sample that draws a textured triangle (and carefully avoids rotating it ;-). Simply go to the install directory (which can be done from your start menu) and click on the project file. Build the project and run. You should see something like this (click to enlarge):


Going Further

• Online OpenGL-ES 2.0 man pages
  
• OpenGL-ES 2.0.4 full specification
• OpenGL Shader Language 1.0.17 specification
• OpenGL-ES 2.0 Programming Guide Book and samples from the book

In a later article I'll cover getting started with OpenGL-ES 1.x. Whilst phones and mobile devices are surging ahead into ES 2.x, its is possible that DTV will remain in the 1.x world for a while. The future is a little unclear and unfortunately though the languages are similar, they are conceptually different in approach. The vast majority of OGL-ES devices in the world will run EGL 2.x of course but I have this suspision that due to the status of of OpenGL implementations in DTV and the fact that some companies are beginning to use whats out there already, we will see OpenGL 1.x quite popular in DTV at least for a couple of years.

WebGL brings 3D to Browsers (soon)

If its not clear yet, then I'll state it clearly. Whatever 3D graphics technology we are using in DTV in ten years time, it will be OpenGL. FLASH 10 will require OpenGL and render to it. The base of everything will be OpenGL, in one form or another.

It should be no surprise that browsers too are jumping on the OpenGL bandwagon then. The Khronos Group , the guardians of OpenGL and other related standards, has announced WebGL.

Wikipedia has this to say:

WebGL is a standard specification to display 3D graphics on web browsers. It enables hardware-accelerated rich 3D graphics in web pages without the need for special browser plug-ins, on any platform supporting OpenGL or OpenGL ES. Technically it is a binding for JavaScript to native OpenGL ES 2.0 implementation, to be built in to browsers.

So, essentially OpenGL ES 2.0 bindings in Javascript. Not yet relevant in the DTV world as we don't have OpenGL ES 2.0 hardware in most set tops or TVs yet but its coming. Browsers are part and parcel of DTV experience now and are supported by GEM based languages too. Webkit based browsers already offer a version of WebGL and the full specification will be available in 2010. In the mean time, here is a demo video:

A White Paper on Broadcoms Graphics Architecture

How I missed this before is beyond me, but here is a link to a white paper on Broadcoms 3D graphics architecture supporting OpenGL ES.

There are a few interesting things in this paper. Firstly texturing:

[The px3d] Reads 40 million unrotated texels per second. Rotations around the Y-axis and, to a larger extent. the Z-axis cause texture cache misses, with larger rotations causing more misses.

40million texels is very very few (a texel is a texture pixel). And thats a peak figure. The rotations make things worse, the worst case being rotations around Z or spinning an image. Actually, four texels are usually required to render a single pixel on the screen as hardware does a weighted average four texels to get the best colour.

The practical upshot of this is, as the paper says:

The 3D device cannot render at full screen at 60 fps when the display is
1920x1080, but can only do it at about 15 fps. Instead, the 3D device could render
to a quarter the display size (960x540), and have the 2D device scale it vertically to
960x1080 with filtering.

And thats without "excessive" use of transparency, rotations or even scaling. In other words, if we don't actually use much 3D.

The px3d quoted in the paper is used in the 7400 device but has been improved for later Broadcom chipsets. Nontheless, the barriers still exist, as this is essentially the same architecture, and even in newer BCM devices, in my opinion, expert design is required to get 60fps out of the device in full screen 3D. For the time being , as the paper says, 3D rendering should be limited to small areas of the screen. The image below illustrates that:

The blitter is much faster at filling, even greater than ten times faster when scaling is used! However with a more modest goal of 20-30fps, no transparency, only small amounts of rotation and rendering at 1/4 full HD and using the blitter to scale, full screen 3D can be achieved. I'm a purist from the graphics world, not DTV, and maintain that full screen 3D at 20fps is a painful, to be avoided experience. It never ceases to amaze me the happiness some DTV folks show when their 3d user interfaces run at 10fps and cause me a headache in minutes.

FLASH 10.1..everywhere, really?

Incase you missed the big announcement...FLASH 10.1 is the future of everything. Only, no mention of set top boxes. Then, a few days later, Nvidia announce they have stopped making chips for Intel chip sets. Opinion seems split on FLASH 10 for DTV. Some believe it will be on the market in 2011, others (including one DTV chip manufacturer) think 2012.

What a mess.

OpenGL ES 2.0 - What has been cut out

OpenGL is changing with each release. For OpenGL programmers its a busy time to keep up. OpenGL ES 1.1 and OpenGL ES 2.0 are virtually different languages. The approaches programmers must take to getting 3D on the screen change dramatically. The best known change is the addition of shaders (programmable bits of the graphics pipeline) but whats left out is almost as dramatic:

• Immediate mode (glBegin/End and glVertex)
• Per primitive attributes like glColor, glNormal
  
• Explicit image drawing (glDrawPixels, glCopyPixels etc.)
• Antialiased points and lines
• The matrix stack and commands for co-ordinate transforms (eg glRotate)
  
• Non power of 2 textures with mip-mapping
  
• 3D textures
• Many data commands like glNormalPointer, glColorPointer, client state etc.
• Material and light mode
  
• Fog
• Texture environment (no fixed function blending)

In summary OpenGl coders must now:

• Provide their own maths libraries
• Use arrays for declaring 3d data
• Write shaders and pass in their own data for transforms and lighting and materials
• Handle images via textures

The language is streamlined, more in line with modern graphics thinking and better for the hardware to accelerate. However, its also no longer an easy start language (which was one advantage over DX) and lots more burden is on the programmer.

Microsoft Set Top Box

I'm not convinced by the arguments yet but this article, makes an interesting read nontheless. Could it be that Microsoft plans a gaming set top box? The key patent contains the following claims:

1. A system for integrating media on a gaming console, comprising:a first subsystem for displaying a dashboard having a plurality of media selections, wherein said dashboard is native to a gaming console and wherein at least one of said media selections is a television selection; and a second subsystem for providing users with the option of switching back and forth between said television selection and other media selections in said plurality of media selections.

2. The system according to claim 1, wherein said television selection is branded with a logo of a service provider providing content to said gaming console.

3. The system according to claim 1, further comprising a third subsystem for providing users with the option of selecting starting of said gaming console as a set-top box.

4. The system according to claim 1, further comprising a third subsystem for providing users with the option of selecting remote starting of said gaming console as a set-top using a gaming controller.

The patent goes on to clima DVR features. So are we are looking at a set top with DVR based on the xbox 360 console?

Notice the magic wand trademark they have too. I previously blogged about such a device which when used is very compelling and fun.

Graphics in MHP Tutorial

Incase you are interested in MHP or GEM graphics there is a great tutorial on the web already. I couldn't write better so I'll just provide a link:

Introduction to MHP Graphics

My only caveat is part on colour is a little out of date now. Most receivers support way more than 256 colours these days.

I should add that the article comes from a book called Interactive TV Standards: A guide to MHP, OCAP and JavaTV.

Tutorial: Double buffering (and more)

Buffers are areas of memory specifically reserved for rendering graphics. The number of buffers affects what can be done by the graphics subsystem well and also has a memory cost. In general the number of buffers, the better.

Single Buffering

One buffer is used for drawing and also used for display simultaneously.
There is usually a problem with flickering. This happens if rendering of graphics is partially complete when frame fly back (the process of copying the buffer to the output screen) occurs. We see half of one frame and half another and, over time, this looks like a flicker in a small area or a band moving down the screen if the whole screen is being redrawn.

This is really only useful on software based, SD projects, with little or no animation and so very few pixels are being rendered each frame. The outcome is very little evident flicker. If the drawing process can wait for frame fly back, render everything in the next 50th of a second and then wait again for frame fly back, no flicker will be seen.

Graphics libraries go to a lot of trouble to identify the minimal area of the screen that needed to be redrawn. Remember that moving one icon across some image requires both the icon and the image to be redrawn in many cases.

Double Buffering

Double buffering or better is used in all PC games, and in simulations and anywhere that moving most of the visual scene is required. One buffer is being used as a display (front buffer), whilst the other is drawn to (back buffer). This means there is no flicker. Simply put, the buffer being used for display is complete and is never drawn to. When drawing is finished, the buffers can be swapped (a pointer index, not a copy usually). However, if this is done at any time, tearing can occur still (as half of one buffer and then half of another is copied to the output). Usually therefore, we lock the swapping of buffers to vsync. However, the problem of blocking occurs. Imagine that we must redisplay the graphics every 1/50th second, or every 20ms. Now if we take 21 ms to actually render our scene, we miss the 20ms critical barrier but as we do not wish to risk tearing, we wait 19ms before swapping again! This means our rendering rate drops to 25 fps from 50 fps as we always take 40ms to render (and then block) each frame.

Triple Buffering

To avoid this drop in frame rate, a third buffer is used. Here,
One buffer is being being displayed, one is ready to display, one being drawn to. The two buffers not being displayed can be swapped at any time with no effect. This allows our graphics process to render as fast as it can. In modern graphics cards on PCs double buffering often really implements triple buffering without the user or coder knowing in the background by default, though controller dialogues that come with the graphics card may also expose this, but they not.

N-buffering
Of little ineterst in DTV but just for completeness, triple buffering can be extended to cushion buffering, a logical extension of triple buffering that uses N buffers rather than just 3:
One buffer being displayed, N-2 buffers ready for display, one buffer being drawn to.
The idea is that some frames of animation are harder to draw than others and by amortising across many buffers the frames that would slow the 3D down do not. The problem with this technique is lag and it has not become popular.

On Memory
So the preferred technique is triple buffering. However, the cost is memory. Example: HD 32-bit buffer (RGBA, 8 bits each) is 1920x1080x4 = 7.9Meg. So triple buffering would be 7.9 x 3 = 24 Meg just for graphics buffers. In a unified memory architecture (where the graphics chip uses main memory for rendering) thats 24 Meg of DRAM gone, just for the graphics architecture to operate.

This can be cut down using pre-multiplied alpha format (ie 24 bits per pixel) or even a limited colour range of 565 (green usually gets more bits as the human eye is more sensitive to green) and by reducing to double buffering:

So a double buffered, limited colour, with pre-multiplied alpha could be: 1920x1080x2x2 = 8 Meg roughly. Remember its just for the graphics architecture to render, nothing to do with user defined graphics yet.

Thats without a z-buffer of course but thats for another tutorial, suffice to say in this context, if a full z-buffer is used it may require 1920x1080x3 bytes (24 bit z-buffer). Only one is required regardless of how many drawing buffers are used so an additional 6 Meg. However many of the architectures in DTV devices are smarter than this and use only small amounts of memory for z-buffering.

Conclusion
Triple buffering is the best for performance but double buffering is a good performance/memory balance. Single buffering is only for low end, software rendering on SD devices. Unfortunately, these buffers use large amounts of memory and will increase the BOM of set top boxes accordingly.

Fonts for TV

If you are a designer, you take fonts very seriously. Its hard for most techies to understand this and get their heads around it. Don't believe me? Look here, here, or here to see passion about fonts. Or, more entertainingly :



I suspect as we go HD and graphics become more important to our lives, fonts will be too. Ten years of staring at Tiresias may be enough for anyone. Here it is, in all its glory:


Bitstream, who offer Tiresias screen font, also offer other TV fonts.

Another company offering TV fonts is Ascender. One font of particular note is their Mayberry(TM) font which is a one for one replacement for Tiresias. An image below illustrates how close the fonts are visually:



The top font is Teresias, the one below is Mayberry. The most obvious difference is how open the font is (compare for example the letter "S"). Open fonts will render much better at lower resolutions, something important for TV. Ascender also offer fonts for all sorts of TV functions and regions such as these teletext fonts.

This all said, a big part of the design of TV fonts has been geared to low resolution displays historically. In HDTV, I suspect a wider range of fonts are acceptable visually. So long as its not Comic Sans Serif of course ...

Tutorial: Antialiasing for DTV

The point of this blog was always to help bring people who are experts in DTV up to speed in the emerging graphics technologies. This mini tutorial looks at antialiasing and aliasing.

There are many types of aliasing but the most easily understood is that seen when drawing lines. The screen is composed of pixels. A perfect line cannot be drawn, instead one is approximated by filling in pixels. Close up it might look something like this:
This is aliasing. Its the result of sampling something which is continuous (the line equation) into digital discreet samples (the pixels). This can be unnoticeable to extremely annoying, especially if the line moves. When the line moves slowly, crawl occurs as first one set of pixels and then another is highlighted. The line is meant to be moving slowly and smoothly but the pixels suddenly switch. Over many frames of animation this makes the line look like its changing shape and crawling across the screen.

Antialiasing can be seen as an attempt to smooth out the digital sampling and have less harsh edges. Here is an antialiased version:

Pixels around the line are measured for how close they are to the line and the colour chosen depending on distance. It looks like the line is just blurred but it isnt. Blurring would not use distance from the line equation. The antialiasing make all the difference visually.

Fortunately 3D chips have antialiasing built in. This is all very well but TVs have digital filters that filter the output of any set top box of their own rendering pipeline. These filters blur or sharpen the image AFTER the graphics output. In experiments I've done, the filters within a single TV can make a huge difference and the filters between different TVs can make any attempt to compensate useless.

On LCD television this situation is even more interesting as each pixel is composed of three different coloured bulbs in a grid. Something like this for a single pixel row:
Now its possible to use each coloured bulb and measure its distance from the mathematically perfect line. This is "sub-pixel" antialiasing. It means we get more antialiasing and less blurring in effect. The result on black text on white background, looks like this:

Its worth taking a moment to see the left of the letter is more red and the right more blue. Looking at the pixel grid and distances you might think it would be the other way round but this is drawing black on white so things further away from the line or letter are brighter and so red is brighter on the left. There is a great article on this at java.net from where I blatantly stole the images.

So in antialiasing we might deal with: pixels and digital sampling, LCD bulb colours and layout. There is no easy way to deal with TV filters. The variation is simply too high. Yet there are other kinds of aliasing we can hope to deal with directly.

Temporal aliasing occurs when we sample a continuous animation. Imagine an image moving from one side of the screen to the other. It takes one second. At 50 fps the image will digitally be sampled at 50 locations as it moves from one side of the screen to the other. At 5 fps only five locations. This is a form of aliasing and explains why high frames per second are critical for rendering smooth graphics.

We can do better. Antialiasing of motion is called motion blur. It attempts to add graphics in the direction of motion. Here is a photograph of a pool table. Because the shutter of the camera remains open for a short time, the balls move across the image and leave a motion trail:
A single image again looks blurred but when seen as an animation, it all makes much more visual sense. This is computationally very expensive to render in a user interface, however some approximations can be done such as provided in this Flash tool. Disney artists long ago used techniques to approximate motion blur (reminder: temporal antialiasing) and actually deformed objects as they animate:



Notice the shape deforming on the ball, particularly just before it hits the ground. This is a crude, but highly effective, version of motion blur. The same technique ca n be applied to images as they move across the screen. They can be stretched slightly during fast motion to suggest motion blur.

Image sampling can also suffer from aliasing. There is considerable hardware built into modern 3D chips to avoid this. Mip-mapping and anisotropic texture filtering are used to avoid aliasing in images when scaling them. A blitter also uses a many TAP (texture accesses per pixel) filter to draw nicely scaled images without aliasing. The idea in all cases is simle. One pixel on the screen does not correspond to a single pixel in the source image when an image is scaled or rotated. The colour of many pixels in the source image is needed in order to draw one pixel on the screen. The source image pixels are then averaged depending on distance (compare with the line).

So, antialiasing is critical technique for producing compelling graphics. It has spatial and temporal forms. Some spatial antialiasing is done for us by the 3d hardware and blitter but then often ruined by the filters on TVs. Temporal antialiasing is usually achieved by rendering more frames per second. However, its possible to consider deforming objects in the direction of motion as cartoon animators have done for a century now.