The One True Graphics Technology for TV

When chosing a service based solution, an operator would like the option to ignore the underlying technology and simply pick the best cost/features services. Its my firm belief that in the case of graphics and user interfaces they don't have this luxury. The right choice of technology will have a dramatic effect on the quality of user interface experience., the abilities the designers can use, the memory required, the performance, the ease of workflow and so on. Ignoring the tech, and swallowing the marketing will not work. Operators need to be savvy here, not just in their core business.

Worse, the user interface experience is expanding. Its now required to have an offering of some sort on client, mobile and PC. Worse still, OpenGL-ES is not the same level of language as FLASH or Java-FX. There are strengths and weaknesses to them all. Then add in the complexity of input devices (touch pads, absolute pointers, relative pointers, traditional) all of which can radically change the experience and we are in a confusing situation.



So, which technology is the best choice?

Let me ask you this. Did you go to IBC this year? Or CES? Did you fly? Drive? Train? Bike? Did you stay near the show and walk or far away and use public transport each day? Why? Cost contraints, time constraints, distances? OK, was it the best choice? For you? For me? For everyone? No, of course not.

The choice of UI tech is no different. Marketing and sales would have us all believe that their solution is the best, for every purpose, for every solution and service. The truth is when our current technology upheaval that we see now ends, several techs will be left standing just the way satellite, cable and terrestrial all stand side by side. Why? Because you and your customers have different needs from your competitor and their customers.

To this end, lets look at some strengths and weaknesses of various graphics techs in the next article.

3D Movies - they are here to stay

This blog normally deals with 3D graphics for set top boxes and DTVs. However, its hard to ignore another type of 3D - 3D video. Avatar was rightly passed over at the Oscars but still remains one of the biggest cinema events of all time thanks, almost entirely, to its 3D technology. Its not the first time Hollywood has been excited by 3D movies of course. The fifties were full of 3D movies.



For a year now I've kept silent and watched how 3d movies developed and its becoming clear that now, almost one hundred years after the first 3D movie (and nearly 60 after the first by a Hollywood studio), that 3D is here to stay this time.

The two major objections to 3D movies are:
1. Its a tiny proportion of all movies out there and so its a gimmick only, there is not enough source material to justify the average consumer upgrading.
2. You have to wear glasses or watch silly lenticular lensed screens. The natural, confortable viewing experience is not available.

Point 1 assumes that 3D is only available with new movies. That a massive VoD catalogue is unavailable for 3D source material. Though its true, it is no longer technically impossible. It is now possible using obscure and carefully guarded trade secrets to back fit 3D to existing 2D movies. Examples of this are quite shocking:

• The new Alice in Wonderland was filmed in 2D and the 3D effect was added in post processing!
• There exist examples of real time 2d to 3d conversion.
  
• There are companies in Hollywood who specialise in this kind of post processing such as In-Three Inc who were responsible for Alice in Wonderland.
• Major directors have announced intent to convert movies to 3D in post processing including Titanic, Matrix and rumour has it, Star Wars.

Now the conversion process with quality is not cheap. It requires some human intervention but the price will come down in time.

So thats issue 1 - a large catalogue of 3d titles taken care of. In time at least. Computer vision and 3d graphics techniques combine to give us a moment in history that enables 3d movies.

Issue 2 is glasses. In case you missed it, Engadget ran an article about i3d and their new 3d screen with no glasses. Not only does it do real time upo conversion from 2D to 3D, it also does not require 3D glasses to view the resulting 3D.

How it works, is still a mystery. When it will be available even more so. The point is though, it can be done. For the first time in cinema history.

So the two major issues have technological solutions and now its a matter of volume v cost. Hollywood so much wants all this to succeed that there is now a sense of inevitability about 3d movies.

3d is here to stay this time.

LWUIT

LWUIT is a lightweight user interface toolkit from Sun, I mean, from Oracle. It was born from the area of user interfaces for mobile phones and is developed by Oracles team in Israel and is available to run on CLDC1.1 MIDP2.0/CDC PBP/SE, currently at version 1.3.

In 2009 Sun began to push LWUIT for DTV not just for mobile devices.



There is a lot of support for LWUIT on the internet including Shai's Java and LWUIT blog.
and even a book, but unfortunately for version 1.1.

LWUIT is a toolkit with an advanced widget set for user interface development. It is pretty and offers lots of features. The toolkit has a porting layer so that LWUIT can be placed on a variety of platforms.

LWUIT includes transitions such as slide and fade and animations from point to point with acceleration and deceleration if required. The platform can embed existing AWT based applications such as games and can handle video in widgets too. It is based on SWING design principles (such as model-view-controller) and the toolkit includes a designers tool to change the graphics.

However, the toolkit has come from a reduced screen size world and the full HD resolution of televisions may be a problem. Advice on writing efficient applications includes reducing transparency and keeping images small. This is worrying and although the demo is pretty it runs very slowly and largely shows a static demo with few animations or transitions.

What isn't so clear is how Oracle now position this against GRIN and Java-FX. Java-FX was previously said not to appear on set top boxes but now Oracle have retracted that. GRIN, a scene graph based graphics API, with a modern design philosophy compared to LWUIT (scripting, scene graph) that grew up in the DTV space seems to compete with LWUIT but with a lot less information and impetus behind it currently.

Normal Service Resuming

A few kind people have asked me why the blog has been inactive for a while. Nothings is wrong. Infact my employer simply had more of a share of my life than is normal for the last quarter. However, things are returning to normal and so I'm reactivating this blog. Thanks for your patience.

How to Get Started with OpenGL ES 2.0

OpenGL ES 2.0 is coming to DTV. Like Java before it, learning OpenGL ES will become a necessary part of programming in the DTV world. However, its not easy, even for old OpenGL programmers like me. As already noted in a previous article, it is very different in philosophy from even OpenGL ES 1.x. Secondly, its not easy to write code for an embedded environment without having that device. Fortunately AMD have solved that problem. Here then are the steps to setting up and beginning to learn OpenGL ES 2.x.

Prerequisites

1. A windows based PC, with an AMD/ATi graphics card. Nvidia or Intel platforms may work but are untested.
2. Knowing how to program in C or C++
3. Experience with projects and IDEs
   

Installation

1. Install the latest catalyst drivers or equivalent for your card
   
2. Install Visual Studio Express C++ from Microsoft
3. Install the AMD OpenGL ES 2.0 Emulator If you are not already an AMD registered developer, you will need to do so but its free and painless.

First OpenGL ES 2.0 Program
The emulator from AMD comes with a single sample that draws a textured triangle (and carefully avoids rotating it ;-). Simply go to the install directory (which can be done from your start menu) and click on the project file. Build the project and run. You should see something like this (click to enlarge):


Going Further

• Online OpenGL-ES 2.0 man pages
  
• OpenGL-ES 2.0.4 full specification
• OpenGL Shader Language 1.0.17 specification
• OpenGL-ES 2.0 Programming Guide Book and samples from the book

In a later article I'll cover getting started with OpenGL-ES 1.x. Whilst phones and mobile devices are surging ahead into ES 2.x, its is possible that DTV will remain in the 1.x world for a while. The future is a little unclear and unfortunately though the languages are similar, they are conceptually different in approach. The vast majority of OGL-ES devices in the world will run EGL 2.x of course but I have this suspision that due to the status of of OpenGL implementations in DTV and the fact that some companies are beginning to use whats out there already, we will see OpenGL 1.x quite popular in DTV at least for a couple of years.

WebGL brings 3D to Browsers (soon)

If its not clear yet, then I'll state it clearly. Whatever 3D graphics technology we are using in DTV in ten years time, it will be OpenGL. FLASH 10 will require OpenGL and render to it. The base of everything will be OpenGL, in one form or another.

It should be no surprise that browsers too are jumping on the OpenGL bandwagon then. The Khronos Group , the guardians of OpenGL and other related standards, has announced WebGL.

Wikipedia has this to say:

WebGL is a standard specification to display 3D graphics on web browsers. It enables hardware-accelerated rich 3D graphics in web pages without the need for special browser plug-ins, on any platform supporting OpenGL or OpenGL ES. Technically it is a binding for JavaScript to native OpenGL ES 2.0 implementation, to be built in to browsers.

So, essentially OpenGL ES 2.0 bindings in Javascript. Not yet relevant in the DTV world as we don't have OpenGL ES 2.0 hardware in most set tops or TVs yet but its coming. Browsers are part and parcel of DTV experience now and are supported by GEM based languages too. Webkit based browsers already offer a version of WebGL and the full specification will be available in 2010. In the mean time, here is a demo video:

A White Paper on Broadcoms Graphics Architecture

How I missed this before is beyond me, but here is a link to a white paper on Broadcoms 3D graphics architecture supporting OpenGL ES.

There are a few interesting things in this paper. Firstly texturing:

[The px3d] Reads 40 million unrotated texels per second. Rotations around the Y-axis and, to a larger extent. the Z-axis cause texture cache misses, with larger rotations causing more misses.

40million texels is very very few (a texel is a texture pixel). And thats a peak figure. The rotations make things worse, the worst case being rotations around Z or spinning an image. Actually, four texels are usually required to render a single pixel on the screen as hardware does a weighted average four texels to get the best colour.

The practical upshot of this is, as the paper says:

The 3D device cannot render at full screen at 60 fps when the display is
1920x1080, but can only do it at about 15 fps. Instead, the 3D device could render
to a quarter the display size (960x540), and have the 2D device scale it vertically to
960x1080 with filtering.

And thats without "excessive" use of transparency, rotations or even scaling. In other words, if we don't actually use much 3D.

The px3d quoted in the paper is used in the 7400 device but has been improved for later Broadcom chipsets. Nontheless, the barriers still exist, as this is essentially the same architecture, and even in newer BCM devices, in my opinion, expert design is required to get 60fps out of the device in full screen 3D. For the time being , as the paper says, 3D rendering should be limited to small areas of the screen. The image below illustrates that:

The blitter is much faster at filling, even greater than ten times faster when scaling is used! However with a more modest goal of 20-30fps, no transparency, only small amounts of rotation and rendering at 1/4 full HD and using the blitter to scale, full screen 3D can be achieved. I'm a purist from the graphics world, not DTV, and maintain that full screen 3D at 20fps is a painful, to be avoided experience. It never ceases to amaze me the happiness some DTV folks show when their 3d user interfaces run at 10fps and cause me a headache in minutes.