True 3d in User Interfaces

The set top box is unifying with mobile devices in its CPU and GPU (graphics processing unit) capability. For example, Orly from STM will have an ARM9 core and Mali 400 GPU, the same combination found on many tablets and phones today.

So, predicting the future of interfaces for STBs is as easy as looking at some of these devices in action. Today we are used to the "point and swipe" interfaces of tablets but next generation interfaces for mobile devices (and in some cases for TV) are moving to true 3D with 3d objects, not just flat images, rotated about an axis.

True 3D user interfaces have been a hit and miss affair over the years. Infact the only successes I remember are the floating cube found on the Gamecube and a little known file system browser from SGI, called fsn, which was genuinely useful and had a moment of fame in the film Jurassic Park ("I know this , its a Unix system") and was later adapted to fsv under Mac OSX. Otherwise there have been countless failures.

fsv on MAC OSX

However, a number of YouTube videos show that the trend is changing so, I've gathered a few here for viewing. At each stage more the 3D is bettrer crafted than the last.



The Inflexion UI uses a carousel of true 3D and one of pseudo 3D. The effect is one of adding a bit of 3D to a more traditional interface.



The Optimus user interface makes very nice transitions from 2D into 3D objects, fully animated. Its sugar on what is essentially otherwise a normal carousel style user interface but its this sugar that makes the interface sweet - if you pardon the pun.



The FUSE UI is 3D most of the time. All items appear to be true 3D objects. Some freaky use of shaders for distorting the screen and lighting adds a wow factor and things spin off the screen after deslection or rotate in glorious 3D revealing they are solid objects. Yet the 3D objects are still in a 2D space, we dont fly anywhere or feel we are in a 3D world.



And strangely, buried in this tech demo for Mali 400, is our first example of a true 3D interface. Skip Quake and you will see not only are there 3D objects used in menu sections but the transition between menus flies us through a 3D space. We get a feel of being in a 3D world. The ball of light we used as our cursor gives a feeling of continuity in the environment. True 3D.

OpenGL is Easy

Insofar as GUIs are concerned, you're going to have to trust me on this: OpenGL is easy. Sure, building an SSAO, displacement mapped, deferred rendering shader with bloom, glow, volumetric fog inifite texturing and HDR is not easy. But thats for games.

Making some boxes and text move smoothly on screen is, at least as far as graphics technology is concerned, easy. Yet, so many people faced with OpenGL for the first time panic. I constantly get questions about sourcing or building graphics libraries that remove the need for programmers to learn OpenGL.

Experience shows that 2-3 weeks is sufficient for any competent programmer to learn enough OpenGL to build competent GUIs on STBs or TVs. Alternatively we can spend 3 months developing a graphics lib or 2-3 weeks learning to use a third party library. In the latter case, you may think productivity is higher - after all , its a higher level language, but the day your boss enters the room and asks for s feature your graphic library does not support is the day you throw it all away and switch to OpenGL.

With OpenGL ES 2.0 you need the following:

1. A way to open and close a surface or window, usually not a matter for OpenGL in any case.
2. A way to animate things smoothly using a real time clock and some animation library. Real time clocks are platform dependent and provided by the OS. Smooth animation ? Start by normalising the input value then use SIN(value) or as a colleague recently used sin(sin(sin(value))).
3. A maths library. Well the basic maths for a UI is addition for translation, multiplication for scaling, and rotating around x,y and z axes maths can be found anywhere on the net but devolves to cos and sin in each case.
4. Ability to read textures/images. OK I'll admit, here a library is easier but normally your environment comes with this capability and if not try looking up stb.c and stb.h for a very easy to use, self contained piece of source
   

You do not need a specialist graphics library.
You do not need to be an expert in graphics.
You do not have to be a mathematician.

Any competent programmer will be writing UIs in OpenGL in less than 3 weeks. You'll have to trust me on this.

Alternatively, you can always just hire me :-).

FLASH in the pan

Those who read thise blog and have met me professionally know that for about 3 years now I have been advising against FLASH as a solution for DTV graphics. There were two major reasons, performance and business related.

Now the discussion is moot.

Adobe have announced they are dropping support for FLASH on mobile devices. More than this, Adobe, I'm absolutely sure, will now drop support for STBs and TVs.

So now a third reason ,as a technology direction, its simply not strategic anymore. FLASH is, and as far as I am concerned, always was a dead end for graphics on STBs and DTVs. Only now I am not the only one saying it - Adobe is too.

However, one company that continues to impress me and is certainly a solution for those who must have FLASH for the next few years is Animated Media Inc.

AMI is run by Chris Brady who has been in embedded graphics as long as there were embedded graphics. He has some very impressive demos of his FLASH technology, including iphone FLASH.

The best Interface is No Interface?

Minimalism has come back in style in Europe and rumours of Steve Jobs and a new form of voice interface just won't go away. The natural extent of minimalism, the singularity in the graphics universe, is of course, no graphics at all. Keen to find out what this would mean, I asked a few colleagues at Alticast to create a voice controlled interface and gave them just a few working days.

The results, using Siri Proxy, Siri from Apple, an iPhone 4S and some custom software on a commercial set top box can be seen below.



So what did we learn?

• Voice is very useful for some functions, pointless for others. Search in particular could be very useful in speech.
  
• Graphics must re-inforce each command on the screen AND a voice must respond simultaneously, otherwise the viewer feels lost.
• Remote voice recognition (like Siri) has too much lag to be entirely comfortable and speech should be recognised locally.
• "next episode" means different things to different people (next in schedule, next in series etc.)
• Voice complements but does not replace on screen graphics.

Thanks to Maciek and Rafal who did all the hard work on this.

OpenGL ES 1.x and 2.X

Intel platforms, the new BCM chipsets and STM chipsets all now support OpenGL ES 2.x. It seems 2011 is the year 3D graphics become a reality in the TV industry, at least for those of us working in the industry. The consumer will most likely see effects of this en masse in 2012.

Now, as detailed in my previous article, OGL-ES 2.X and 1.X are practically different languages, requiring very different approaches to coding. Infact I would say OGL-ES 2.X onwards require graphics programmers. However OpenGL-ES 1.X requires no such expertise and is infact "obvious".

The good news is that OGL-ES 2.X chipsets are backward compatible. This means OGL-ES 1.X is easy to use on these new chipsets.

In a recent trial in my office a programmer who had never programmed graphics before, started with OpenGL 1.X and within 2 weeks had a full 3D user interface running on a set top box with smoothly scaled and moving video, smooth animations. Better yet, he had done it to a 3rd party spec (and so had a precise design to master). 30 fps HD resolution, scaling and moving video, 3D user interface.

The conclusion? Any competent programmer can have modern user interfaces running in a short time on set tops.

Youview Graphics Specification

Youview, the renamed Project Canvas is not a bad specification. Its obvious it has been created with considerable real world experience of key areas that need to be defined for a broad purpose hybrid platform. The specification can be downloaded at the resources site. Covering areas of hardware and software I was naturally curious what it has to say about graphics.

To summarise:

• Blitter graphics required (filled boxes, scaled images, alpha blending)
• RGBA8 and RGBBA4 formats
• DirectFB acceleration
• MHEG mandated
  

For a spec that must target as many entry level platforms as possible, it seems to make sense. Keep the platform cheap and single chip. Reduce memory requirements. Still no requirement for vector acceleration or 3d. The most the spec is brave enough to say is that OpenGL (or OpenVG) could be used.

Yet, for a spec that is scheduled to launch in 2012, its a little worrying. Iphone 5 (q1?), ipad 2, Samsung Galaxy (2?) are all going to be available and competing for business, not to mention other smart phones, tablets and laptops. There seems to be no model for device capability discovery, effectively preventing an apps store model exploiting higher spec hardware, for encouraging games on STBs for example. The spec several times mentions FLASH applications, though its obvious FLASH cannot run well on this platform.

We have seen time and again in the DTV world in Europe that if something is not mandated, it does not happen. If Youview succeeds in its ambitions to become a European wide platform for hybrid content delivery, graphics on DTV will fall two to three generations behind competing devices in Europe.

Augmented Reality TV Graphics

With the advent of smart phones and pad type devices, augmented reality is becoming, ahem, a reality. Augmented reality is the idea that a screen is held infront of a something you are looking at in the real world and the screen projects an aimge of that thing and adds some (3d) graphics to it.

In the TV world, Kia have jumped on this and produced adverts.



Infact simpler approaches can be used and a logo on screen can cause a companion screen , when pointed at the logo, to run certain advertising, whcih does not have to match exactly what is on screen. A similar idea is seen here for Superbowl.



Now the ideas start to come thick and fast.
It would be possible to "zoom reality" by simulcasting broadcast and IP signals and having a device zoom the IP stream while the TV plays the live broadcast. If placed correctly in augmented reality, this would seem to zoom the TV image but maintain quality. All sorts of things arise from this...

* A ball in a football match could be highlighted on the companion device held infront of the viewer.
* Character names could be added above the characters
* Face tracking and subtitles could be used to place speech bubbles next to charcaters
* Adverts could suddenly spring from objects on the screen (clothing, jewelry etc)
* You could even carry your favourite TV clips on your t-shirt...



The possibilities are endless...