Minority Report: Gestural Interface for TV sets from Canesta

In January of this year Canesta partnered with Hitachi to produce the worlds first gesturally controlled TV set.

At GraphicsforTV on Youtube, there are interesting videos about Canestas user interfaces, including a live video of the Hitachi set being demonstrated.

Plenty of technical information is available and Canesta look set with a pool of nearly 40 patents granted to dominate this area of the business.

I'm unconvinced by some of the demos and increasing the volume is a laborious function in the video. However, the tech is so cool and so waiting to happen, it just can't be ignored and there is an intersting case study for TV available.

Graphics for TV Adds YouTube Channel

Announcing the new Graphics for TV YouTube channel at http://www.youtube.com/GraphicsforTV

Keep an eye out for lots of videos related to user interfaces, games and applications on TV.

The first set of videos is a presentation given at JavaOne 2009, entitled: Move Your Users which applies the classic 12 principles of animation from Disney to user interfaces. Mostly useful for designers, but programmers should also be aware of these basic ideas.

Enjoy!

Console Games on Set Top Box

Playcast becomes the worlds first to provide console quality games on set top boxes in a live service. The idea is simple. The game is run and rendered on a remote service, the resulting images being sent back to the set top box. The hard part is tackling the real world issues of lag and latency. There are strict definitions but in real world terms, latency can be thought of as the constant delay a user experiences from pressing a key to seeing a result, and lag as the effect of the game freezing at intervals.

The technology works for cable and IPTV but not for terrestrial or satellite. It plays on existing set top boxes by integrating a software client and will be available in Europe. A beta trial is underway in Israel now, and expected to roll out in Q1 2010.

More details.

Playcast isn't the only company trying to deliver console games remotely. OnLive too deliver this service. Their web pages show much more detail including specific gaming titles (eg Tomb Raider: Underworld and Bioshock). In the case of OnLive, a small console is required to connect to the TV though a Mac or PC may be used too, and the service will only be available in the USA.



It seems for now the two companies have different approaches and territories.

Video Game News - Digital TV games continue to grow overseas. Can they make it big in the States?

Video Game News - Digital TV games continue to grow overseas. Can they make it big in the States?

Shared via AddThis


Denki are a games company specialising in limited resource platforms; one being the set top box. Based out of Dundee and founded by former DMA design employees (remember Lemmings?), Denki is a games company that seems to be taking TV seriously.

What is a Scene Graph?

Advanced Graphics is new to the DTV arena. Occasionally, this blog will explain some traditional graphics terminology that may be new to even experienced engineers in DTV.

Scene Graph

Traditional computer graphics is rather like impertaive programming. In order to draw things we do a tiny bit of setup for the window and so-called device context and then we issue commands such as set colour, draw a line in 2D, draw a triangle in 3D. A piece of OpenGL-ES code, an example of this traditional style, will have a few functions for setup and the rest of the code will be drawing commands. The important point is that each time we draw a new frame of the animation, we loop around all the drawing code and issue all the commands again.

Scene graphs turn this on its head. Scene graph based languages (eg Java3D) usually have a tiny draw loop. It may infact be reduced to one line of code. Instead all the code is used to setup the scene graph which is a description of all objects, colours, animations and even event triggers in the scene. Code is often replaced by some XML or scene graph description language such as with GRIN.

The graph itself contains nodes which represent each type of drawing function - objects, animations, lights, appearance, events etc. The nodes are arranged in a graph. Drawing starts form the top most node and typically proceeds left to right, top to bottom. Thus nodes higher up in the graph, typically affect more of the scene. For example, overall lighting would appear high in a scene graph. The graphs are usually directed acyclic graphs.

Below is an example of a scene graph taken from SGI publications:



Scene graphs are often a higher level construct and deal with higher level entities (objects). Thus its often much easier to get graphics up and running using scene graph languages. They are thus very attractive in the DTV world where typically users are less experienced. Conversely, they may not be as flexible or performant as dropping to OpenGL and thus may limit advanced users.

Scene graphs are both for 2D vector based languages and 3D triangle based. The initial definition of a scene graph on wikipedia is thus misleading. However the rest of the article is fine.

Job: UI designer California, set top boxes

Job this way...

Rovi: The Everything Guide

Macrovision Solutions, the holder of key patents in the field of user interface design, has a new name: Rovi. Rovi offers a range of products but one of interest for this blog is their next generation user interface called the Everything Guide.



There is vast amount of marketing information on their website about the guide. Once again its clear that 3D has been avoided and 2D, with proper design of details, is seen as the way forward.

On 20th July Zacks Investment Research listed Rovi amongst its top four stock investment opportunities.

Paper: Microsofts Dynaline Confirms the Obvious

Microsoft Research published a paper this year called: DynaLine: A Non-Disruptive TV User Interface for Passive Browsing of Internet Video (pdf is here). . The image above shows "coming up" events, or internet videos, related to the content of the current viewed video, rather than a simple whats on next view.

Unsurprisingly in tests, viewers strongly preferred the list to update automatically and so maintain as passive a TV viewing experience as possible. Viewers also strongly preferred a list of related content rather than a simple ordered list (now-next).

No real surprise but its good to have these things confirmed.

Brief Survey of GEM Authoring Tools will have you GRINing

A great presentation is available from the IEEE which not only explains the relationship of various flavours of Java to each other for TVs and set top boxes but also reviews the authoring tools available and compares the styles. No conclusions to be drawn here, but Bill Foote ,the author, is heavily involved in grin which is a scene graph language that sits on an animation framework, ultimately relying on GEM graphics libraries and has an associated authoring tool mainly aimed at Blu-ray but which is claimed ot be compatible with OCAP, Tru2way and MHP. The syntax is very similar to JavaFX and certainly this merits a much closer look in a future article...in the mean time here is a presentation on the GRIN technology family from JavaOne 2009.

Your TV is watching you

Face recognition on a set top box would be in my top 5 must haves. How many times has someone said - who is that? Haven't I seen them in another movie? A facial recognition program with a link to imdb would be great. Better yet for cable operators, how about facial recognition with a link to your VoD offering? The work could be offloaded to a server from the set top box due to the compute involved.

There is a darker side to facial recognition on set top boxes. Imagine a system where a camera is attached to your STB and the STB wathces you - 24 hours. It switches on the TV when you sit down and switches off when you leave. The menu is personalised to you or whoever comes into the room and of course, the operator gets to know a lot more about your watching habits than you might like. All of this is a lot closer than you might think.

This IEEE paper, from 2007, proposes such a system. And Comcast may well be considering it for real.

Top 3 Set Top Box User Interfaces of all Time

We are reaching a point of huge change in set top boxes. The user interfaces of the next 5 years will bear almost no resemblance to the more traditional user interfaces found on TVs and set tops. Partly this is because hardware to render graphics quickly is coming in. Mostly though it is because design and useability are being taken seriously. Now there are companies specialising in just the user interfaces on set top boxes - its becoming a serious business.

So its an interesting time to look at the best of the best so far. What do you thinnk are the three best user interfaces to hit the television in the last ten years? Here is my list and my reasons:

In third place we have...Hillcrest Labs Home Interface

Home comes in at number 3. A 2.5D user interface (scaled surfaces are placed facing the viewer and never rotated) which enables a unique moving through menus feel. We do not move from one menu to another but literally through them, flying into the choice we just made and back out when we chose. Hillcrest combine this with a pointer device, not a traditional RCU, to give a true step up in user interfaces. Home isn't just a user interface, its a configurable platform for CE manufacturers to use in their own way, including reference games, music player etc.



Home has that elusive cool factor, it shows a really great 3D effect but does so by never leaving 2D and scaling. User interfaces that go the 3D route are usually doomed. Hillcrest Labs clearly understand that presenting information in 3D is simply wrong and instead limit the 3D to transition effects that truly involve the user in the interface.

A great modern interface that runs on todays devices and addresses the needs of CE manufacturers. The interface only makes it into third place because the pointer device requirement means it is not a cheap, standard, solution for use with traditional RCUs. Go to another article about Hillcrest Labs here.

The Runner up is...Apple TV

Apple produced a set top box that whilst not their most successful product, sent a shockwave into the industry with its user interface. By exploiting 3D hardware in subtle ways, and using minimalist design principles, the user interface was almost completely intuitive and super cool in that unique Apple way.

Apple again undertood the role of 3D in user interfaces. They place all primary information flat, in one area of the screen. Re-inforcing, or secondary information (such as icons, images etc.) are allowed in 3D in another part of the screen. The 3d is minimised to a supporting role. Where Apple led of course , others have followed since but it was Apple who first showed the way.



Minimalism too isn't an easy design. The key is to use very small effects to great purpose. The user barely notices changes in colour, light, animation entry and exit speed, shadow and reflection but all are present and all add tot he feeling. I once had the priviledge to sit with a designer and listen to what he saw in the interface and I was astonished at the detail Apple have gone to. Piror to Apple TV user interfaces were busy and included classic screen entries like "Press exit key to Exit". Apple ended all that.

The RCU was minimalist too, up, down, left, right, OK. The interface did not require the annoying coloured buttons and no help was necessary on the screen.

Yet for me the interface felt dumbed down. It was almost too slick, too easy. There were no corners to explore or surprise menus. The remote felt like a childs toy and all too plastic. Like the guy who prefers his Jaguar E-type to the XK, just because, well, its an E-type, Apple does not get my top spot.

And The Winner Is... TiVo Series 1

What? That blue, 2D mess? Yes.

TiVo was released in 1999 - archaeology in tech terms. Yet the interface was streets ahead of others. Using alpha blending, sliding and - horror - curved edges to boxes, it just looked so much better than anything else. It moved smoother than anything else. The animations has fade in and fade out on the motion. The highlight moved smoothly from one choice to another, it didn't jump. The graphics were large and bold and even included a cute TV character logo, long before Clippy destroyed our love of such things.

Plugging in the device for the first time brought up a cute animation of the TiVo TV character swinging across the screen and around plastic constructions, bringing the TiVo platform to life, full of character. The out of box experience was fun and exciting. It had character!

The user interface itself was a traditional blue colour, something much copied since. It worked very well and at the time seemed fresh and vibrant. The remote was unique, easy to hold and well organised. It stood out, against the black remotes lieing around the living room, in glorious silver. A brave decision.

TiVo chose to use sound better than any other user interface since. A ping of congratulations greeted correct actions whilst a dong of remonstration warned of incorrect action. This simple addition made it possible to complete actions without looking at the screen! A user interface where you didn't focus on the screen? I'm still amazed today. Just by pressing left, you moved back up menus to the main menu, presing quickly until dong, dong, dong, meant you could go no further.

Using left to exit a menu screen was a true touch of interface design at its best. It meant the user could repeatedly press left to exit menus, with no fear of going too far and back to the TV image. Too many user interfaces use the same button to return from a menu screen and to return to the main broadcast. Not being forced to find back with your thumb for moving around menus was a joy.

The DVR functionality allowed for new interface conceptions such as time shift buffer and thumb keys.

So it was bright, vibrant, used sound, presented a unique user interface, was cool and had character! Elusive qualities indeed. I read recently someone criticising TiVo for not improving the user interface much in the last ten years. In response I would say, the user interface and product were ahead of their time. Only in the last few years have they begun to look old fashioned. TiVo was for me, the best user interface of its time. Of all time.

So far...

In the future we will have FLASH, Java and GEM, 3D and other technologies to play with. The heat is on, lets hope that user interface designers don't forget the nice touches done in the past in the race to be new and fresh.

(postscript: I have not yet found an image of TiVo series 1 user interface on the net! If you find one let me know in the comments.)

Game Style and Set Top Boxes

I noticed a post over at Suns Java forums the other day. Essentially it asks about fundamental features of graphics on set top boxes and what that translates to in terms of games. Nobody ever answered.

The Past
About five years ago, common hardware for set top boxes, particularly in Europe, included the ST Microelectronics 55XX series of chips. These chips had no blitter hardware. Thus there were an awful lot of set top boxes with no blitter. Companies producing graphics libraries for these set top boxes did everything in software. Image copies, boxes, lines, circles...all were done by smart graphics libraries running on the main CPU. The trick in achieveing animation lay in one simple task: update the minimum amount of pixels as possible. The easiest way to achieve that was to move only small things on the screen. Thus were born typical set top box games that move small coloured sprites in low numbers around the screen. It was enough for Pacman , invaders, tetris, card games and other 2D, sprite based games where the screen never moves and only a minimum of pixels must be updated each refresh. The background could be placed in an iframe to speed things up. The CPUs were 100-200Mhz.




The Present
Today the vast majority of set top boxes have a blitter - a hardware device for doing DMA copies of memory with some small combinatorial functions (for blending). The blitter can copy memory from one or more sources, combine the data and write to one destination. It can also scale the output. Speeds of copying are high at scale factors of 1.0 and slower at other scale factors. It is thus easy and fast to copy whole screens now at high refresh rates. Companies with graphics libraries for set top boxes compiled their libraries with the blitter drivers enabled and allowed images and filled boxes to be drawn quickly.

Blitters have enabled 2.5D games such as Space Harrier or Burn Out.



However these games are relatively niche in their scaled 2.5D sprite based approach. Another style of game enabled is the scroller game. Now with a blitter it is possible to scroll a background image or even blit many subimages forming a background (such as required in MIDP) composite. Perhaps the classic example of a scrolling game would be Sonic the Hedgehog.



The blitter enables much bigger and bolder graphics and even HD resolution games. As yet largely unexploited.

With the addition of mass storage to set top boxes, be that in hard disk or DVD form, a further category of game became feasible. The laserdisc cartoon game is much maligned but Dragons Lair was an outright sensation when it was released. Here the player simply choses from a limited set of options as the cartoon plays. The game jumps from cartoon sequence to sequence or a death sequence if timing or choices are bad. There are a surprising number of these types of games available and some are even released in HD form. Let me know if you have seen them on set top boxes.



All this said, many games companies for set top boxes have not yet caught up and the vast majority of games available today fall into the small sprite category still.

Adding an internet connection to set top boxes enabled community and multiplayer games. So far these have been fairly limited to such things as small groups (eg 4 players) playing a game against one another and sending pre-programmed messages from a menu. Th world of true multi-player gaming has not yet reached the TV screen.

Even today Broadcom produce set top boxes with 3D hardware. Its unclear what the hardware could do with SD 3D games - could Tomb Raider I style games be achieved in SD or lower resolution? Perhaps. Texturing is still relatively slow and current 3D architectures have restrictions on the numbers of polygons in an area of the screen before dropping in performance. This makes free roaming cameras (as in Tomb Raider) diffcult to achieve. However with careful planning and design, even today on a BCM 7413, 3D games are possible at low resolutions. However, at HD, the hardware isn't yet powerful enough for 3D games for sure.


Yet even if the games can be rendered, the RCU is a crucial problem. The delay on key presses can be 0.3 seconds - fatal for most reaction games. Those complex moves in Tomb Raider would seem to be too much of a strecth for todays RCus. No, to enter into 3D worlds, to have true twitch games, hardware will have to change.


The Future
Graphics hardware is coming that will make 3D games possible, even at HD resolution. Just as interestingly lines and polygons will be added to the set top box in 2D. This means Vector graphics and FLASH based games. Sort of.

FLASH will be able to play on set top boxes well enough that some popular FLASH games like Swords and Sandals will be possible. Though, whether SnS will play unaltered is not clear yet. The point is though, similar games will play.



However equally, or even more, popular games will not. Bloons 3 Tower Defence, and in fact all Tower defence games, simply use too much CPU and most likely too many threads to be viable. Many FLASH games are simply too heavyweight on the CPU to work at all well on the set top box even once the graphics work. That is aside from the memory issues of the bigger games such as Evony.


Casual gaming would seem to be the future for set top boxes. Destined to be behind the PC in terms of raw gaming power and aiming at an audience who are more passive than those infront of a PC.

Multiplayer servers should be very possible with languages like MHP available on the set top box. Community game servers, where league tables, challenges are issued, teams form etc. will all be accessible from the set top box, not just the PC.

New hardware for controlling set top boxes will come. It is currently expensive but Hillcrest are showing the way with pointer devices as optional purchases for set tops on Amazon. Will casual gamers pay? It remains to be seen.

Last Thoughts

Set top boxes and the cable industry are facing a fight now. The internet is upon them. Embracing the internet will be the key to success. Many TVs and set top boxes are beginning to embrace widgets from the internet: small applications that can be downloaded and used directly on the TV or STB. I foresee a time when the games are widgets and app stores for the TV and STB are open and flourishing, selling games worldwide on set top boxes. In such a market it is viable for games developers to create bespoke software, games specifically aimed at set top boxes.

MHP Games and Applications: sveng.com

Sveng.com Production Ltd was established in 1998. The founders, Jan Olof Svarvar and Anders Engström had worked professionally for ten years in the TV -and film industry and decided to create a unique TV, interactive TV and post-production house.



They make, amongst other things mhp games and applications.

Get started with OpenGL ES 2.0 Programming

You can do worse than have a look at Imagination Technologies introduction to OpenGL ES. With a link to Khronos SDK for OGL-ES, example programs and shaders and a bunch of proprietary software linked to the PowerVR SDK platform. Mainly useful for games programmers for set top boxes in 2010 onwards as opposed to UI developers.

Investors, get your cheque books out, stb gaming is going to be big...

Masters Thesis: Good Introduction to Graphics on STBs

Yongjun Zhang Masters Thesis (pdf file) makes a good read as an introduction to graphics on set top boxes. Recommend for anyone who is just starting into this field. Although written in 2003 and thus qualifies as archaeology in Blog terms, it is an easy read and shows how little has changed in the last 6 years.

The thesis correctly identifies that STB chipset manfacturers will supply OpenGL ES in the future and that programmers on set top boxes will require some higher level API, not just OpenGL. It does unfortunately miss (understandably at that time) the emergence of OpenVG, the 2D vector oriented language.

Patent: Launching UI Components from service

A very interesting patent has been granted in the US. Essentially it divides functionality of a TV user interface into logical components (EPG, VoD, Photo viewer etc.) and assigns each component to a channel. The user, by tuning to the channel, launches the relevant part of the GUI. This is especially interesting for middlewares like Tru2way and MHP which allow the applications to be downloaded and stored on the set top box and so would be launched immediately on tuning.

I'm a little dubious if the patent will hold up in court. It seems to me the DVB must have discussed using the stored application parts of the GEM specification in this manner during the creation of the spec. That would be prior art.

Intel Set Top Box Demo

Here is a really nice video of an Intel Powered set top box. Intel has 3D tech and multiple video decoders. However, note the CE3100 is stated as only decoding two HD channels at the same time (so this demo must be SD). Nonetheless its impressive.

Intel also announced that they have ported FLASH to the CE3100.

It all sounds too great to be true. Rumour is that Intel have trouble getting the price of the device down to the levels where it is viable in typical set tops. I also wonder about power dissipation and standby power rules for Europe...

EGG from Osmosys: Enhanced Graphics enGine

EGG is a 3D software library from Osmosys that enables exceptionally fast psuedo-3D effects from C or Java. It eases the route to 3D interfaces by staying as close as possible to the OpenGL standard whilst enabling set top boxes with limited or no 3D hardware support to match the performance of those with full 3D hardware available in 2009. EGG is used in many of Osmosys products and videos of its capabilties are available at their website.

I can't say too much about EGG. Despite being the inventor, I am under NDA and can only repeat publically available information from Osmosys. Programs written in Java/MHP/Tru2way can use the EGG library to create accelerated 2D and easy 3D effects. Effectively any OpenGL programmer can instantly use EGG as the API is very close. EGG is targetted at platforms pre-3D hardware (ie blitter based platforms) and was named one of the top 3 must see technologies at IBC 2007.

EGG is a low level graphics programming language but Osmosys also produce a higher level companion API which has a suite of off the shelf effects - EGG-FX (also my invention).

Its not clear what strategy Osmosys will adopt with EGG once true 3D hardware arrives at commodity prices but certainly EGG could layer on top of 3D hardware for future proofing.

Social TV News: ClipSync Launches Social TV Application for Facebook

Social TV News: ClipSync Launches Social TV Application for Facebook

Posted using ShareThis

Interesting news about Facebook being used to form a social group events and allow pop-ups your friends to appear on your television set.

Alticast Launches AwakenTV GUI for MHP and tru2way

Alticast Launches AwakenTV GUI for MHP and tru2way

Posted using ShareThis

Its frustrating to know that Java in set top boxes is capable of so much more than people have done so far in terms of graphics performance. I'm really pleased therefore that AwakenTV is finally showing the real power of GEM graphics. Smooth, fast, animated HD visuals that really impress in standard GEM. Multiple videos, images, transition effects etc. all running smooth at HD. AwakenTV is a huge visual step in GEM applications.

Underlying it are two technologies. FastMHP is a version of the MHP stack which is completely optimised. The JVM itself is optimised per processor! The graphics exploit whatever graphics hardware is available - AwakenTV runs very fast on set top boxes without OpenGL hardware. This FastMHP enables the incredible performance of AwakenTV.

The second technology is altiExcite(TM) which is Alticast back office and client solution for displaying many moving videos at once. In the example from which the image above is taken, the video previews are playing real time. It also enhances channel zap time to unbelieveable speeds.

POWERVR SGX520, the world’s smallest OpenGL ES 2.0 core achieves Khronos conformance

POWERVR SGX520, the world’s smallest OpenGL ES 2.0 core achieves Khronos conformance

Posted using ShareThis

The news is a little old but as this blog is new, its worth noting. PowerVR is a direct competitor to ARM Mali for licensing in set top boxes. Its evident that OpenGL ES 2.0 is coming to set top boxes which means 3D and shaders. Shaders are a sort of programmable hardware targetted at graphics and are the direction all desktop 3D is taking both in OpenGL and DirectX. They allow the application writer (from C or Java) to submit mini programs to be run directly on the graphics hardware for maximum performance. This could be anything from a video filter to a reflective water effect.

Shaders mean that future set top box coder teams must include at least one true graphics expert. Thats a few years yet though.

Hillcrest User Interface for Set Top Boxes

In March 2009, the New York Times reviewed the ZillionTV and said "It’s the nicest interface on a set-top box I’ve seen yet." The user interface is the HōME™ user interface from Hillcrest Labs. This is a uique, patented, 3D navigation system which can be viewed in full on the website by filling in a form. The user experience is one of moving into and through levels of menus in a highly intuitive way. The whole 3D experience is rendered as 2D flat images that scale away and zoom toward the viewer and so runs on just about any set top box.

The user interface design would not work with the tradition remote control unit. Instead Hillcrest license a patented pointer device design kit called Freespace. This is a design kit for building an in air pointer device, including reference designs, software and hardware. Using the Freespace tech, Hillcrest have developed a ring shaped device with a thumb wheel, two buttons and a pointing device (accelerometer perhaps?) called the Loop Pointer.

Together, this hardware/software solution produces stunning results and is one of the few demos that truly uses 3D in an experience enhancing way. I have never used it, and my previous experience of pointing devices on set top boxes has been that they lose calibration quickly. Hopefully Hillcrest have overcome this. The Loop Pointer is available for $99.

One very interesting side effect of a pointer device is text input. Traditionally very hard on RCUs, a pointer device allows fast input by free navigation of a keyboard image. Hillcrest show this briefly in their video. This makes services like Twitter and so forth much easier.

Twitter and TV: The New Backchannel

Twitter and TV: The New Backchannel

Shared via AddThis

Twitter makes a lot of sense on TV. Comments on favourite shows or sporting events can enhance the program for a lot of viewers. The link shows an interesting graphic with what looks like live data for the players.

One point the article makes is that viewer twitters may not be feasible due to the ned to enter text. An interesting challenge...

In the meantime, Vizio also announced Twitter, as well as other social networking for its TVs.

Arm Mali: 3D core for set top boxes and more

In January of this year, ARM announced that ST Microelectronics had licensed the ARM Mali-400 MP. Later in June, ARM announced compliance of the Mali-400 with OpenGL ES 2.0. Infact it is conformant with OpenGL ES 2.0 and OpenVG 1.0.

This video shows a Mali-200 (275Mhz), in action:



More videos are available at the ARM website.

The 400 should cope with HD resolution graphics with 4 cores. However, manufacturers like ST Microelectronics would be free to use less cores as the Mail-400 is configurable. Nonetheless, its an exciting time. With OpenVG and OpenGL ES, Mali offers the potential 3D and vector graphics on HD set top boxes in 2010.

AmanithVG - Vector Graphics for embedded Systems

Mazatech is a brand new company, based in Italy, founded by a team of young professionals who have been working together developing technologies for realtime vector graphics, since 1998.

Mazatech aims to produce an OpenVG compliant library that uses OpenGL ES as a base. The product is called AmanithVG and is available for evaluation on win32 as a free download from their website. Other platforms can be evaluated under NDA.

OpenVG is to 2D and vectors what OpenGL is to 3D and triangles. That is, OpenVG is an open, royalty free, standard developed by Khronos, aimed at addressing vector rendering and 2D drawing.

OpenVG is interesting because it is an ideal base on which to map vector rendering libraries such as FLASH and SVG. An open question for me is if AmanithVG has relevance to the set top box world where undoubtedly chip manufacturers will supply similar technologies.

3D from Cubiware

MPX-3D (trips off the tongue that one) is a product from Polish company Cubiware.




The video demonstrates the software which requires 3d hardware (BCM7400 and BCM7413) in the set top box to run. What impresses me is not so much the 3D but the system behind it. XML is used to describe the 3D scenes and predefined templates can be used to ease development. Cubiware clearly understand that 3D is possible but highly dangerous on todays generation of set top box 3D devices. That is, 3d works, but barely and not full screen. Using templates allows them to control and limit the flexibility of the system to something that will work.

Small code for JPG and PNG decode

Part of the art of embedded systems is having small code to fit into flash and DRAM. For image decoding, most devices, including set top boxes, must support JPEG and PNG images.

Libjpg and libpng seem obvious choices. They are tried and tested, in some sense, official libraries for jpg and png. Yet they have flaws. Firstly they are not easy to use. Even experienced programmers think twice when they first encounter the API. Secondly, they are simply heavyweight. Both include every possible variation of the image format, some of which are very rare in practice and secondly they also include encoding routines.

Of course it would be possible to delve into the code and begin removing unwanted sections. However this would result in losing the tried and tested added advantage.

Instead why not consider this using the code from this website. stb_image.c, (stb, suprisingly nothing to do with set top box but instead the name of the author) can be found on this website. Its light, solid, does what is needed and is about 1/20th of the size of the other libraries.

Traditional Graphics Hardware in Digital Receivers

What Blitter Does (and does not) Do
In set top boxes without OpenGL hardware, the usual graphics hardware is a blitter. Although each blitter is unique they all share some basic functions:

• Fill a rectangular area of the screen with a fixed colour
• Copy a block of memory (image) to another block of memory (screen)
  
• Scale copies

Its worth saying what blitters cannot do (in the general case): draw lines, draw triangles, 3D, draw text, draw curves, draw circles, fill areas with a gradient of colour, decode images, decode video, draw surfaces at angles.

Boxes and images (with alpha). Thats all we have.

How Blitters Perform

There are three critical measures of blitter performance:

• Setup time - the time to issue a command to the blitter to draw something which involves setting blitter registers
• The fill rate - the number of pixels a blitter can fill in one second at some bit depth with a scale of 1.0
• Time taken to complete a blit request - how fast can the code continue after submitting a blit request
  

Fill Rate
The fill rate is obvious and for most applications not an issue. Modern blitters can fill an HD screen (1920x1080) many times a second. However, quoted figures for fill rate are usually from a local cache for the blitter, whereas in practice the source for blitting is the DRAM of the STB. This means the REAL fill rate will be dependent on how much else the set top box is doing such as video decoding. Nontheless filling an HD screen ten times over is possible and animation speeds of 30fps can be achieved with care.

Fill rate is slowed by alpha blending, though at some predictable rate, and by scaling.
Scaling an image up slows a blitter down. As a guideline - an image zoomed by 200% will be drawn 2x slower. This depends on caches and write bandwidth but its a good guideline. Scaling an image down is not slower than drawing a normal size image but obviously results in less pixels being displayed and thus the effective fill rate is 50% of that at scale factor of 1.

Setup
Blitters in digital receivers are designed to copy large visual areas. Setup time is often less optimised. A typical user interface for set top boxes may only require a few tens of blit commands to redraw and so this usually is not an issue, except in one very important case: text. Text, even vector fonts, is usually prerendered as character bitmaps and characters drawn to the screen to form words. If each character is blitted seperately, we may have many hundreds of blits to render text. It can even occur that setup time becomes so significant with smaller text that using a software loop on the main CPU is faster than using the blitter!

It is setup that prevents us using blitters to draw arbitrary shapes. One can imagine drawing a triangle by drawing each of the scanlines across it. Fill rate exists on blitters to do this but the setup for each scanline is both computationally expensive for set top boxes and also too slow on the blitter itself.

(N.B. some blitters can take display lists of edges and produce arbitrary shapes using this but it is not common to all blitters).

Fast Return of Control
Although mainly a driver issue, not a blitter issue, the speed of blitting can be greatly slowed down on some devices because blit commands block instead of returning. Using non-blocking blits allows code to continue with execution. Many blitters have a blitter queue. This is a buffer into which blit commands can be thrown without blocking and the blitter left to complete them as fast as it can. The code can return and complete some non-graphics tasks in the mean time. Using a blitter queue greatly enhances performance.

Conclusion
Blitters then are the basic graphics hardware for driver writers wanting to optimise graphics performance on digital devices. Even with some 3D enabled devices, the blitter is identified as a fast path for 2D operations. They will remain in set top boxes and anyone wishing to know how to develop fast graphic for digital receivers must understand them and how they work.

Nagravision Taps Zappware to Develop New NagraGuide VOD Applications

Nagravision Taps Zappware to Develop New NagraGuide VOD Applications

Posted using ShareThis

JavaFx Authoring tool

A good blog about the forthcoming JavaFX tool suite can be found here. It is clear it targets Java and JavaFX against Flash and provides a Creative Suite link to the Java world. This is interesting because JavaFX plus Java performance is really very good compared to Flash and Actionscript currently.




However at JavaOne this week Bill Sheppard, Chief Digital Media Officer, Java Software Group at Sun explained that JavaFX is essentially incompatible with GEM! This seems to be because the definition of required hardware is too low for JavaFX.

Flash Comes To The Set-Top Box And TV - Connected Home News - Digital Trends

Flash Comes To The Set-Top Box And TV - Connected Home News - Digital Trends

Posted using ShareThis

Comment: Flash is a designers wet dream. It promises ease of development and the possibility to run huge amounts of existing content. Adobe is certainly serious about this. Its Creative Suite allegedly contributes about 60% of revenue but sales have topped out, meaning they need new markets. Adobe is suffering and has closed offices twice this year, for a week each time, to save costs. This is a determined and co-ordinated push into DTV to generate new revenue. Adobe therefore are deadly serious.

Yet, I'm sceptical. Despite demos and heavyweight partnerships, I have two main technical concerns. The first is graphics performance. Simply put, Flash requires graphics hardware support to perform well. Text, lines, shaded areas, polygons etc. all require hardware at HD resolution to run fast enough. Perhaps this is why we are told the first applications will be next year when graphics hardware will be available in top end systems. More crucially, the Flash virtual machine is notoriously slow when compared to C or Java. This means business logic in applications is currently very slow indeed. This is a problem Adobe must address to win in this space.

Is this a nice story or a reality? Its too early to tell.

Graphics come to Digital TV

Graphics for television, specifically for set top boxes and iDTVs is an interesting field. A few years back nobody considered graphics an important issue. TV is about Video right? Well, yes it is. That is to say, watching, previewing, buying, and in the future, sharing of video. None of which is possible without a user interface. Not to mention that your average set top box these days shows images, plays music, shares files with the PC or other devices and a host of PC like functionality. All of which needs an interface.

Prior to 2007, user interfaces for TV viewing were slow, poorly designed, 2D and limited to boxes, text and occassional images. Interfaces were at worst designed simply as an access point to underlying technology and at best as access to specific features such as an EPG. The screen would be splattered with too much info, helpful messages such as "Press OK button for OK" and a bare minimum of animation.

Then in March 2007, Apple shipped a device, Apple TV which changed the game. It showed movie posters and photos in carefully designed minimalist 3D. It worked beautifully with up, down, left, right and OK. No colour buttons here. No help appeared on screen - it was intuitive, it was smooth, it was ... shock 3D. AppleTV contained a 3D chip from Nvidia. Though the product itself was a failure, it succeeded in raising the bar in every way for graphics in TVs.

Round the same time, the first 3D interfaces and games in mobile phones began appearing. The user was becoming used to small devices doing 3D, looking good and reacting with smooth animation to button presses.

Since then, the industry has woken up. Now design and 3D technology is emerging in the TV space that will make yesterdays interfaces seem a joke in comparison. Yet there is still much ignorance surrounding graphics in the TV space. With this new need and a lack of experience, come the wolves seeking easy prey: a hundred new technologies and companies who proclaim to have the Holy Grail for Digital TV graphics. Thus we enter an interesting arena, and the battle begins.

Faced with this, the challenge of combining internet and traditional TV and the ever increasingly vast library of content that must be presented, its not easy on DTV companies. This blog aims to track the development of graphics in DTV and hopefully present useful information for those in the industry.