Access Grid Projects - VP

VP (Video Presence) is an application designed to replace vic on the Access Grid display, and on the desktop. It tries to simplify the management of an AG node, and make the user interface dramatically friendlier and easier to use. It is also designed to scale upwards and outwards to take advantage of newer hardware and software capabilities, and application needs. This includes making use of multiple CPUs; VP is multithreaded. While at this stage VP is a display application, we have begun work on the capture side as well.

Current features

-  Site grouping
-  Site-tile manager
-  Display manager
-  Smooth scaling
-  Multiple codecs (2)
-  Preview mode
-  Ports to Linux, Windows and the Mac

Site grouping
Every Access Grid site can transmit an arbitrary number of video streams. These sources can appear, disappear and reappear at various random times throughout an event, requiring operator interaction. VP uses rtcp data to bring together all the video sources from a single site, and keep them grouped together on a single site-tile. The site-tile contains all of the video sources, and rearranges itself as streams appear and disappear. In future it will also present other metadata such as icons, audio indications, network indications, etc.

Site-tile manager
Each site-tile can be laid out in various ways. Multiple video streams are laid out on the tile by the site-tile manager, which can be easily extended to support any kind of layout. The default layouts are 'grid', such as 2x2, or 'prioritised' where 1 video source is displayed larger, and the rest are smaller, such as 2 or 3 across the bottom of the main video picture. Each site-tile has its own manager, and can be laid out independently of the other site-tiles. When managing the display, the operator moves the site-tile as a single entity, not as a set of individual video tiles

Display manager
All of the site-tiles are automatically laid out on the full VP window by the display manager. Currently it lays them out in a grid, but the mechanism to add more complex automated display layouts is reasonably simple. It could even dynamically adjust the display as e.g. site-tiles are moved around or scaled.

The examples below show (i) a small number of sites, automatically laid out, with grid-layouts on the site tiles, and (ii) a larger number of sites, after manual adjustment of positions and scaling, with prioritised layouts on the site tiles:

Smooth scaling
VP uses OpenGL hardware assistance to actually display the video tiles. This means that the OpenGL hardware is used to offload display issues such as scaling video streams to any size, rather than the limited half/full/double approach of vic, with no additional CPU load. This allows users to layout their displays more flexibly, scaling site tiles and video tiles to any size, from zero to bigger than the whole display.

Multiple codecs
Out of the box today, VP supports h.261 and DV(*). More codecs are planned to be released as we progress, or as people contribute the code.
[* The DV code does not yet use the mmx code under Windows, so performance is slower than it needs to be.]

Preview mode
If a video tile is not displayed at full resolution, VP will use a lower-resolution preview-mode to render that video tile. This reduces CPU load on the display machine, while still providing a useful view. The preview-level will vary with the size of the displayed tile (not fully functional in current release)

Ports
VP comes in two versions. One is Java, which is more experimental for trialling new functionality. The other is in Python and C++. Both versions have been ported to Linux, Windows 2000/XP and Mac OS-X. The release version is based on the C++ code base.

Planned features
The first releases of VP are review releases for the wider community, to seek comments, suggestions and criticisms. We have a plan for future (versions to 1.0 and beyond) releases throughout 2004. The features on our roadmap currently include:
-  Visual audio cues
-  Automated dynamic audio localisation
-  Lip-synch
-  Display caching
-  Display hints
-  More metadata
-  More codecs
-  Remote console
-  Clusters for decoding
-  Clusters for displaying
-  Integration with generalised AG recording tool
-  Integration with remote camera/device control
-  Changes of listener address without restarting

Visual audio cues
VP will provide a visual highlight of site-tiles when they are transmitting audio. The highlight will provide both a 'presence' indicator as well as a 'level' indicator, depending on user preferences.

Automated dynamic audio localisation
Given the large width of most AccessGrid displays, it can be difficult to identify which site is talking. The visual highlights will assist, but is not ideal as it still requires scanning the screen. VP will provide audio-localisation, by communicating between the audio application (which identifies the source) and the VP display manager, which knows where on the display the source is, and where the room speakers are. This will yield a stereo (or better) signal, that appears to come from where the video tile is.

Lip-Synch
As the Access Grid (and similar projects) use separate applications to display video and audio, and media streams that are not transported together, lip-synch is often not correctly maintained. VP will use mbus to keep correct synchronisation.

Display caching
One of the common tasks in AG sessions is laying out the display for a given event. While VP will automate the procedure, it starts at a random point. A better approach will be to create a display-layout description that can be used if, for example, the same sites are participating. That description could for example layout sites by geographical information, over a background map.

Display hints
Given multiple cameras covering a room, it is possible to sometimes align video-feeds in such a way to generate e.g. a full panorama of a single room, or to do a very simplistic stereo projection. VP will accept an rtcp provided display-hint string and use it to guide the site-tile manager. Since there does not appear to be a standard for this, we have to develop it. At this stage only our VP-capture project plans to provide this, but it should be portable to other applications.

Site metadata
To ease the cognitive load of large-scale events, VP will accept rtcp metadata to provide a nicer identification of remote sites. This would include for example site localtime, an organisational crest, and national flag. Another benefit may be to identify camera priorities (or "importance"), for sites that may be bandwidth constrained, sitting behind an application gateway like rcbridge or that need to talk through some protocol gateways (e.g. to H323).

More codecs
Many applications want higher-quality video, but not always at higher bandwidth. Since h.261 was ratified we have also seen h.263, h.264 and mpeg4, as well as mjpeg2000. This is on top of more common, higher-bandwidth codecs such as mpeg1, mpeg2 and mjpeg. VP will happily accept plugin decoders from any source.

Remote console
Managing a large-scale display like the AG is often a long-distance mousing marathon. Furthermore, trying to manage a display from another location is often a battle with vnc and friends, XWindows, etc. VP will have a console application that can be run anywhere (on a small display) and signal the display component over the network to react accordingly.

Clusters for decoding
We are working on a cluster-transcoding engine for various projects. VP will be able to control that process to deliver appropriately decoded video to the AG display machine, allowing for much higher bandwidth/CPU-load codecs to be scalably included. To HDTV and beyond!

Clusters for displaying
Currently most AG nodes use a single display machine. VP will allow itself to be run across multiple machines, with coordination via a console application. This will allow more graphics hardware and network interfaces to be brought to bear on the display-scalability problem.

Integration with generalised AG recording tool
One of our student projects this year is developing a wide-ranging AG recording and transcoding tool. VP will be able to dynamically signal the recorder to start/stop recording streams (all or some), and any application data.

Integration with remote camera/device control

Changes of listener address without restarting
In the AccessGrid changing venues results in a kill()/exec() of the media tools. VP will shortly allow you to change venues (listening address) without a restart. This makes venue changes less of a visually disorienting problem. We're toying with ways of changing venue - having the site tiles fade to grey, crumple like papers, collapse like a deck of cards, etc. Ideas are welcomed :-)

How to install and run VP:

This software is released © ANU Internet Futures 2004. No warranty is expressed or implied. Use at your own risk.

Download:
You can download the following packages:
- Binary for Windows 2000/XP - v0.8, 24 September 2004 (886kB)
- Package for Mac OS-X - v0.8, 24 September 2004 (4MB)
- Source for VP - v0.8, 24 September 2004 (114kB)
- Source for librtp - v0.8, 24 September 2004 (228kB)
- AccessGrid video consumer service for Windows 2000/XP - v0.8, 24 September 2004 (894kB) [Thanks to Tom Uram at ANL]

Installation for AccessGrid nodes

For Windows:

1. Copy VPConsumerService.zip to c:\Program Files\AGTK-2.3\NodeServices directory
2. From the Venue Client, select Node Management from the Preferences menu
3. On your display computer, add the VPConsumerService service

For MacOSX: We're working on a VPConsumerService package - please stay tuned. There are a few integration issues with the AGTk on MacOSX. In the meantime, you can unpack the dmg file and manually point VP at the appropriate multicast address.

For Gentoo Linux:

1. Copy VPConsumerService.zip to the appropriate NodeServices folder
2. From the Venue Client, select Node Management from the Preferences menu
3. On your display computer, add the VPConsumerService service

Compiling on Gentoo

1. Download scons from www.scons.org (the current gentoo version (0.95) does not work)
and install: http://prdownloads.sourceforge.net/scons/scons-0.96.1.tar.gz
>tar -xzf scons-0.96.1.tar.gz
>cd scons-0.96.1
>sudo python setup.py install

2. emerge wxGTK and libdv if you haven't installed these yet. Make
sure opengl is in your USE variable (it is by default)

3. build and install librtp:
>cd librtp
>scons
>sudo scons install

4. build VP
>cd VP
>scons

Running VP

Please Note
VP uses OpenGL hardware-assist to offload a lot of work from the CPU. It allows for smooth scaling of video tiles to any size, and in future to have very rich, 3-D, textured user interfaces. Almost all graphics cards today of any reasonable price will have hardware OpenGL support. Where people sometimes have problems is:

• having opengl available on only some cards on a multihead system. This really depends on the drivers (presuming the hardware has it). Having an AGP and a PCI version of the same card (or at least the same family) usually solves this.
• having the primary and secondary monitors the wrong way round on the display and the console. You want the high-quality video to go to the display, not the console (right?).

VP can be started from the command line or graphically from the desktop. The user interface is very simple, partly because this is an early version and mostly because we believe the software should automatically handle as much of the grouping and layout tasks as possible.

Multicast sessions can be given on the command line as address:port pairs or from the control menu.

Select a site tile by clicking on it with the left mouse button. This also allows you to drag site tiles around within the VP window.

To change the site-tile size, select the site tile, then drag while holding the middle mouse button, or scrolling the mouse wheel. Lacking those, pressing the '-' or '+' keys also works.

You can change site-tile layouts by right-clicking on a site-tile.

In the prioritised layout, one stream within a site is given priority and displayed at a larger size with best quality decoding. The right mouse button brings up a menu with Previous and Next commands to cycle through the streams within that site. Single-button users can also select a site tile and press the 'P' or 'N' keys.

In the grid layout, all video feeds are equally sized and fully decoded.

VP has a range of keyboard shortcuts:

 n - next source is displayed first in currently selected site tile
 p - previous source is displayed first in currently selected site tile
 +(=) - increase size of the currently selected site tile
 - decrease size of currently selected site tile
 esc - press twice to exit
 0 - 9 - set the size of the currently selected site tile to the selected custom size
 ctrl + 0 - 9 - set the size of *all* site tiles to the selected custom size
 ctrl + a - show the about dialog
 ctrl + r - reset the layout ( including user manipulated site tiles )
 ctrl + f - toggle fullscreen mode
 ctrl + x - exit
 

You can configure colours, custom sizes, etc. via the menu, and save the configuration.

Almost everything in the user interface for VP can be controlled by Python scripts. You can create new site-tile layouts quite simply, drop them in the appropriate folder under VP/ and they will appear in the pop-up menu. Similary you can change the display manager, and even the site grouping to revert to a vic-like single-window-per-video-source. We're working on more documentation for this area, email us for any assistance.

Contacting us
We're looking for comments, suggestions, requests, criticisms and code :-). Email us via vp@if.anu.edu.au