[AG-TECH] Advice sought on AG server upgrade proposal

Christoph Willing c.willing at uq.edu.au
Wed Oct 13 21:31:18 CDT 2010


On 14/10/2010, at 3:11 AM, Stanford P. wrote:

> Hi
> I am currently trying to put together a proposal to upgrade our rather
> ancient Access Grid servers. I have looked through this list and  
> gleaned
> various bits of useful information about possible configurations. I  
> would be
> grateful for comments about any aspect below - apologies in advance  
> for the
> length of this mail.
>
> Background:  we installed a pretty traditional room-based node about  
> 6 years
> ago  - a two-box HP workstation system running AG Toolkit under Linux,
> XAP800, Sony EVI-D100 cameras using s-video into Osprey capture  
> cards, and
> using a NVidia quad-head display card.
> The aim is to update the servers, but keep the XAP and the existing  
> Sony
> cameras for standard definition video feeds. If possible we also  
> want to add
> two HD feeds for future use, from the Tandberg Precision HD cameras  
> we also
> have in the room for H.323 videoconferencing (these can output
> simultaneously on HD-SDI, as well as the HDMI connection we use for  
> the
> H.323 videoconferencing).


Phil,

We've tried that with a Tandberg C20 HD system - the camera has  
simultaneous outputs on HDMI and HD-SDI outputs. We found that  
although the HDMI output to the codec is fine at 1920x1080, we can  
only output 720p60 from the SDI output. Although the camera DIP  
switches seem to allow full 1080 via SDI, I've worked through te issue  
with the Tandberg support system who told me that 720p60 is the best  
that can be achieved at the moment. That was about 9-10 months ago and  
subsequent firmware updates have made no difference to the attainable  
resolution.

Furthermore, that camera's HD-SDI isn't stable enough for for the  
blackmagic card. It shows up as intermittent black (actually missing)  
frames - roughly 1 per second. I've coded another application I use it  
with to detect the missing frames and insert the previous frame to  
replace the black; it sort of works but still doesn't feel right. Our  
supplier has swapped cameras without any change; they also confirm  
that the camera looks fine on their own SDI monitor. It looks like the  
blackmagic card is just very picky about the timing of the incoming  
SDI signal. Someone has suggested running the camera signal through a  
re-clocking amplifier  before sending to the blackmagic card but I  
haven't tried that yet.



> As well as Access Grid, we want to use this infrastructure with other
> collaborative software such as EVO, or even to use Skype in a room- 
> based
> environment.  Although we have a strong preference to run things  
> under Linux
> (Fedora or RHEL), we will probably want to make the new server(s)  
> dual-boot
> to some flavour of Windows too so we can cope with any eventuality.
>
> We¹ve found our current hardware to be very reliable, so for  
> starters I¹ve
> looked for updated versions of what we¹ve got rather than changing
> everything in sight.
>
> I¹m still undecided about the one box / two box question.  In  
> support of the
> two-box approach, I can understand the complication of the operator  
> seeing
> multiple VIC instances on a one-box system, rather than just having  
> a single
> instance on the display machine.

Apart from the usability issue (of multiple vics on operator's  
console), I've been observing that using plain VideoServices for  
blackmagic HD streams is not sustainable - the packet loss and image  
quality deteriorate hugely when vic is being used to capture and then  
display multiple HD streams. Yet, if there is no capturing involved, a  
single vic can display multiple HD streams i.e. in practice, separate  
vics are needed for capture and display; that is only possible with  
separate producer and consumer services which, in turn, mean either  
separate physical machines or separate virtual machines, like Linux  
Containers (LXC) which I emailed about recently.


> Also, I do still wonder about the wisdom of
> expecting one box to cope with (say) capture of two HD streams and 2  
> SD
> streams, plus audio, plus quad-head graphics output and networking.  
> That
> seems an awful lot of i/o (translating to CPU load). However, for  
> use with
> EVO, both capture and display would need to be on the same machine  
> (unless
> you signed in on two boxes, I guess). In the end of course it will  
> probably
> come down to money.  Your current opinions are solicited on this  
> issue.

I don't have a machine available with enough slots/cards to do all of  
that but I've just set up a machine with LXC so that the main system  
does audio and video display (though only for a single LCD) and the  
container system has 2x blackmagic cards and an Osprey 440 with 2x  
cameras attached. This machine is well over a year old with an Intel  
quad core Q6600 2.4GHz cpu and 4G RAM. With 2x 1080i50 and 2x h261  
streams using LXC, I see each HD stream using about 110% cpu, the  
display vic uses about 60%, the 2x h261 streams use about 1% each. The  
system remains responsive and the only problem seems to be when there  
is lots of movement in the HD streams; then there is sometimes some  
slight breakup of the image and vic reports some loss (up to about  
1%). I think thats due to some of the cores running flat out - newer  
higher speed cpus may not have to work so hard.

If I try the same number of streams without LXC, i.e. using multiple  
plain VideoServices, the system becomes almost completely unresponsive  
- actually its pretty bad with just a single VideoService for the  
first HD stream.

So I think a single machine solution is viable for AG with multiple HD  
streams if using LXC or some other virtualisation system. That may  
help with your EVO need for everything in a single box. Does EVO work  
with multiple blackmagic cards?


Nevertheless, 2 separate machines would be even better - completely  
separating the capture and display resources.


> New server(s): The HP z800 looks a pretty good spec, particularly  
> when it
> comes to i/o in the form of the number and type of PCIe slots  it  
> would
> seem to give some options for future growth, even on a one-box  
> solution.
> Also we mostly use HP for our other servers, so are generally  
> comfortable
> with their support.
>
> Processors: I would prefer dual quad-core processors, but would  
> welcome
> advice as to whether this would be overkill, particularly in a two-box
> solution.

Sounds good for a single machine system though. I notice that the base  
z800 has a 2.13GHz cpu; I'd say you probably want at least the next  
speed up from my test machine (2.4GHz) i.e. 2.53GHz or higher


> Memory: I¹m assuming that memory requirements are not particularly  
> great,
> and that something like 12GB per box would be ample (memory is triple
> channel on the z800, so for dual processors needs to be installed in  
> sixes,
> hence 6 x 2GB)
>
> Disk: Similarly the disk requirements should be modest  we¹d  
> probably keep
> the Linux and Windows installs on separate drives for simplicity   
> ideally
> I¹d like to mirror each OS on a pair of drives.  There seemed to be  
> some
> references to Linux on the z800 needing SAS rather than SATA drives  
> for any
> RAID config, but whether that applies to a straight RAID1 mirror I¹m  
> not
> sure.  At a pinch we could live dangerously and just have a single  
> drive
> (146 or160GB) per OS I guess.
>
> SD video and audio capture: The initial need is to be able to  
> support the
> current 4  s-video feeds, and I was looking to use the Osprey 450e if
> possible (though there seemed to be some doubt as to whether this was
> supported under Linux or  VIC yet).   A second best might be the  
> Osprey 440
> (normal PCI).  Either should be able to do the audio capture too.
>
> HD video capture  The Blackmagic cards seem to be the popular  
> option, both
> for AG and EVO. The ŒDecklink Duo¹ takes two HD-SDI inputs, which  
> would seem
> ideal for our system., and it seems to have a Linux driver. It isn¹t  
> crucial
> for it to be supported Œright now¹ in AG Toolkit / VIC, but it would  
> be nice
> to know it¹s on the horizon.

We don't have a Duo card to test with but since it uses the normal  
Blackmagic-supplied driver, it _should_ just work. Although, remember  
the gotcha we had with the SDI output from the Tandberg C20 system's  
camera. BTW the same blackmagic SDI card worked perfectly with the  
Sony EVI-HD1 camera.


chris


> Graphics card:  One quad-head card supported in the z800 seems to be  
> the
> Nvidia Quadro NVS 450  we¹ve previously used the similar PCI card  
> with good
> results. The current PCIe model outputs on DisplayPort connectors,  
> which we
> would have to convert to VGA.   An alternative could be two NVidia  
> Quadro
> FX580 cards, as suggested here:
> http://www.arcs.org.au/index.php/tturnkey-video-collaboration/318-room-based
> -turnkey-vc-system but I am not sure what the benefits would be  
> compared
> with the single card.
>
> There¹s probably something else obvious I¹ve missed but this is long  
> enough
> already.
>
> Thanks in advance for any comments
>
> Phil Stanford
> National Oceanography Centre
> Southampton, UK
>

Christoph Willing                       +61 7 3365 8316
QCIF Access Grid Manager
University of Queensland



More information about the ag-tech mailing list