Access Grid Audio Troubleshooting
An Elementary Guide for Technical Users

All rights reserved. Do not copy or redistribute in any form. Copyright © 2002-2005 Board of Trustees of the University of Illinois. Copyright © 2003 - 2005 Ohio Supercomputer Center. Copyright © 2002- 2005 University of Chicago. All Rights Reserved. Copyright © 2003 - 2005 Trustees of Boston University.

Introduction

The Access Grid (AG) is the ensemble of resources that can be used to support human interaction across the grid. It consists of multimedia display, presentation and interactions environments, interfaces to grid middleware, interfaces to visualization environments. The Access Grid will support large-scale distributed meetings, collaborative work sessions, seminars, lectures, tutorials and training. The Access Grid design point is group to group communication (thus differentiating it from desktop to desktop based tools that focus on individual communication). The Access Grid environment must enable both formal and informal group interactions.  Large-format displays integrated with intelligent or active meeting rooms are a central feature of the Access Grid nodes (AGNs). Access Grid nodes are "designed spaces" that explicitly contain the high-end audio and visual technology needed to provide a high-quality compelling user experience.

http://www.accessgrid.org/

Objectives

This tutorial identifies and discusses solutions to common audio problems people encounter on AGNs.

Target Audience

Technical individuals who operate AGNs.

Lessons

  1. Overview of AGN audio components
  2. Echo
  3. Garbled audio, periodic drop-outs, and clipping
  4. Feedback
  5. Troubleshooting a network/telco bridge
This tutorial assumes that you are using software and hardware listed in the AG hardware specification. It also assumes that you are using a ClearOne device that has been configured with the three suggested presets described in the document, How to Configure the Gentner AP400 for an Access Grid Node.

Throughout the tutorial, you will find audio files, with examples of what certain audio problems may sound like. These files are in .aiff format, and should play back easily in most web browsers.
 
For a brief overview of an Access Grid Node, see our 2.5 minute movie (~26 Mbytes).

Acknowledgements

Many thanks are due to Robert Olson of Argonne National Laboratory, for his kind and patient assistance in developing this tutorial, and Kevin Gecsi of OSC for recording the audio clips.

Finding Help

For information on WebCT, please see the "Using WebCT" link in the navigation bar to the left. This document contains navigational information, tips about style sheets and pointers for further assistance. For questions or comments about the tutorial content, please use the Discussion Space. This is available to all users and is continually monitored by the content providers. For additional information concerning the Access Grid, please see the "AG Support" link in the navigation bar to the left. Here you will find technical mailing lists and information on the AG community MOO.


 

Lesson 1: Overview

This section provides a general overview of some of the major components of an AGN audio system to give you some context for the troubleshooting advice to follow. For more technical detail on the audio set-up, refer to documents on the AGDP, including Building an Access Grid Node: From Room Construction to Equipment Configuration and Access Grid Hardware Specification.

AG audio is a complex topic because audio travels through many different systems including your local audio set-up, your local network, the wide area network, the local network at the receiving site(s), and the local audio set-up(s) at the receiving site(s). You will find the most common problems either in the local audio set-up of the transmitting site or in the networks (wide-area and/or local).

When your systems are functioning properly your audio should be of excellent quality. Please listen to this audio file for an example of normal audio. The audio on your AG node should sound at least as good as this recording (and it may sound even better).

rat

The rat (robust audio tool) software transmits and receives audio streams over the network. You can make adjustments in rat to individual sites' audio streams, such as increasing or decreasing their volume or muting them. You can also adjust the level at which you are transmitting your site's audio . However, you can only adjust the level on a per-site or all-sites basis in rat. There are other places where you can make more granular adjustments.

Echo Cancellation Device

The Echo Cancellation device (EC device) cancels echo, which would otherwise be heard on the audio channel. The device can also adjust a large number of settings, including individual microphone gains. Many echo cancellation devices also have a telephone interface that can create a network/telco bridge either as a back-up in case of network audio failure or as a way for people who do not have access to a node to join into an AG session (e.g., calling in on a cell phone from the airport).

Formerly known as Gentner, ClearOne is the most frequently used EC device for AG nodes. The AG community may also refer to echo cancellation devices generically as "Gentners".

Some EC devices, including ClearOne devices, use PC-based software for configuration. This approach is a good practice because it makes configuration management easier.

Microphones

Microphones are the primary input devices for AG audio, so they are crucial to audio troubleshooting. Make sure you are familiar with your particular microphones. Appropriate use will depend on such factors as whether they are omni-, bi-, or unidirectional as well as the correct range to pick up audio.

Monitoring your own audio

When troubleshooting audio, one good technique is to connect a pair of headphones to an output from your echo cancellation device. This test will help you determine whether the quality of your outgoing audio is being affected before or after the audio is sent to the computer that is running rat. If you discover that the audio you hear in your headphones is of poor quality, then the problem is usually related to a Gentner misconfiguration or problems with one or more of your microphones. If you are not hearing any audio, try testing your sound card by playing a music CD.

Another good option for monitoring your own audio is to run a second instance of the AG software on a laptop. This is commonly known as a Personal Interface to the Grid, or PIG.


Lesson 2: Echo

What it sounds like

In the AG context, echo refers to audio from a given site, that is reflected back by another site, which causes everyone (except the originating site) to hear the given site's audio more than once.

Please listen to this audio file for an example of what echo may sound like in some situations.

Using rat to figure out who is causing the echo

A general rule: The site causing the echo is typically unable to hear the echo. However, an echo may be faint enough that some sites will hear it and others will not. It is also possible for more than one site to be sending an echo at the same time.

To determine which site(s) are sending an echo, use rat to mute each audio stream, one at a time, until you find the one that makes the echo stop. You will need to continue transmitting audio yourself to be able to determine the source of the echo. If no one is speaking, there is no audio to echo back. That site should follow the steps below to stop transmitting echo.

Correcting the problem

If you are transmitting echo, make the following adjustments:

If another site is transmitting echo and is unable to remedy the situation, you may wish to ask them to not transmit audio except at the moments when someone at their site is speaking. This is called "push to talk" mode.



 

Lesson 3: Garbled Audio, Periodic Drop-Outs, and Clipping

What these problems sound like

When audio is garbled, you can tell that someone is speaking, but they are difficult (or impossible) to understand. This garbled effect may be constant, or it may come and go.

One particular type of garbled audio, "clipping," may also result in increased room noise when no one is speaking. Clipping occurs when a signal is overamplified. Analog and digital devices have a maximum level of signal they can handle. If a signal is driven past that level, the value of the signal is clipped at the maximum.

Please listen to this audio file for an example of garbled audio. (This particular example was caused by routing microphones to the EC reference.) Please also listen to this additional audio file for an example of more simple type of distorted audio (caused in this case by an overdriven microphone).

How to determine whether the problem is the network or a local configuration

Audio is extremely sensitive to network-related packet loss. A loss rate of even one percent can cause serious disruption to audio. The quality of the sound you will hear depends in large part on the nature of the loss. A constant loss or a periodic loss occurring at rapid intervals may create a sound of unintelligible audio.

If your audio sounds unintelligible, follow the steps described in Tutorial 5: Access Grid Network Troubleshooting to ensure that your network is operating properly.

If your network is operating properly, it is likely that your ClearOne device is misconfigured or your microphone does not support the configuration and placement you are using. For example, you may be sitting on the wrong side of a non-omni-directional microphone. It is also possible that poor audio is being caused by an overloaded CPU. Ask the offending site to check their CPU utilization levels on the audio machine.

Correcting the Problem

Microphone problems: Does your microphone support the way in which you are using it? Is the person speaking in proper proximity to the microphone (i.e., not looking away)? If you switch to a different microphone, does it make a difference in sound quality? If it does, check to see if there's anything different about the ClearOne configuration for each microphone. It is also possible that one microphone may be faulty.

EC device misconfiguration: Is your EC device configuration properly set up? If you are using a ClearOne device, check the configuration matrix, ensuring that the microphones are not routed to the EC reference, but PC and telco (if used) outputs are. A microphone routed to the EC reference will result in audio dropouts from that microphone and low levels that don't go up when gains are raised.

TipMake sure you know how all the cables to your EC device are configured. Tracing all the wires will familiarize you with the setup, which may help you to quickly diagnose many types of audio problems.

Overdriven microphones: If your rat or microphone gains are too high, distorted or clipped audio occurs. Often, if microphone inputs are too low, people will attempt to compensate by increasing the rat gain. This causes the rat settings to appear within normal limits, but clipping occurs.

To fix this problem, bring down your rat gain and bring up your microphone gain. You can make this fix in your EC control software by accessing dialog boxes for individual microphone gains. You'll find "coarse" and "fine" adjustments. If the "fine" gain does not provide enough adjustment, check to see if your microphones are designed to use the "coarse" setting. For example, crown PCC-160 are popular microphones for use in an AGN. These microphones normally use the 25db setting. When using a line-leveling device, use the 55db setting. You can also use autogains, especially in rooms without a lot of noise, and in later-model devices such as the ClearOne XAPs.

Adjust the microphone input level so that it does not create distortion. If you are using a ClearOne EC device, use your device control software to adjust the microphone's input level so that it shows green, peaking at about 0db. Then, adjust your rat gain so that it is also peaking out at the top of the green. It is okay if it occasionally goes a bit into the yellow, but it should not go into the red.



 

Lesson 4: Feedback

Feedback is a high pitched squeal usually produced by a positive feedback loop in the microphone/amplifier/speaker system. Feedback usually occurs when the microphone is located in a sound field generated by a speaker that is driven by an amplifier with its level set too high. An example of feedback may be heard in this audio file.

During an Access Grid session, it is strongly recommended that you use an echo canceling device, such as a ClearOne system (formerly Gentner). EC devices allow full-duplex audio so that participants can speak and listen at the same time without experiencing echo, clipping or cutting out of audio. The EC device can also monitor each microphone independently, observing ambient levels and making decisions based on each microphone’s environment. Thus, the input for a microphone located in a highly reverberant part of a conference room is gated, mixed, and processed to compensate for the changes in that area and to reduce echo and noise. This setup improves audio quality and decreases the possibility of feedback.

One downside of full-duplex audio capabilities is that your local microphones may pick up the audio coming through your speakers, which can result in feedback. To stop this type of feedback, move your microphones farther away from your speakers. If this solution does not work, then the feedback may be the result of an overdriven amplifier. Try turning down the level on your amplifier.

Another possible cause of feedback occurs when more than one site has their echo cancellation device on preset 1. Preset 1 is for the Network-Telco Bridge. Only the site that is serving as a bridge between network audio and telephone audio should be on preset 1. It is absolutely necessary that only one site use this preset during any AG session. If more than one site in a session is on preset 1, it is likely that all sites will receive very bad feedback.

Summary

To eliminate feedback:
1. Turn down your volume
2. Increase the distance between your microphones and speakers
3. Make sure your EC device is not on preset 1 unless you are the site bridging network and telephone audio

NoteInformation on the Clear One (Gentner) systems is available at: http://www.clearone.com/products/audio/index.php




Lesson 5: Troubleshooting a Network/Telco Bridge

A network/telco bridge allows the ClearOne (formerly Gentner) echo cancellation device to connect Access Grid network audio to a telephone conference call. Network/Telco bridges are used in the event of serious multicast problems which cause the loss of network audio. This type of bridge is also used in situations where not all participants have an AG node available to them at the time of the meeting. Bridging the AG session with a conference call allows those sites to participate in the session. The Telco operates independently of the Internet and therefore avoids multicast problems.

NoteAG documentation defines three different presets for EC devices. The AG standard recommends that all sites use these presets. For information about setting up these presets see: http://www.accessgrid.org/agdp/howto/config-gentner-ap400/1.4/html/setting-up.html.

Network audio cannot be heard on the telco
In order to bridge network and telephone audio, one AG site in the session must have their EC device on preset 1. If the network audio cannot be heard on the Telco (preset 2), it is likely that the bridge site is not on the correct preset. To solve the problem, set the bridge site switch to preset 1. If the audio is a local problem (only one telco site cannot hear network audio), the error may be an incorrect configuration in the telco sites preset 2. Check that your EC device matrix is set up to receive audio from the Access Grid, so that any audio leaving the sound card enters into the room speakers. However, if the problem is widespread, the bridge site should check their preset 1 configuration to make certain they are sending network audio to the telco.

Telco audio cannot be heard on the network
As with network audio, incorrect bridging may be causing telco audio not to be heard on the network. One AG site in the session must have their EC device on preset 1 for the telco audio to be heard on the network. If the telco audio cannot be heard on the network, it is likely that the bridge site is not on the correct preset. Set the bridge site switch to preset 1 to solve the problem. If this is a local problem (only one telco site cannot be heard on the network), an incorrect configuration in the telco sites preset 2 may be the cause. Check that your EC device matrix is set up to transmit audio to the Access Grid. Audio from any of the audio sources at your node should enter into the sound card and get transmitted to the Access Grid. However, if the problem is widespread, the bridge site should check their preset 1 configuration to be sure they are sending telco audio to the network.

Feedback
One of the most likely causes of feedback occurs when more than one site has their echo cancellation device set to preset 1. Only the site that is serving as a bridge between network audio and telephone audio should be on preset 1. Only one site should use this preset during any AG session. If more than one site in a session is on preset 1, it is likely that all sites will receive very bad feedback. If you are not using the bridge, make sure your EC device is on preset 3 or preset 2. Once all non-bridging sites are on preset 2 (telco) or preset 3 (network) the feedback should stop.