From the editor: A SearchUnifiedCommunications.com member submitted the aforementioned question about background noise and echo issues in their 100-seat call center. Matt Brunk, our unified communications strategies expert, generously provided a detailed, step-by-step strategy to accurately identify, address and resolve voice background noise and echo problems.
According to Matt, in most cases, the fastest route to resolving noise and echo problems is answering the who, what, where, and when questions before moving ahead. Who is experiencing what problem? When does the problem occur? At what times/dates? Are they outbound or inbound calls and what kind of calls are they?
Those having issues with background noise and echo would be wise to bookmark this page.
I have no experience with any of those solutions listed. First I suggest you review dealing with echo. You said you have a 100-seat call center with "some" calls suffering with background noise and echo. The part that worries me is "suffering."
Causes of background noise
Background noise and echo are two issues. The background noise could be introduced by several things including: the environment, the user phone, the headset, the softphone, the workstation, cables, the ground loop on the equipment, network gear issues and/or circuits.
Causes of echo
With respect to echo, is it that you have some congested links either with the LAN and/or WAN? Jammed up WAN links can account for some, but not all, issues. Look at the dropped packets and out-of-order packets for the LAN and look at packet loss and out-of-order packets on the WAN links. Check your switch port metrics, too. Alignment errors point to cabling issues that contribute to noise.
When you remove everything from the call path -- voice line (analog, PRI, T1, SIP trunk) -- and do end-point, outbound calling, do you get background noise and/or echo? Peel back and away from the demarcation. But before you do that, you really must identify who, what, where, and when to resolve background noise and echo issues.
Resolving background noise and echo: First identify who, what, where and when
Who is experiencing what problem (background noise and/or echo)? When do the problem(s) occur (beginning or end of call, during call)? At what times/dates? Are they outbound or inbound calls and what kind of calls are they: internal, local, LD or international? Now go forward with a call and map out the path it takes from the desktop all the way through your gear and then to your carriers and/or providers including on-site gear for them.
Determining if the source of echo is internal or external
You could setup a port mirror in your switch and run WireShark (packet trace) for a period of time and then spend even more time digging through the calls and playing back each call to determine if the source of echo and noise is internal or external. Check out Chris Greer's post,WireShark quick tip: Replaying captured VoIP calls.
The solutions you mentioned (SoliCall PBXMate and Octastic) indicate you may have a deficiency in your configuration, some limitations that lack in gateway capabilities, such as the ability to compensate for echo. Or perhaps you are doing something with protocol conversions?
The other possibility is that your client management has endured these issues for a length of time and just want relief. Another thing to consider -- does the Asterisk solution have echo cancellation? If not then find out why.
Onboard echo cancellation – a must have
I looked at both SoliCall and Octastic solutions -- if onboard echo cancellation wasn't included in the initial configuration then that was an oversight. The SoliCall solution on the surface seems really cool. But still, you could be asking your client throw money at the symptoms and still not solve the problems.
Let me throw some advice out there for other readers – never buy any VoIP/SIP/IPT solution without echo cancellation onboard. Why? You will need it. The North American Telecommunications Network is not all IP -- far from it -- and your configurations are not all IP. If they are, you may be facing trouble.
Let's find out if your agents are using noise-canceling headsets? Are the headsets VoIP compatible -- meaning do they have echo-canceling features? Do you have POTS or FXO/FXS in the configuration? Is there a legacy PBX tied into this configuration?
Call center managers are allured by the low costs of thin clients and softphones. Hopefully your configuration supports them if they are used. Desktop switches and hubs need to be eliminated. You may have a bad router interface or gateway equipment that is electrically damaged because WAN and power isn't protected.
Before you buy, you really need to identify and isolate the sources of both background noise and echo. The sources may be simple, but you may find they are complex. It's laborious, but without some investment in time, you may be hiding/masking other issues that could arise in the future.
The trouble with headsets
Because you didn't elaborate on details of the background noise, but did say you have a call center, I strongly encourage you to use noise-canceling headsets and make sure they are VoIP ready, but that's only a scratch. Telephone headsets are battery powered, so you want to avoid a host of other issues by simply using AC power adapters for them. When the batteries run low they may not have enough power to amplify voice over the always-present noise.
Noise and echo are always present, but you can mitigate the impact
Noise and echo are always present – remember this. The agents' workstations also need sound and acoustical considerations as do the work areas because outside noise may be heard. Work surfaces, wall, ceiling and flooring all contribute to the conditions of how well or poorly sound will travel.
I'm guessing you have managed switches and some distance among the 100 souls in that call center. If you have switches at two locations, then you may want to link them via fiber and get off of copper. If you don't have two different switch locations, then you may want to look hard at long cable lengths. You didn't say whether this was a new condition after implementation or the same condition since implementation.
Look for peak-traffic load patterns
You also need to look for patterns such as peak traffic loads/busy times, overflow voice traffic hitting other pipes/routes/circuits that may be the source of background noise and or echo. Long local loops in the PSTN have a high-loop current – that means echo in IP. You can attenuate loop current, but you need to test the levels first. There are two test devices I like:
- MetrolTel Voice Network Analyzer is good for POTS/Centrex services and it has a digit grabber good for VM/AA and it will display loop current in all call states.
- Loop current tester from Mike Sandman. Sandman also sells loop current attenuators.
Eliminate desktop switches and hubs
If you realize you have desktop switches, hubs, or more than one router or firewall, then you need to eliminate the desktop switches and/or hubs and assess the value of having more than one router and/or firewall.
New IP systems will still have old problems like cords or broken handsets
You may have a T1 that is extended from the minimum point of penetration (MPOP) -- usually the basement or underground service to the building to the equipment location. Telcos don't test the "extended demark." They only test to the smart jack at the MPOP. You may have a bad or defective cable pair or simply a bad smart jack. Thus my note about when these problems started – since implementation or afterwards?
For example, I recall a car dealership that installed an IP PBX. The dealership was swapping hardware to no avail in an attempt to solve a noise problem. Later the dealership discovered the real issue, but only after documenting the problem. The noise was isolated to several sales desks on the showroom floor. The patch cables to and from each phone were exposed on the floor and the sales people used duct tape to secure the patch cables. Customers and personnel walked on the patch cables and, over time, the cables shorted. The phones were powered via PoE and thus the source of the noise (static).
A couple of years ago while attending VoiceCon in Orlando, I asked the panel covering Troubleshooting converged networks session some warm-up questions. The panelists were all CTOs and held doctorates in the field. All the questions were geared toward users calling the help desk with problems about their telephones. Each user complaint was addressed based upon the symptoms I described.
First symptom: Noise. Not one of these panelists addressed faulty handset cords, bad resistors in the handset (dropped handset), bad line, cord/patch cord or any of the old familiar causes of noise. There was no wrong answer but the exercise proved my point. In IP you must include the old with the new – just because you are on an IP solution doesn't mean you won't have bad handset cords or broken handsets causing noise, because you will and you will also inherit all the issues possible along with new ones in an IP environment.
QoS and compression
I need to mention QoS and compression, too. If your QoS policy is mismatched with your available useable bandwidth, then QoS won't do you any good especially when traffic starts to climb and peak. In your compression methods, you want to be uniform and consistent and ensure that your providers/carriers support compression and whether or not they have a first, second, third-choice preference.
Next, check the sampling rate in your compression method. Are these rates supported by all providers/carriers you are using or do you have separate entries for each? Do you have voice metrics tools to view your calls? I can assume not since you are seeking answers. So this directs me to advise you to get the tools.
A call center without these tools spells disaster because of sheer volumes of calls that are handled. My favorite tool embedded in enhanced software of ADTRAN gear is VQM (Voice Quality Management). This tool takes the mystery out of a lot of issues.
Quick fix: Could bad cabling be the cause of noise and echo?
Let me swing back to some other easy things to look for: Have all the Ethernet drops been tested and scanned? During testing did you include patch cables at both ends? The really annoying issues with cable pairs terminated incorrectly, mix and match of 568A and 568B patch cables, bad cable terminations and just bad cabling make life challenging. Hand-crimped patch cords belong in the recycling bin, too. How many different handoffs do you have? Do you have ATA devices, a router with some cards, FXO/FXS, multiple routers, analog, T1, PRI, SIP trunks, etc?
The point I'm trying to make is to answer these key questions first: Who? What? Where? When? You will probably get to a resolution faster. I've thrown a lot at you and I could go on and on about background noise and echo, but just remember to do everything systematically after you first address and answer the key questions.
This was first published in June 2010