Unified communications (UC) managers can measure user adoption of communications and collaboration tools fairly easily -- the video conferencing room was booked 12 times this week or the intranet received 957 hits last month. But adoption rates aren't a full measure of the success of a unified communications strategy. UC pros must quantify more subjective metrics like user satisfaction to ensure that UC investments are achieving the productivity gains promised by vendors.
"[Measuring productivity] can be pretty difficult and subjective. It's not a reason not to do it and we probably should be doing it more," said Steve Brescia, enterprise architect for telephony, contact center and collaboration technologies at American Water, a water and wastewater utility company based in Voorhees, N.J. "We deploy it, we support it and we troubleshoot it, but we aren't necessarily taking the pulse as much as we should … which is not good because you're building business cases on those things."
With 7,000 users and a bevy of collaborative tools -- IBM Lotus Notes and Domino 8.5.1, Lotus Sametime, Microsoft SharePoint and Cisco Systems WebEx -- sampling user satisfaction and measuring productivity can become a daunting task for Brescia and his team.
Brescia tries to gauge user satisfaction and measure productivity by occasionally surveying small groups of users about specific UC applications or services. The IT organization as a whole also tries to do enterprise-wide surveys about all IT services each year, he said.
The IT organization has also started a monthly webinar series for users, dubbed "What's going on in ITS," which features long-term unified communications strategy and roadmap updates as well as information on specific applications, Brescia said.
Users can also request software and make "technical enhancement request" through a portal application. These requests also offer Brescia insight into user satisfaction with existing tools and it highlights parts of the unified communications strategy might need more attention.
"We have to take a view of, 'We can't get in front of everything.' We're just never going to be able to do that. We're not large enough," he said. "Through communicating, listening and surveying, we try to get in front of things so that the wave doesn't run right over us."
Keeping in close touch with the end user community has not only helped reaffirm or redirect Brescia's unified communications strategy, but he has also learned that end users are becoming increasingly more tech-savvy, giving technical staff and end users a common language for discussing UC and collaboration technology and user satisfaction.
"We're happily surprised by the fact that when we're describing things or asking them about things that they actually get it or understand it -- that they can actually take the survey," he said. "Not that it was a big surprise to us … but how far it reaches into the organization [did]."
Gauging user satisfaction by monitoring usage
Understanding user satisfaction and measuring productivity remains an enigma for many UC pros -- even those in smaller organizations. Scott Bolser, messaging and collaboration team leader at Children's Hospital Boston, oversees 400 Microsoft Office Communication Server (OCS) users and tries to infer as much as he can from Quest Software's messaging reporting and analysis tool, MessageStat.
"We utilize Quest to see top users and who's using it, so we do get some insight, Bolser said."Instead of us having to reach out and say, 'Are you using it? Are you using it?' we can actually see, 'User A has sent 500 IMs this week.' We can kind of gauge [user satisfaction] from that."
Although Bolser doesn't monitor messaging or do content filtering, he knows that OCS users sent 20,000 instant messages (IMs) one week and he receives ongoing requests from other users for access to the software. This activity suggests that the tool serves its purpose, he said.
"With a lot of products, it tends to be viral. Someone is talking to somebody and they'll say, 'Hey, are you using Office Communicator?' and we'll get a call from someone saying, 'Hey, can you add this person to the Office Communications Server?'" Bolser said. "We want to provide tools for people to enhance [their productivity] as best we can."
Unified communications strategy: When are too many UC tools counterproductive?
With the proliferation of devices and communications media, UC pros must also be aware that end users can see their productivity sapped by having too many tools.
IM can get a bad rap if users are doing more chatting than collaborating, Bolser acknowledged. But keeping a tight leash on who gets access and maintaining OCS as a closed environment -- so that users can't chat with friends on consumer platforms -- prevents IMing from becoming counterproductive, he said.
"Our philosophy is to try to keep it for the user as simple as possible … so they're not bogged down by this barrage of various communications devices coming their way," Bolser said. "We do video conferencing when we need to -- [Adobe Systems'] Breeze, WebEx and those things too -- but for the majority of users, it's a traditional telephone, IM for those people who need it, which is a small subset in the organization, and email."
Having users go through a request form for new technology and enforcing an "approved tools list" also helps control the influx of applications, Brescia said.
"Somebody recently wanted to have us add Microsoft Office Communicator for their collaboration tool and we said, 'Well, why don't you try WebEx?' because WebEx can do the same kinds of things … and it's on our approved tools list," Brescia said.
Although it can be difficult to gauge how much UC and collaboration applications may help productivity, Brescia said end users are never shy about letting him know when they hinder it. After users expressed dissatisfaction with a recent SharePoint application deployment, Brescia and his team had to shut it down, redesign and redeploy it.
"When people aren't able to do their jobs, it's like talk radio," he said. "You're not going to hear from the people who are happy or think the team did the right thing. It's, 'Why didn't we win?' and 'Why didn't we trade this guy?'"