Problem solve Get help with specific problems with your technologies, process and projects.

Detecting malicious traffic on VoIP-enabled networks with Wireshark

Network engineers can detect malicious traffic on their VoIP-enabled networks using a free network sniffing tool called Wireshark.

Many types of packets, protocols and penetration attempts bounce against the exterior of a network, and many devices...

have been developed to distinguish between them all. Simultaneously, in an effort to consolidate network resources and leverage existing, on-staff talent, many organizations have moved swiftly in the direction of unified communications solutions. Furthermore, security appliance vendors are stretched thin trying to detect malicious traffic and keep up with the latest malware developments. So how is a network engineer able to secure their networks under these circumstances? Outside of typical solutions, like IDS/IPS, next-generation firewalls and endpoint security, one tool that system administrators may want to consider is Wireshark.

Because Voice over Internet Protocol (VoIP) leverages existing IP infrastructure, many vulnerabilities that currently affect traditional IP communications also affect VoIP communications. Therefore, Trojans, DDoS attacks and unauthorized exfiltration of sensitive information are very much a concern within VoIP-enabled networks. One malicious traffic attempt that merits attention here is the SIP-flood attack.

Session Initiation Protocol (SIP) has quickly become the de facto standard for application-layer signaling within VoIP networks. Defined by RFC 3261, SIP requests consist of the following items: REGISTER, INVITE, ACK, CANCEL, BYE and OPTIONS. The portion of the SIP request that will be focused on here is the INVITE request.

SIP has often been described as HTTP-like in its methodology of setting up communication sessions, and the portion of SIP that most closely resembles the HTTP-GET request is SIP INVITE. A SIP INVITE request is the portion of SIP that sets up a session between two user-agents, and within this setup lies a profound denial of service (DoS) vulnerability. Much like HTTP utilizes a three-way handshake via TCP to establish logical connections with remote hosts, SIP utilizes a similar handshaking method that consists of INVITE and 200 OK messages. When a disproportionate number of incomplete handshakes are detected, this is a telling sign that an INVITE flood is underway.

So how does one detect such malicious traffic? Several security appliances, such as firewalls and intrusion detection systems, currently exist that use statistically based, algorithmic methods to detect the previously mentioned disproportionate, incomplete handshakes. However, if an IT professional were to become curious as to just how effective their particular security appliance is at detecting such events, creating the requisite Wireshark filters will do wonders in the way of quality assurance.

Wireshark filters for detecting malicious traffic on VoIP-enabled networks

First, ensure that Wireshark is installed on a device that can receive SIP calls from within the same network that maintains the SIP server. Open Wireshark, and select the interface that VoIP communications traverse. This is often done on eth0 (your Ethernet card). Next, begin the Wireshark packet capture, and a flood of network traffic should begin crossing the monitor.

At this point, you can begin flooding one of the SIP end-devices by starting the invite flood tool of your choice. A free version known as inviteflood can be found in most distros of BackTrack Linux. Assume that the SIP address for one of the end-devices is [email protected], and begin flooding this address with INVITE traffic with approximately 200 requests per second. The Wireshark application should begin capturing numerous packets of the REGISTER, INVITE, and OK variety. Next, insert the following filter into the filter field within Wireshark:

sip.Method=="INVITE" || sip.Status-Line=="SIP/2.0 200 OK" || sip.Method=="ACK"

This filter will allow the end user to view all INVITE, OK, and ACKNOWLEDGE (ACK) traffic, and if the invite flood attack was configured correctly, the end user should see no ACK packets. Why is this?

In a traditional INVITE flood attack, an attacker sends an INVITE request to a specific SIP address, and when the SIP end-device eventually responds with a 200 OK message, the attacker never responds with the third leg of the three-way handshake -- the ACK message. However, the victimized end device does not know that the expected ACK packet will never arrive, so it sets aside resources while more INVITE packets enter into its queue. At a certain threshold the end device's buffer fills up, and the end device crashes, or in some cases goes through a factory reset.

Read the previous tips in this Wireshark series

Tip 1: Learn to get a Wireshark VoIP packet capture

Tip 2: Wireshark packet capture filtering: Learn how to parse VoIP traffic

Where Wireshark can be of assistance is it allows the end user to view the ratio of INVITE/OK to ACK packets. If the end user determines that there is too much disparity in the ratio, security administrators can be alerted so that certain measures can be implemented. For example, network security administrators can block IP addresses or domains that are generating unusually high numbers of unacknowledged OK messages. Furthermore, Wireshark can be utilized as a method to ensure that in-place security appliances are in fact blocking the malicious traffic they claim to be blocking.

In a unified environment where a blurring of the lines between voice and data communications is constantly underway, it's nice to know that a traditionally data-centric application can also be utilized to monitor voice communications as well.

Dig Deeper on Unified Communications Security