How does a free software firewall compares to a commercial firewall solution ? In order to get some answers, O'Reilly SysAdmin team set up a benchmark test for Cisco PIX, SmoothWall and OpenBSD. The results are quite interesting.
The evaluation criteria included :
Discovery tests :
Network sniffer - Documents the discovery of the target IP address and any other useful information, such as protocols being used on the target network
Traceroute - Attempts to locate the target device and all intermediate routers, switches, and systems
Penetration tests
Synflood attack - Used to see whether the firewall can overcome a repeated open connection request and also log the attack
Garbage attack - Used to see whether the firewall can overcome random data packets on random ports
UDP Ping - Used to see whether the firewall can overcome a large UDP ping packet sent to it
TCP Ping - Used to see whether the firewall can overcome a large TCP ping packet sent to it
Ping of death - Used to see whether the firewall can overcome a single over-sized packet sent to it
Cisco PIX Results Regardless of which port the attack used, with the state full packet inspection activated, the Cisco PIX blocked all transmissions on every test we conducted. The PIX also continued to allow the proper connections that were not considered attacks during the tests. The PIX effectively blocked the outgoing and incoming packets. One of the few issues with the PIX is finding proper documentation. The PIX was designed with a professional support team in mind, not the typical home user.
OpenBSD Results OpenBSD is everything one might expect from an open source firewall. It has the power and potential of the PIX without the cost. As for performance, OpenBSD performed just as well as the Cisco PIX at blocking unwanted incoming or outgoing packets with no degradation to the system. OpenBSD also kept detailed text logfiles of each attack, which were fairly easy to read. Like SmoothWall, OpenBSD does not provide any type of graphical results analysis of the logged attacks; the PIX does provide this
SmoothWall Express Results Compared to the PIX, SmoothWall was more simplistic in design and easier to configure, but also less robust. Unlike the PIX, SmoothWall uses stateless packet inspection. Attacks on specific ports locked up the firewall system until the attack stopped. SmoothWall was designed with the home user in mind, not corporations.
I tried to get a confirmation about stateless vs statefull inspection from SmoothWall, but I couldn't find a contact email address on http://smoothwall.org/
