When I started working in network security, it was like drinking from a fire hose, there was so much to learn.
Up to this time I thought I knew someting about protocols, but I discovered I only knew what they were intended for, I had not studied the protocols to learn precisely how they were supposed to be used and how they could be perverted to gain information about a host, or firewall, or internal network.
Suppose I want to attack hosts with public IPs sitting in someon's DMZ, I can craft a ping request packet destined to a server in the DMZ, with a return address of broadcast on the DMZ's network. The server would respond to the ping on the DMZ broadcast address and all the servers in the DMZ would echo, creating a lot of traffic on the DMZ network. Now if I sent this kind of ping request to each server in the DMZ, think of how much traffic it would create on an already busy network, or set the return. I could also send a similar packet to the broadcast address for a DMZ but set the return address to a server somewhere else on the internet so it would look like company A's servers were attacking a server at company B. Of course, now firewalls and routers are not supposed to allow this kind of traffic in, and servers not allowed to respond to pings.
One of the tools I learned the most from was Shadow, from Naval Surface Warefare Center. The oringinal Shadow was written in perl and used the tcpdump command to gather packet information. A process running on a "sensor" computer would run tcpdump for an hour saving the output to a file. At the end of the hour that process would be killed, and a new tcpdump started spooling to a new file. The previous hour's tcpdump output file would be copied to a server via ssh where it would be analyzed by a number of perl processes creating new files with different kinds of data in each. There was an http part to the system that would display the perl output files for an analyst. The analyst could look at a list of all IPs that had visited his sensor's monitoring point, sorted by IP number an all the traffic that that particular IP had created. Once somtehing interesting has been found you could drill down to see more precisely what had gone on. This gave the analyst the ability to see what each remote IP, or possible attacker, was doing at that monitored point. It was great and man could you learn all about protocols and how they were supposed to be used and how attackers/spammers misused them try to break into your internal system or an individual host. Spammers a usually in a hurry and hammer at a potential soft spot, but real attackers are much more stealthy. It was easy to pick out someone up to no good by just seeing what his packets did. Shadow is a Behavior Based Intrusion Detection system, that is, the analyst must observe the behavior of packets (what kind, how many, etc.) to determine what an attacker is up to. The only problem with Shadow was all the data was over an hour old.
Now after using this tool for a while, and experiencing emergency calls from management about the network being choked up and having to wait an hour to see who was doing it, I started wishing for something a little more on-time.
So I wrote my version of Shadow or just my "Intrusion Detection System". The first system I wrote used tcpdump just like Shadow, to capture the packets, but had a process, written in 'C', "tailing" it's output. The tcpdump captured packet output would be identified, broken apart then sent to a MySql database in real-time. Then I wrote web apps in php, to read, arrange, and display the database information triggered by mouse clicks.
Later I rewrote the packet capture front end to make direct calls to libpcap which speeded things up. Thats how the system works now. I used this system as a prototype for a Logger Daemon I wrote for one of my employers. I even integrated Snort into the packet capture daemon so we could have behavior based and knowledge based IDS in a single, multi threadded, process.
The original Shadow only made one hour's traffic visible at a time, but my IDS can look at an entire 24 hours of traffic or many days if needed. The data is there, it just depends on what php scripts you use.