centos - Too many TIME_WAIT state connections!

I've been reading about this everywhere all day, and from what I've gathered, TIME_WAIT is a relatively harmless state. It's supposed to be harmless even when there's too many.

But if they're jumping to the numbers I've been seeing for the past 24 hours, something is really wrong!

[root@1 ~]# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
  1 established)
  1 Foreign
 12 CLOSE_WAIT
 15 LISTEN
 64 LAST_ACK
201 FIN_WAIT2

334 CLOSING
605 ESTABLISHED
816 SYN_RECV
981 FIN_WAIT1
26830 TIME_WAIT

That number fluctuates from 20,000 to 30,000+ (so far, the maximum I've seen it go is 32,000).
What worries me is that they're all different IP addresses from all sorts of random locations.

Now this is supposed to be (or was supposed to be) a DDoS attack. I know this for a fact, but I won't go into the boring details. It started out as a DDoS and it did impact my server's performance for a couple minutes. After that, everything was back to normal. My server load is normal. My internet traffic is normal. No server resource is being abused. My sites load fine.

I also have IPTABLES disabled. There's an odd issue with that too. Every time I enable the firewall/iptables, my server starts experiencing packet loss. Lots of it. About 50%-60% packets are lost. It happens within an hour or within a few hours of enabling the firewall. As soon as I disable it, ping responses from all locations I test them from start clearing up and get stable again. Very strange.

The TIME_WAIT state connections have been fluctuating at those numbers since yesterday. For 24 hours now, I've had that, and although it hasn't impacted performance in any way, it's disturbing enough.

My current tcp_fin_timeout value is 30 seconds, from the default 60 seconds. However, that seems to not help, at all.

Any ideas, suggestions? Anything at all would be appreciated, really!