Thursday, May 26, 2016

linux - All HTTP connections refused from public IP yet all others are fine?



I wasn't sure how to describe this problem in the title, so I'll give you a little background on it.



I have a machine running Ubuntu 12.10 Server with Apache 2.2.22 (all packages at latest release version). It has been running very smoothly for about 7 months now, and I've only encountered minor problems with it every so often, mostly as a result of my idiocy.




Last night, I was bored and decided to try out Subterfuge, to see whether or not I could make it work. I was well aware of the fact that its web interface could interfere with my Apache setup, so I installed it and fired it up on TCP port 81. I then immediately checked my sites, and they were all still running, meaning that subterfuge was not interfering with Apache. I started it up and did some scanning, intermittently checking to make sure that my sites were still up; which they were.



After I finished messing around with Subterfuge, I stopped it and tried to CTRL + C the subterfuge process (which was 'attached' to my SSH session). It wouldn't quit, no matter how many times I hit CTRL+C. I closed my SSH session, and logged back in. When I logged back in, all was well and good, but I noticed that my sites were no longer responding (giving a 'connection refused error'). I didn't try to fix it, I just went to bed, figuring that the problem might resolve itself.



Fast forward to this morning. Sites still weren't responding, and I SSH'd into the server to check things out. When I logged in it told me that there were 3 zombie processes, which I then saw when I opened htop. The zombies were all subterfuge processes. I quit them normally (using SIGTERM in htop), and they went away like good little zombies. My sites still weren't responding to connections.



At this point I assumed that this was a problem with my router configuration, so I logged in. I changed the port that was forwarded to an HTTP alt, then an arbitrary private one. That still didn't solve the problem.



A summary of how things stand right now:





  • The server is responding to all other types of connections (SSH, HTTPS, VNC)

  • The server won't respond to HTTP requests from the Internet.

  • The server will respond to HTTP requests from the local network.

  • The server will respond to HTTPS requests from the Internet.

  • An Nmap scan shows port 80 as 'open' when scanning from the Internet.

  • IPTABLES, ufw, etc. are all disabled.

  • I've rebooted the router

  • Another server on the network responds to requests




UPDATE:



I haven't changed any Apache configurations, but now, when you visit the server from a browser, it responds with the default "It works!" page, saying that there is no content in the document root. All of the files that were there before are definitely still there, so I'm about to look into the possibility of permissions problems (or the document root was somehow changed). At this point, it's probably just the Chinese hacking me.



Can anyone think of anything? Thank you for your time in advance, I know this was a very long question.


Answer



In the end, I was unable to solve this problem. This system actually ended up bringing our network to a grinding halt; I think it's infected with some malware. I shut the machine down (after 57 days of uptime), unplugged it from the network, and have yet to bring it back up. I'm currently trying to work out a solution on some really old machines involving XenServer or ESXi. Thanks for the suggestions, but I think this machine has a much bigger problem.


No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...