I have been running my server for the past 6 years without any problem, but recently ran into some trouble. I am getting a lot of SYN_RECV connections which is pushing the backlog of connecting devices so far back and nothing is updating as it should. Now my server is a Intel(R) Xeon(R) CPU E31230 @ 3.20GHz with 7 cores with 32GB memory and running CentOS 6.7 Red Hat. I have about +- 800 active connections running on this machine. And I have 12 ports running and all +-800 devices is split over these 12 ports with max 100 devices per port. I have my sockets created/running through PHP. I would appreciate ANY advice.
-Would it help to have everything on 1 port?
-Would it help to extent them over more ports (Max 50 devices per port)
-Would it help to tweak the linux settings (which settings would it be)
-Would it help to write it in another language (which would be best if any)
-What else could cause this problem?
I have SYN Cookies enable to help with SYN Attacks:
sysctl -n net.ipv4.tcp_syncookies
This is my sysctl.conf:
/# Kernel sysctl configuration file for Red Hat Linux
/#
/# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
/# sysctl.conf(5) for more details.
/# Controls IP packet forwarding net.ipv4.ip_forward = 0
/# Controls source route verification net.ipv4.conf.default.rp_filter = 1
/# Do not accept source routing net.ipv4.conf.default.accept_source_route = 0
/# Controls the System Request debugging functionality of the kernel kernel.sysrq = 0
/# Controls whether core dumps will append the PID to the core filename.
/# Useful for debugging multi-threaded applications. kernel.core_uses_pid = 1
/# Controls the use of TCP syncookies net.ipv4.tcp_syncookies = 1
/# Disable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
/# Controls the default maxmimum size of a mesage queue kernel.msgmnb = 65536
/# Controls the maximum size of a message, in bytes kernel.msgmax = 65536
/# Controls the maximum shared segment size, in bytes kernel.shmmax = 68719476736
/# Controls the maximum number of shared memory segments, in pages kernel.shmall = 4294967296
/# Retry SYN/ACK only three times, instead of five net.ipv4.tcp_synack_retries = 5
/# Try to close things only twice net.ipv4.tcp_orphan_retries = 5
/# FIN-WAIT-2 for only 5 seconds net.ipv4.tcp_fin_timeout = 30
/# Increase syn socket queue size (default: 512) net.ipv4.tcp_max_syn_backlog = 1024 net.core.netdev_max_backlog = 1000
/# One hour keepalive with fewer probes (default: 7200 & 9) net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_keepalive_probes = 5
/# Max packets the input can queue net.core.netdev_max_backlog = 65536
/# Keep fragments for 15 sec (default: 30) net.ipv4.ipfrag_time = 30
/# Use H-TCP congestion control net.ipv4.tcp_congestion_control = htcp
net.core.rmem_default = 256960 net.core.rmem_max = 256960
net.core.wmem_default = 256960 net.core.wmem_max = 256960
net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 0 net.ipv4.ip_local_port_range = 15000 61000
net.core.somaxconn = 1024
Minus the "/" before the "#"
Here is my PHP code for running the ports:
#!/usr/bin/php -q
error_reporting(0);
set_time_limit(0);
ob_implicit_flush();
$address = '123.123.123.123';
$port = 8000;
//Check for the connections
if (($master = socket_create(AF_INET, SOCK_STREAM, SOL_TCP)) < 0)
{
logs("socket_create() failed, reason: " . socket_strerror($master) . "\n", $enable_logging);
}
socket_set_option($master, SOL_SOCKET,SO_REUSEADDR, 1);
if (($ret = socket_bind($master, $address, $port)) < 0)
{
logs("socket_bind() failed, reason: " . socket_strerror($ret) . "\n", $enable_logging);
}
if (($ret = socket_listen($master, SOMAXCONN)) < 0)
{
logs("socket_listen() failed, reason: " . socket_strerror($ret) . "\n", $enable_logging);
}
$read_sockets = array($master);
//Read all data from buffer
while (true)
{
$changed_sockets = $read_sockets;
$num_changed_sockets = socket_select($changed_sockets, $write = NULL, $except = NULL, NULL);
foreach($changed_sockets as $socket)
{
if ($socket == $master)
{
if (($client = socket_accept($master)) < 0)
{
logs("socket_accept() failed: reason: " . socket_strerror($msgsock) . "\n", $enable_logging);
continue;
}
else
{
array_push($read_sockets, $client);
logs("[".date('Y-m-d H:i:s')."] ".$client." CONNECTED "."(".count($read_sockets)."/".SOMAXCONN.")\r\n", $enable_logging);
}
}
else
{
$buffer = socket_read($socket, 8192);
if ($buffer === "")
{
$index = array_search($socket, $read_sockets);
unset($read_sockets[$index]);
socket_close($socket);
}
else
{
//Do DB connections etc here
}
}
}
}
?>
I open and close the ports with a bash script.
Any any help, I'm open for anything.
No comments:
Post a Comment