Wednesday, May 31, 2017

linux - Cygwin 2.9.0 cat/tac Commands Fail on Large Files when Piping to grep -q -m1




I am seeing some odd behavior using Cygwin x64 2.9.0 on Windows 10 Pro x64. The command I am attempting to run is the following:



tac  | grep -q -m1 -F "literal string"


The above command succeeds on all small files that I throw at it (small means <= 15kB). It also succeeds if the final occurrence of literal string is near the start of the file (e.g., literal string appears near the top of the file and nowhere else). Finally, it also succeeds when neither of the {-q, -m1} flags is passed to the grep command.



However, when the file is around 680kB, and the literal string appears near the end of the file, then the tac command prints "tac: write error" to STDERR. Despite this error, the command appears to have succeeded, printing the matching line to output (when the -q flag is omitted) and getting the appropriate return value from grep.



Further testing has revealed that this same error occurs when using cat, except the literal string must appear near the start of the file to generate the error, and the generated error is "cat: write error: No space left on device".




Note that this only occurs if at least one of the {-m1, -q} options is passed to the grep command, the match is near the first processed line of the file (for cat it is near the beginning, for tac it is near the end), and the file is large.



I have run the df command, and it reports 14 MB available on the Cygwin drive, with 60 GiB free on the actual disk. I know I could simply redirect STDERR to the NUL device, but that seems like a hacky work-around. Does anyone know how to fix this properly?



BEGIN EDIT



I found another report of the same error from May 2017, but no solution was presented. The OP of the other post does indicate that he thinks this is a pipe buffer size limitation (perhaps on Windows, perhaps in Cygwin).


Answer



I have discovered a few work-arounds. Simply change the command:




tac  | grep -q -m1 -F "literal string"


to one of:



bash -c "tac  | grep -q -m1 -F 'literal string'"
stdbuf -o L tac | grep -q -m1 -F "literal string"



I think the first works because it is using a Linux pipe, and the second because it forces the tac command output to be line-buffered. Both of these forms make the error go away.



Since this works, I'm guessing the issue is that grep stops processing the input buffer once it finds the first match, but tac keeps processing input. Once the buffer is full (probably 64kiB), the buffer blocks and tac exits with the specified error. However, since tac successfully processed the line I care about before crashing, everything is working as intended.



Timing these options indicates that the call to bash is the faster option. This is probably because using the Linux pipe, tac is able to return immediately once grep finds the first match.


ubuntu - Apache Passenger can't find gem




I'm running Ubuntu 10.04 and I've transferred over some sites built in Sinatra. I've set up Phusion passenger, but when I visit the sites I'm getting a Passenger LoadError claiming that passenger has 'no such file to load -- sinatra' yet when I run gem list or sudo gem list, I clearly see sinatra listed. Why can't passenger find this gem?



My sudo gem env output looks like this




RubyGems Environment:
- RUBYGEMS VERSION: 1.3.5
- RUBY VERSION: 1.8.7 (2009-12-24 patchlevel 248) [x86_64-linux]
- INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8
- RUBY EXECUTABLE: /usr/local/bin/ruby
- EXECUTABLE DIRECTORY: /usr/local/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- /usr/local/lib/ruby/gems/1.8
- /root/.gem/ruby/1.8
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://gems.rubyforge.org/




running 'sudo ruby -v' I see the following:
ruby 1.8.7 (2009-12-24 patchlevel 248) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2010.01




Is that correct, or should the two ruby versions match up correctly, displaying REE in both?



Thanks in advance!


Answer



Check your Passenger module configuration files. In my case



content of /etc/apache2/mods-enabled/passenger.load:



LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger2.2.5/ext/apache2/mod_passenger.so



content of /etc/apache2/mods-enabled/passenger.conf:




PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.5
PassengerRuby /usr/local/bin/ruby




PassengerRuby is telling module what ruby interpreter to use.


Tuesday, May 30, 2017

networking - Forward port to another Ip/port




I recently found out that my ISP disables port 135. I have a server and personal computer behind my router. A friend and I are working on a programming project and are running into some sql issues because of this blocked port. For him to be able to access everything, I have dynadns configured



Personal Computer: 192.168.1.100
Server Computer: 192.168.1.110



Is there a way for me to forward localhost traffice from port 135 to say port 8081 and then have my server take incoming requests from port 8081 to 135? I have heard a little bit about ssh tunneling but it seems odd to be using Putty to login to my local machine where the ssh server is running to forward ports.



Any advice would be very helpful.




Thanks



EDIT: I am using a router with dd-wrt installed and have installed the vpn addon. I was trying to get VPN working on my windows 2003 computer but was having problems with the GRE protocol not being configured. I will attempt the vpn option with my router and post my results later today.


Answer



You want to set up a VPN on one of your computers, and have the other one log into that one. That way they will appear to be on the same subnet, and have full access to each other without exposing vulnerable ports to the world.



A really good Open Source VPN is OpenVPN. I've only used the server under linux though, so I can't comment on how easy it is to set up on windows.



Another option is to flash your router with dd-wrt firmware. There is a version that includes an OpenVPN server, and would connect your entire network to his.


Cron emails output even when redirecting to /dev/null 2>&1



The code below should redirect all output to /dev/null however root still receives an email with output from the first command (/usr/bin/svnadmin). Why?



#!/bin/sh
# Dump repo and upload via tarsnap


/usr/bin/svnadmin dump /srv/repos/www | gzip > /srv/repos/backup/www-repo-`date --iso-8601`.gz > /dev/null 2>&1
/usr/local/bin/tarsnap -c -f www-repo-`date --iso-8601` /srv/repos/backup/www-repo-`date --iso-8601`.gz > /dev/null 2>&1

Answer



You have piped the second commands STDOUT and STDERR to /dev/null, and only piped the first command's STDOUT to the second command. So the first command's STDERR is e-mail-ed to you.



You should write



( /usr/bin/svnadmin dump /srv/repos/www | gzip > /srv/repos/backup/www-repo-`date --iso-8601`.gz ) > /dev/null 2>&1


linux - installing latest apache on centos



I'm trying to install the newest version of apache on my centos server. I did the following:




Download   $ wget http://httpd.apache.org/path/to/latest/version/
Extract $ gzip -d httpd-2_0_NN.tar.gz
$ tar xvf httpd-2_0_NN.tar
Configure $ ./configure
Compile $ make
Install $ make install
Test $ PREFIX/bin/apachectl start


And that all worked except the last step, when i type apachectl start it says 'command not found'. I ran this command from /usr/local/apache2/bin/ where it is installed but no cigar. Any idea what i am doing wrong? Thanks.


Answer




It's not working because you're literally giving it the string PREFIX/bin/apachectl start instead of replacing PREFIX with the location into which you installed Apache, as the instructions you are following intend.



In your case, that's apparently /usr/local/apache2, so run



/usr/local/apache2/bin/apachectl start


But, that still leaves us with the larger issue of Why Are You Doing This? The vendor packages provided by CentOS include bug fixes and security patches, and in fact are regularly updated to ensure that newly-discovered issues are fixed.



Meanwhile, Apache development within the 2.2.x branch is rather conservative, and there aren't major new features (if any!) you'll gain by going with 2.2.17 (the current latest). And, that's not even going into the fact that you seem to have gotten the rather old 2.0 version. Did you maybe want the incompatible, unstable 2.3 development version?



Monday, May 29, 2017

windows - What is Group Policy and how does it work?





This is a Canonical Question about Active Directory Group Policy Basics




What is Group Policy? How does it work and why should I use it?



Note: This is a Question & Answer to new administrator that might not be familiar with how it functions and how powerful it is.


Answer



What is Group Policy?




Group Policy is a tool that is available to administrators that are running a Windows 2000 or later Active Directory Domain. It allows for centralized management of settings on client computers and servers joined to the domain as well as providing a rudimentary way to distribute software.



Settings are grouped into objects called Group Policy Objects (GPOs). GPOs are linked to an Active Directory organizational unit (OU) and can be applied to users and computers. GPOs cannot be applied to groups directly, though you can use security filtering or item-level targeting to filter policy application based on group membership.



That's cool, what can it do?



Anything.



Seriously, you can do anything that you want to users or computers in your domain. There are hundreds of pre-defined settings for things like folder redirection, password complexity, power settings, drive mappings, drive encryption, Windows Update, and so on. Anything that you can't configure via a pre-defined setting you can control via scripting. Batch and VBScript scripts are supported on all supported clients and PowerShell scripts can be run on Windows 7 hosts.





Professional tip: You can actually run PowerShell startup scripts on Windows XP and Windows Vista hosts as well as long as they have PowerShell 2.0 installed. You can make a batch file that calls the script with this syntax:




powershell Set-ExecutionPolicy RemoteSigned
powershell \\\\server\share\script.ps1
powershell Set-ExecutionPolicy Restricted




The first line allows unsigned scripts from remote shares to be run on that host and the second line calls the script from the batch file. The third line sets sets the policy back to restricted (the default) for maximum security.




How are Group Policy Objects applied?



GPOs are applied in a predictable order. Local policies are applied first. There are policies set on the local machine via gpedit.msc. Site policies are applied second. Domain policies are applied third, and OU policies are applied fourth. If an object is nested inside of multiple OUs, then the GPOs are applied at the OUs closest to the root first.



Keep in mind that if there is a conflict, the last GPO applied "wins." This means, for example, that the policy linked at the OU that a computer resides in will win if there is a conflict between a setting in that GPO and one linked in a parent OU.




Logon and Startup Scripts seem cool, how do those work?



A logon or startup script can live on any network share as long as the Domain Users and Domain Computers groups have read access to the share that they are on. Traditionally, they reside in \\domain.tld\sysvol, but that's not a requirement.



Startup scripts are run when the computer starts up. They are run as the SYSTEM account on the local machine. This means that they access network resources as the computer's account. For example, if you wanted a startup script to have access to a network resource on a share that has the UNC of \\server01\share1 and the computer's name was WORKSTATION01 you would need to make sure that WORKSTATION01$ had access to that share. Since this script is run as system, it can do stuff like install software, modify privileged sections of the registry, and modify most files on the local machine.



Logon scripts are run in the security context of the locally logged on user. Hopefully your users aren't administrators, so that means that you won't be able to use these to install software or modify protected registry settings.



Logon and startup scripts were a cornerstone of Windows 2003 and earlier domains, but their usefulness has been diminished in later releases of Windows Server. Group Policy Preferences gives administrators a much better way to handle drive and printer mappings, shortcuts, files, registry entries, local group membership and many other things that could only be done in a startup or logon script. If you're thinking that you might need to use a script for a simple task, there's probably a Group Policy or preference for it instead. Nowadays on domains with Windows 7 (or later) clients, only complex tasks require startup or logon scripts.




I found a cool GPO, but it applies to users, I want it to apply to computers!



Yeah, I know. I've been there. This is especially prevalent in academic lab or other shared computer scenarios where you want some of the user policies for printers or similar resources to be based on the computer, not the user. Guess what, you're in luck! You want to enable the GPO setting for Group Policy Loopback Mode.



You're welcome.



You said I can use this to install software, right?



Yep, you can. There are some caveats, though. The software must be in MSI format, and any modifications to it must be in an MST file. You can make an MST with software like ORCA or any other MSI editor. If you don't make a transform, your end result will be the same as running msiexec /i /q




The software is also only installed at startup, so it's not a very fast way of distributing software, but it's free. In a low-budget lab environment, I've made a scheduled task (via GPO) that will reboot every lab computer at midnight with a random 30 minute offset. This will ensure that software is, at a maximum, one day out of date in those labs. Still, software like SCCM, LANDesk, Altaris, or anything else that can "push" software on an on-demand basis is preferable.



How often is it applied?



Clients refresh their Group Policy Objects every 90 minutes with a 30 minute randomization. That means that, by default, there can be up to a 120 minute wait. Also, some settings, like drive mappings, folder redirection, and file preferences, are only applied on startup or logon. Group Policy is meant for long-term planned management, not for instant quick-fix situations.



Domain Controllers refresh their policy every five minutes.


linux - IPv6 working fine, IPv4 throws OpenSSL error



I am building a webserver ( http://blog.linformatronics.nl/ ), which functions just fine on both IPv4 and IPv6 and when using a non-SSL connection. However when I connect to it through https, IPv6 works as expected, but an IPv4 connection throws a client side error. Server side logs are empty for the IPv4/https connection. Summarized in a table:



     | http  | https
-----+-------+-------------------------------------------------------
IPv4 | works | OpenSSL error, failed. No server side logging.

-----+-------+-------------------------------------------------------
IPv6 | works | self signed certificate warning, but works as expected


Apparently the SSL tunnel isn't even set up, which accounts for the Apache logs being empty. But why does it work fine for IPv6 and fail for IPv4?
My question is why is this OpenSSL error being thrown and how can I solve it?



Below is some extra information about the setup.







IPv6 https



Command used to reproduce IPv6/https behaviour:



$ wget --no-check-certificate -O /dev/null -6 https://blog.linformatronics.nl
--2012-11-03 15:46:48-- https://blog.linformatronics.nl/
Resolving blog.linformatronics.nl (blog.linformatronics.nl)... 2001:980:1b7f:1:a00:27ff:fea6:a2e7
Connecting to blog.linformatronics.nl (blog.linformatronics.nl)|2001:980:1b7f:1:a00:27ff:fea6:a2e7|:443... connected.
WARNING: cannot verify blog.linformatronics.nl's certificate, issued by `/CN=localhost':

Self-signed certificate encountered.
WARNING: certificate common name `localhost' doesn't match requested host name `blog.linformatronics.nl'.
HTTP request sent, awaiting response... 200 OK
Length: 4556 (4.4K) [text/html]
Saving to: `/dev/null'

100%[=======================================================================>] 4,556 --.-K/s in 0s

2012-11-03 15:46:49 (62.5 MB/s) - `/dev/null' saved [4556/4556]



IPv4 https



Command used to reproduce IPv6/https behaviour:



$ wget --no-check-certificate -O /dev/null -4 https://blog.linformatronics.nl
--2012-11-03 15:47:28-- https://blog.linformatronics.nl/
Resolving blog.linformatronics.nl (blog.linformatronics.nl)... 82.95.251.247
Connecting to blog.linformatronics.nl (blog.linformatronics.nl)|82.95.251.247|:443... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Unable to establish SSL connection.


Notes




  • I am on Ubuntu Server 12.04.1 LTS


Answer



You have some serious firewall/NAT misconfigurations. You aren't actually running a web server on port 443...




$ telnet 82.95.251.247 443
Trying 82.95.251.247...
Connected to 82.95.251.247.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1


And your IPv6 service seems to be firewalled off...




$ telnet 2001:980:1b7f:1:a00:27ff:fea6:a2e7 443
Trying 2001:980:1b7f:1:a00:27ff:fea6:a2e7...
telnet: connect to address 2001:980:1b7f:1:a00:27ff:fea6:a2e7: Permission denied


Fix your firewall and/or service-on-the-wrong-port issues, and you should find things start working.


Sunday, May 28, 2017

iis 7 - IIS 7 ssl port problem

I am using ssl certificate for https://example.com/Pages/Membership/Registration.aspx and ı want to use ssl for subdomain https://host.example.com but IIS 7 error= port is using how can i bind sub domain 443 port

Saturday, May 27, 2017

performance - Using Rack servers and Blade servers in the same lab?




I'm working on pulling together a comprehensive performance testing lab. Part of this is the question of the physical lab hardware, so I'm learning about rack and blade servers and the positives and negatives of each.



To give a little background, it's a performance testing lab for database applications, so there are going to be around 8-10 machines in each lab rig. This includes things like a monitoring server to collect data during test runs, web servers, database server, and all the pieces needed to actually generate the load.




Because of the specifics of the company, we have the opportunity to re-purpose existing blade servers (Dell M610s) to fill some of these roles. Namely, the web servers and data collection machines.



What I'm unsure of is if there are any problems for using a mixture of rack servers and blade servers in the same lab (not in the same chassis/rackspace). Due to the requirements of our database servers (mainly the need for local attached storage) we are going to use rack servers for those.


Answer



No, there's no problems at all - so long as you're within your power and cooling budgets - and you're no more likely to encounter a network problem dealing with both blades and rack-mounters than if they were all one type or the other.


Friday, May 26, 2017

linux - Web Server Running Low in Memory



I have an EC2 small instance that has 2 gigs of memory, running Fedora Linux.



Typically, I get about 275 page views per day on average, and I have a monitoring agent on the box. Some of the stats are little worrying in terms of free memory.



Last week, at it's lowest point we had only 30 mb free of memory, this morning it appears to have increased to about 150 mb. Hyperic is our monitoring agent which runs Java. A top shows that its memory usages is only about 3.4%.




When I add up all of the httpd processes I get about 15-20% memory usage with mysql using about 1%.



Top doesn't reveal where the rest of the memory is going. What could I do to find out whats causing the high memory consumption? Could it be the 275 hits a day? About 95% of our code is PHP and HTML. MySQL is being used lightly from an application called OpenVBX which is only used internally.



If it's apache, an upgrade in memory should solve our issue right?



Any advice would be a huge help, thanks!



UPDATE: free shows this:




             total       used       free     shared    buffers     cached
Mem: 1718724 1575024 143700 0 234328 969284
-/+ buffers/cache: 371412 1347312
Swap: 917500 1140 916360

Answer



Basically you want to look at the free column of the buffers/cache row, so you have about 1.3G free -- you are probably in good shape.



The famous page for this "Help! Linux ate my Ram!" at http://www.linuxatemyram.com/ .




To put it very simply, Linux is going to use ram to cache files and the sort so it can deliver them faster. If a program really needs ram then it will take it from the cache -- but why not use the ram while it can?


redirect - How does URL redirection affect DDoS attacks?




We're moving our company website to a new domain and a new hosting provider as part of a re-branding project. We intend to create completely new content for the new website, remove all content from the old one, and only leave a generic URL redirect in the old domain, which directs to the main page on the new website.



While it is not a reason for the domain change, we had a lot of DDoS attacks on our old domain. I'd like to know if we can do something with URL redirect to reduce the amount of attacks on the old domain that get directed to the new domain?



A 301 redirect is the best option SEO-wise, but does it just relay any attacks to the new domain?



What about meta refresh? Or just a static anchor link to the new website? Are there other options?


Answer



On balance, I would probably go either with a 301, or replace the previous site with a single page providing a link to the new location.




Although I understand your concern about migrating the DDOS attacks along with legitimate traffic, I think it may be a false concern:




  • you do need to provide a means for legitimate users to find your new site, and that being the case, a sufficiently motivated attacker could too

  • you are putting up another public website, so even absent concerns about what was happening to the previous site, there is no particular reason I can think of to assume that site would not come under attack, potentially for the same, or similar, reasons to why your first domain was attacked. It is entirely possible your first domain was under attack for something as simple as 'it exists'.



Ultimately, I am not sure there is much you can do to avoid being attacked again.



However, that doesn't mean there isn't something you could do:





  • rate limit all requests to the old domain, if possible to something as close to 0 as possible (in theory, legitimate users make 1 request, get 1 redirect, and are no longer seen).

  • only redirect your landing page to your new site via 301. Keep other URLs (in particular, any URLs you know to be specifically targetted) without the redirect (you can always have a generic 'we moved' page), and gather any IPs that hit it - then consider blocking those IPs on both your new and old domains/servers.

  • similarly, consider using other methods (e.g. an IDS, a WAF) in front of your old domain's server to more specifically identify attackers, and block those where you can.

  • where you do have a redirect, consider checking user agents. It will be imperfect, but might be of some use in restricting who receives the redirect in the first place.

  • do all you can to ensure your new domain is more resilient than the old one, and use the attack data from your old domain to gauge what level of protection might be necessary.



Although I couldn't find anything specific to answer your question about the role of 301s in 'migrating' a DDoS, I strongly suspect the answer is 'it depends'.




Notably, it would depend on:




  • what kind of DDoS you are under - unless it is a layer 7 DDoS, a 301 isn't really relevant.

  • how sophisticated attacker tools are, and (possibly) how they are configured. I can easily imagine a 301 being used to redirect attackers to a less-visible server/site, so there may be reasons why scripts would ignore them - this is purely speculation on my part, though.


Thursday, May 25, 2017

Managing Linux Directory Permissions & SFTP

Good morning;
I have a RHEL 5.7 web server configured to allow SSH/SFTP only by specific groups. I'd like for content managers to upload content to their respective directories and have that content inherit the user/group ownership of the directory regardless of upload method or application. For example:





  • John is in group "web" for SSH/SFTP rights and "finance" for directory permissions, and uploads to directory "webstuff" via SFTP.

  • Directory "webstuff" has permissions of "2760" (rwxrws---), and ownership of "apache:finance".



If John uploads an update to an existing file in "webstuff", the ownership of the file stays at "apache:finance".
If John uploads a new file to "webstuff", the ownership of the file is "john:finance".



My desire is to have any file from John uploaded to "webstuff" to change to the directory's owner. I've tried with setuid and setgid both set, but the user-ownership didn't take.
I've seen mentions on ServerFault of using ACL's, or a chrooted jail for SFTP but I have yet to configure and test them, and I don't know if they're a viable solution (they could be, I just don't know because I've never done either). Any thoughts and assistance would be greatly appreciated.

Best practices for transcoding OPUS/G711 using freeswitch

Can someone share their experience of transcoding OPUS/G711 ans vice versa using Freeswitch? I am getting call quality issues even if there is a single call on the server. I am getting crackling noise and the end of the words.



SIP Clients HAVE to use the OPUS, it is a customers requirements and there is nothing negotiable on this front. However, for troubleshooting purposes, we requested the SIP clients to switch to G711u and all the test calls completed without any quality issues.



I have installed Freeswitch 1.6 on RHEL6 on a Dell PowerEdge R710 with 16 cores and 96GB RAM.



Call Flow:

SIP Client registered to a SIP Server dials a call to a PSTN number using the codec OPUS. SIP server proxy the call to Freeswitch with codec OPUS. Freeswitch terminates the call to SIP provider using G711u. Freeswitch does the transcoding between OPUS and G711u.



Silence Supression is turned off on both legs.
PTIME is 20 on both legs.



OPUS.CONF:
use-vbr=1
complexity=10
keep-fec-enabled=0
maxavgbitrate=0

maxplaybackrate=48000


Any suggestions would be much appreciated.

email - Connection Refused Mailing Problem



I'm trying to send emails through my server but it's not working.



I just setup this server and it's running ubuntu 10.04 lts




When I try to send an email it's telling me



Unable to connect to mail server: Connection refused(111)



I have no idea what to do.



Thanks :)


Answer



1) stop your firewall
2) post the output of "telnet localhost 25"

3) what mailserver are you using? post the conf file.



HTH...


Wednesday, May 24, 2017

How to protect saslauthd from local brute force attack?

There are numerous instructions over the internet to use saslauthd. I've tried to run the service. It gave me a surprise when I discovered that /run/saslauthd/mux socket and /usr/sbin/testsaslauthd are both available for non-privileged users. So when you have saslauthd started, it makes your system vulnerable.



What is the supposed way to restrict brute force? I've tried to google it, but google shows only SMTP and IMAP stuff, not saslauthd vulnerability itself.

freebsd - freenas disc problems - smart error - how to repair zfs raid?




My problem was that hdds of my zfs raid partly degraded and partly destroyed after a lightning.



I was able to detect the problem with zpool status:



zpool status myzfs
pool myzfs
state: DEGRADED (DESTROYED)


The good news. ZFS seems to be really reliable and I in my case was able to recover the raid fully. How I recovered see below in the answer.







  1. 1 failed drive brings a zpool down. An raid based on stiped mirrows however stays available. Details exlained in ZFS: You should use mirror vdevs, not RAIDZ

  2. Recovering and resilvering a
    zpool based on raidz2-0 takes a really long time.
    You may be better of with a striped mirror. This has pros and cons widely discussed in the internet sdfg

  3. A raid is NOT a backup! Offshoring backups into the cloud or a second location is an big advantage and is possible today without any big affords. Most Raids allow backups to the cloud or ZFS replication to another NAS






This however there not necessarily important to detect and solve the problem.



I have troubles with my freenas 9.2.1. It crashed today. Its running a fileserver on zfs jbod raid 2. I'm not sure what exactly causes the problems. The system is booting however reacting pretty slow. From the logs I couldn't figure anything totally wrong. Thus I'm not sure where to get startet with error analysis and how to solve them.



The problem is that the system crashes and responds pretty slow. The freenas web interface crashes as well since pyhon dies.



Freenas is installed on an usb stick, an additional drive (2tb) is attached for backup. The other 4 drives run as zfs raid.




The harddrives do show smart errors. How can I fix them? May they be the reason for the problems.



TOP



CPU:  0.1% user,  0.0% nice,  2.5% system,  0.1% interrupt, 97.3% idle
Mem: 131M Active, 11G Inact, 3689M Wired, 494M Cache, 3232M Buf, 16M Free
ARC: 3028K Total, 347K MFU, 1858K MRU, 16K Anon, 330K Header, 477K Other
Swap: 10G Total, 636K Used, 10G Free



DF



Filesystem                       Size    Used   Avail Capacity  Mounted on
/dev/ufs/FreeNASs2a 971M 866M 27M 97% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/md0 4.8M 3.5M 918k 79% /etc
/dev/md1 843k 2.6k 773k 0% /mnt
/dev/md2 156M 40M 103M 28% /var
/dev/ufs/FreeNASs4 20M 3.4M 15M 18% /data
fink-zfs01 6.0T 249k 6.0T 0% /mnt/fink-zfs01

fink-zfs01/.system 6.0T 249k 6.0T 0% /mnt/fink-zfs01/.system
fink-zfs01/.system/cores 6.0T 14M 6.0T 0% /mnt/fink-zfs01/.system/cores
fink-zfs01/.system/samba4 6.0T 862k 6.0T 0% /mnt/fink-zfs01/.system/samba4
fink-zfs01/.system/syslog 6.0T 2.7M 6.0T 0% /mnt/fink-zfs01/.system/syslog
fink-zfs01/shares 6.0T 261k 6.0T 0% /mnt/fink-zfs01/shares
fink-zfs01/shares/fink-privat 6.4T 344G 6.0T 5% /mnt/fink-zfs01/shares/fink-privat
fink-zfs01/shares/gf 6.0T 214k 6.0T 0% /mnt/fink-zfs01/shares/gf
fink-zfs01/shares/kundendaten 6.6T 563G 6.0T 9% /mnt/fink-zfs01/shares/kundendaten
fink-zfs01/shares/zubehoer 6.6T 539G 6.0T 8% /mnt/fink-zfs01/shares/zubehoer
fink-zfs01/temp 6.2T 106G 6.0T 2% /mnt/fink-zfs01/temp

/dev/ufs/Backup 1.9T 114G 1.7T 6% /mnt/Backup


/var/log/messages



Jan 21 21:48:32 s-FreeNAS root: /etc/rc: WARNING: failed to start syslogd
Jan 21 21:48:32 s-FreeNAS kernel: .
Jan 21 21:48:32 s-FreeNAS root: /etc/rc: WARNING: failed to start watchdogd
Jan 21 21:48:32 s-FreeNAS root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Jan 21 21:48:34 s-FreeNAS ntpd[2589]: ntpd 4.2.4p5-a (1)

Jan 21 21:48:34 s-FreeNAS kernel: .
Jan 21 21:48:36 s-FreeNAS generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint,name
Jan 21 21:48:36 s-FreeNAS generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: zfs list -H -o mountpoint
Jan 21 21:48:38 s-FreeNAS last message repeated 4 times
Jan 21 21:48:38 s-FreeNAS generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /usr/local/bin/pdbedit -d 0 -i smbpasswd:/tmp/tmpEKKZ2A -e tdbsam:/var/etc/private/passdb.tdb -s /usr/local/etc/smb4.conf
Jan 21 21:48:43 s-FreeNAS ntpd[2590]: time reset -0.194758 s
Jan 21 21:48:45 s-FreeNAS smartd[2867]: Device: /dev/ada3, FAILED SMART self-check. BACK UP DATA NOW!
Jan 21 21:48:45 s-FreeNAS smartd[2867]: Device: /dev/ada3, 164 Currently unreadable (pending) sectors
Jan 21 21:48:45 s-FreeNAS smartd[2867]: Device: /dev/ada3, Failed SMART usage Attribute: 5 Reallocated_Sector_Ct.
Jan 21 21:48:45 s-FreeNAS smartd[2867]: Device: /dev/ada3, previous self-test completed with error (unknown test element)

Jan 21 21:48:51 s-FreeNAS mDNSResponder: mDNSResponder (Engineering Build) (Mar 1 2014 18:12:24) starting
Jan 21 21:48:51 s-FreeNAS mDNSResponder: 8: Listening for incoming Unix Domain Socket client requests
Jan 21 21:48:51 s-FreeNAS mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
Jan 21 21:48:51 s-FreeNAS mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
Jan 21 21:48:53 s-FreeNAS netatalk[3142]: Netatalk AFP server starting
Jan 21 21:48:53 s-FreeNAS cnid_metad[3179]: CNID Server listening on localhost:4700
Jan 21 21:48:53 s-FreeNAS kernel: done.
Jan 21 21:48:54 s-FreeNAS mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000800C2FD60 s-FreeNAS.local. (Addr) that's already in the list
...
Jan 21 21:48:54 s-FreeNAS mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000800C30180 109.1.1.10.in-addr.arpa. (PTR) that's already in the list


Jan 21 22:04:44 s-FreeNAS kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1572950, size: 8192
...
Jan 21 22:05:25 s-FreeNAS kernel: GEOM_ELI: g_eli_read_done() failed ada0p1.eli[READ(offset=110592, length=4096)]
Jan 21 22:05:25 s-FreeNAS kernel: swap_pager: I/O error - pagein failed; blkno 1572894,size 4096, error 5
Jan 21 22:05:25 s-FreeNAS kernel: vm_fault: pager read error, pid 3020 (python2.7)
Jan 21 22:05:25 s-FreeNAS kernel: Failed to write core file for process python2.7 (error 14)
...
Jan 21 22:19:44 s-FreeNAS kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 70 02 00 40 00 00 00 00 00 00
Jan 21 22:19:44 s-FreeNAS kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error

Jan 21 22:19:44 s-FreeNAS kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jan 21 22:19:44 s-FreeNAS kernel: (ada0:ahcich0:0:0:0): RES: 41 40 70 02 00 40 00 00 00 00 00
Jan 21 22:19:44 s-FreeNAS kernel: (ada0:ahcich0:0:0:0): Error 5, Retries exhausted
Jan 21 22:19:44 s-FreeNAS kernel: GEOM_ELI: g_eli_read_done() failed ada0p1.eli[READ(offset=253952, length=4096)]
Jan 21 22:19:44 s-FreeNAS kernel: swap_pager: I/O error - pagein failed; blkno 1572929,size 4096, error 5
Jan 21 22:19:44 s-FreeNAS kernel: vm_fault: pager read error, pid 2869 (smartd)
Jan 21 22:19:44 s-FreeNAS kernel: Failed to write core file for process smartd (error 14)
Jan 21 22:19:44 s-FreeNAS kernel: pid 2869 (smartd), uid 0: exited on signal 11



smartctl --scan



/dev/ada0 -d atacam # /dev/ada0, ATA device
/dev/ada1 -d atacam # /dev/ada1, ATA device
/dev/ada2 -d atacam # /dev/ada2, ATA device
/dev/pass3 -d atacam # /dev/pass3, ATA device
/dev/ada3 -d atacam # /dev/ada3, ATA device
/dev/ada4 -d atacam # /dev/ada4, ATA device
/dev/ada5 -d atacam # /dev/ada5, ATA device



smartctl -a /dev/ada3



smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD4000F9YZ-09N20L0
Serial Number: WD-WMC1F1211607
LU WWN Device Id: 5 0014ee 0ae5c0b4c

Firmware Version: 01.01A01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Jan 21 23:07:55 2015 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
See vendor-specific Attribute list for failed Attributes.

General SMART Values:
Offline data collection status: (0x85) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 73) The previous self-test completed having

a test element that failed and the test
element that failed is not known.
Total time to complete Offline
data collection: (41640) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.

Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.

Extended self-test routine
recommended polling time: ( 451) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 187 187 051 Pre-fail Always - 553
3 Spin_Up_Time 0x0027 142 138 021 Pre-fail Always - 11900
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 93
5 Reallocated_Sector_Ct 0x0033 139 139 140 Pre-fail Always FAILING_NOW 1791
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 090 090 000 Old_age Always - 7553
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0

12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 93
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 59
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 35
194 Temperature_Celsius 0x0022 108 098 000 Old_age Always - 44
196 Reallocated_Event_Count 0x0032 001 001 000 Old_age Always - 353
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 162
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0


SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: unknown failure 90% 7553 -
# 2 Short offline Completed: unknown failure 90% 7552 -
# 3 Short offline Completed: unknown failure 90% 7551 -
# 4 Short offline Completed: unknown failure 90% 7550 -

# 5 Short offline Completed: unknown failure 90% 7549 -
# 6 Short offline Completed: unknown failure 90% 7548 -
# 7 Short offline Completed: unknown failure 90% 7547 -
# 8 Short offline Completed: unknown failure 90% 7546 -
# 9 Short offline Completed: unknown failure 90% 7545 -
#10 Short offline Completed: unknown failure 90% 7544 -
#11 Short offline Completed: unknown failure 90% 7543 -
#12 Short offline Completed: unknown failure 90% 7542 -
#13 Short offline Completed without error 00% 7541 -
#14 Short offline Completed without error 00% 7540 -

#15 Short offline Completed: read failure 10% 7538 1148054536
#16 Short offline Completed: read failure 10% 7538 1148054536
#17 Short offline Completed: read failure 10% 7536 1148057328
#18 Short offline Completed: read failure 10% 7535 1148057328
#19 Short offline Completed without error 00% 7530 -
#20 Short offline Completed without error 00% 7529 -
#21 Short offline Completed: read failure 10% 7528 1148057328

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS

1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Answer




DISCLAIMER:



Repairing a destroyed ZFS Pool and replacing degraded or unavailable process is dangerous. For me it worked. However read all documentations carefully and read about typical failures. Otherwise you may destroy your raid fully. You may'd like to contact a professional helping you recovering your data.



However reading carefully you should be able to recover your zfs raid yourself!



Situation



Further research showed that we had current peaks from an lightning which destroyed the hard drives and corrupted others. Thus the pool zfs pool was defect.




This can be detected using zpool status:



zpool status myzfs
pool myzfs
state: DEGRADED (DESTROYED)


Recovering destroyed ZFS Storage Pool



This problem may be (at least for me) be solved through:




zpool destroy myzfs
zpool import -Df # this made the zpool accessible again


zpool however continued to be degraded in reason of 1 drive being fully destroyed.



Full documentation on recovering destroyed zfs pools see oracle documentation on recovering destroyed zfs storage pools



Replacing degraded ZFS drive




Recovering the degraded zpool did not fully solve the problem since it still had degraded/defect drives



zpool -status myzfs
config:



    NAME         STATE     READ WRITE CKSUM
myzfs DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/uuid1 ONLINE 0 0 0

gptid/uuid2 ONLINE 0 0 0
gptid/uuid3 ONLINE 0 0 0
778923478919345 UNAVAIL 0 0 0 /was /dev/ada4


Drive replacement is also documented very well in the internet. However It depends a little bit if you are using a hotspare or not, the raid level, ...



Basically it worked for me pretty simple though



zpool replace myzfs 778923478919345



The replacement is also documented very well by oracle under Replacing a Device in a ZFS Storage Pool


Tuesday, May 23, 2017

domain name system - Is Round-Robin DNS "good enough" for load balancing static content?




We have a set of shared, static content that we serve up between our websites at http://sstatic.net. Unfortunately, this content is not currently load balanced at all -- it's served from a single server. If that server has problems, all the sites that rely on it are effectively down because the shared resources are essential shared javascript libraries and images.



We are looking at ways to load balance the static content on this server, to avoid the single server dependency.



I realize that round-robin DNS is, at best, a low end (some might even say ghetto) solution, but I can't help wondering -- is round robin DNS a "good enough" solution for basic load balancing of static content?



There is some discussion of this in the [dns] [load-balancing] tags, and I've read through some great posts on the topic.



I am aware of the common downsides of DNS load balancing through multiple round-robin A records:





  • there's typically no heartbeats or failure detection with DNS records, so if a given server in the rotation goes down, its A record must manually be removed from the DNS entries

  • the time to live (TTL) must necessarily be set quite low for this to work at all, since DNS entries are cached aggressively throughout the internet

  • the client computers are responsible for seeing that there are multiple A records and picking the correct one



But, is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content? Or is DNS round robin pretty much worthless under any circumstances?


Answer



Jeff, I disagree, load balancing does not imply redundancy, it's quite the opposite in fact. The more servers you have, the more likely you'll have a failure at a given instant. That's why redundancy IS mandatory when doing load balancing, but unfortunately there are a lot of solutions which only provide load balancing without performing any health check, resulting in a less reliable service.




DNS roundrobin is excellent to increase capacity, by distributing the load across multiple points (potentially geographically distributed). But it does not provide fail-over. You must first describe what type of failure you are trying to cover. A server failure must be covered locally using a standard IP address takeover mechanism (VRRP, CARP, ...). A switch failure is covered by resilient links on the server to two switches. A WAN link failure can be covered by a multi-link setup between you and your provider, using either a routing protocol or a layer2 solution (eg: multi-link PPP). A site failure should be covered by BGP : your IP addresses are replicated over multiple sites and you announce them to the net only where they are available.



From your question, it seems that you only need to provide a server fail-over solution, which is the easiest solution since it does not involve any hardware nor contract with any ISP. You just have to setup the appropriate software on your server for that, and it's by far the cheapest and most reliable solution.



You asked "what if an haproxy machine fails ?". It's the same. All people I know who use haproxy for load balancing and high availability have two machines and run either ucarp, keepalived or heartbeat on them to ensure that one of them is always available.



Hoping this helps!


Slab reclaimable memory is not freed when needed

Correct me if I am wrong, but to my understanding slab reclaimable holds cached kernel objects which can be freed if needed. So if application needs to allocate more space, even if the 'free' memory is low, OS will drop some pages from slab reclaimable and privide application with the requested amount of memory (unless its not possible).



This is how my memory looks:
Mem graph
and /proc/meminfo output:



MemTotal:        8171852 kB
MemFree: 825892 kB
MemAvailable: 6273852 kB

Buffers: 227448 kB
Cached: 1261944 kB
SwapCached: 15324 kB
Active: 2582260 kB
Inactive: 499232 kB
Active(anon): 1460764 kB
Inactive(anon): 131340 kB
Active(file): 1121496 kB
Inactive(file): 367892 kB
Unevictable: 32 kB

Mlocked: 32 kB
SwapTotal: 524284 kB
SwapFree: 440372 kB
Dirty: 372 kB
Writeback: 0 kB
AnonPages: 1579556 kB
Mapped: 40500 kB
Shmem: 4 kB
Slab: 4113080 kB
SReclaimable: 4061308 kB

SUnreclaim: 51772 kB
KernelStack: 6992 kB
PageTables: 70692 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4610208 kB
Committed_AS: 2644508 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB

VmallocChunk: 0 kB
DirectMap4k: 14200 kB
DirectMap2M: 2082816 kB
DirectMap1G: 8388608 kB


First thing I noticed is that the slab and cache are the exact copy of memory used, meaning is contant.



To the problem:




Sometimes when free memory reaches values around 100 Mb, OOM-killer is invoked, killing vital processes (php, clamd, ...). How is that possible? Shouldnt OS free slab reclaimable before invoking OOM?



Things I tried



I tried setting



vm.vfs_cache_pressure=10000


thinking it will force kernel to drop more caches, but the graph didnt change, even after 24H.




Perhaps its a bug in kernel itself https://bugzilla.kernel.org/buglist.cgi?quicksearch=oom&list_id=904801

ssl - Apache mod_proxy: forward secure websocket to non-secure



The websocket library I rely on (PHP-Websockets) does not yet support secure sockets (wss). My website is served over https though, so I cannot use insecure ws connections.



I'm trying to use Apache's mod_proxy to forward the secure request that comes from the browser on to the client.



Javascript




var ws = new Websocket('wss://example.com/_ws_/');


Apache VirtualHosts



ProxyPass        "/_ws_/" "ws://127.0.0.1:8080/"
ProxyPassReverse "/_ws_/" "ws://127.0.0.1:8080/"
# I've also tried "/_ws_/" "ws://example.com:8080/" Same error below



Upon trying to connect, the browser receives a 500 Server Error. The logs show:




No protocol handler was valid for the URL /_ws_/. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.




I've confirmed that if I remove the proxy rules, stop redirecting users to https and try to connect to insecure sockets from the browser: new Websocket('ws://example.com/'), things work just fine.



My loaded Apache modules include mod_ssl.c, mod_proxy.c and mod_proxy_http.c




Apache 2.4


Answer



To get things to work, I also needed to load mod_proxy_wstunnel



Once I did, this ruleset got things working: (in the VirtualHost of the domain that receives and proxies the websocket request)




ProxyPass "/_ws_/" "ws://127.0.0.1:8080/"
ProxyPassReverse "/_ws_/" "ws://127.0.0.1:8080/"




The browser can then contact the backend WS server through the HTTPS port:



var ws = new Websocket('wss://example.com/_ws_/');

Monday, May 22, 2017

linux - Unable to visit virtual host over port 443

I have a Google Compute instance (Debian 8 and Apache) running using Google Compute protocol forwarding to be able to use multiple external IP addresses with one virtual machine instance:





Google Compute Protocol Forwarding (full article)



Google Compute Engine supports Protocol Forwarding, which lets you
create forwarding rule objects that can send packets to a non-NAT’ed
target instance. Each target instance contains a single virtual
machine instance that receives and handles traffic from the
corresponding forwarding rules.





The Problem



I am not able to visit one of the virtual host domains/IP's over port 443. I need this, because I want to enable HTTPS on all these domains.



I have tried multiple options to achieve opening port 443, but none of them works:



Option 1) Virtual name-based hosts:



/etc/apache2/ports.conf




Listen 80


NameVirtualHost *:443
Listen 443



NameVirtualHost *:443

Listen 443



/etc/apache2/sites-enabled/example.com.conf




ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/website1




ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/website1



Option 2) Virtual IP-based hosts:




/etc/apache2/ports.conf



Listen 80


NameVirtualHost *:443
Listen 443




NameVirtualHost *:443
Listen 443



DocumentRoot /var/www/website1

Require all granted




...


Both options work for port 80, but not for port 443.

How to setup fake raid for vmware virtual machines?



I have a vmware esxi 5 running on a server with ssd and two hard drives. The server comes with software lsi megaraid and intel rapid storage raid. Neither of them work with esxi.



I need to somehow setup a fakeraid inside virtual machine to keep a copy data on each drive (raid0).




Is it possible to do so?


Answer



ESXi is not supporting fake raid. You need to create 2 VMFS and add 2 disk each on one of the data stores and setup software RAID on the guest OS. You will have redundancy for VM, but not for ESX itself.


Sunday, May 21, 2017

raid - Equallogic partitioning: pools, allocations, etc



I have some new budget coming up to buy an Equallogic PS6110xv. This unit has 24x600G disks and I'm looking at different ways of setting it up.



The two primary uses will be VMs and MySQL.




MySQL: currently uses about 2T on an older SAN. Expected 3y growth would be an additional 2T. Seems like I should create a 4T partition and be done with it.



VMs: Im consolidating about a dozen retired machines into VMs using two hosts. This is where I'm not fully clear on how much space to allocate and how. Do I create one block with all the expected space or just create space for the basics and then mount additional space from the SAN on an as-needed basis? IE create a volume with 50g x 12 for the OS and then another with 5-6T as a 'pool'? I could mount 'thin' volumes on this as needed.



Last: If I'm using RAID-6 is there any advantage to making the whole device into one raid group? Seems like having that many disks to read for a rebuild is asking for trouble. On the other hand, having multiple smaller groups implies a less efficient use of the media.



Just wondering what the conventional wisdom is.



EDIT: (found this post in a VM group):





No, you can't separate the disks in one array into multiple storage groups or raid levels. Here's a quick cheat sheet on EqualLogic architecture/terminology:



Group - a collection of physical arrays that can be accessed as one resource. They are all managed at the same IP and share the same iSCSI target IP's.



Group member - an entire physical array. The entire (usually 16 spindle) array is essentially one big virtual disk. For RAID-10, by default you have two hot spares which means you have a 14-spindle RAID-10 array. On our PS6000XV with 15K 450GB SAS drives we got about 2.72 TB of usable space with RAID-10.



Storage Pool - a way to logically divide members and volumes within a group. However, both group members and volumes can only be part of one storage pool at a time, meaning you can't take a PS6000XV and dedicate 7 spindles to one storage pool and 7 spindles to the other.




The architecture does seem a bit inflexible on the surface, and it is - at least compared to traditional arrays where you can make as many RAID groups as you want within a single enclosure. Keep in mind that EqualLogic is really good for two things - ease of setup and scaling out.



Ease of setup - With most traditional iSCSI SANs, you have a main enclosure which houses some disks and the controllers. If you need more capacity, you add another enclosure, which usually means bringing the SAN down, adding your enclosure, reconfiguring the array controllers, and finally bringing the array back up. With EQL you just plug it in, enter the groups network information and authentication credentials, and it comes online as a member of your group with no downtime.



Scale out - Again, with traditional iSCSI SANs when you add an enclosure you are essentially doubling (or tripling) the number of spindles that your controllers are responsible for. With EQL, as you add capacity you are also adding dedicated controllers. The marketing people will tell you that this leads to more linear scaling of throughput and latency as you add capacity.




So this implies that I will have one 24 member RAID-6 group. Or 23 with a hot spare...


Answer



ethrbunny,




The VMware community post you mentioned in your edit is a bit outdated... Dell Equallogic recommendations for RAID policy are laid out in the Choosing a Member RAID Policy document though.



With 24x600GB disks, it's presumable you'll be using either 10K or 15K SAS drives. If that's the case, RAID 6, 10, and 50 are the recommended options. In accordance with what tim mentioned, an Equallogic storage array with 24 drives only lets you chose one RAID type/policy for an enclosure; the actual RAID sets are managed by the array on the back-end with no real user visibility through the GUI. Due to the number of drives, you're correct in being wary of having that many drives in a single RAID set, which is why the array splits them into two separate RAID sets (which is both better for data protection/redundancy and performance).



RAID6 is by far the best RAID policy to select for data protection, assuming that you have the system in warranty and drive failures will be dealt with promptly rather than ignored. The aforementioned document details the statistical likelihood of data loss between the different policies available, and RAID6 is a clear win by this measure.



Performance-wise, RAID6 suffers greatly with random writes in comparison to RAID10. It also experiences a greater performance impact during a failure/rebuild in comparison (though this is almost entirely negated with the copy-to-spare operation introduced with new firmware revisions for handling of preemptive failures).



If your current storage solution incorporates 16 or fewer drives of the same or lesser speed, I would nearly guarantee that a RAID6 policy would provide ample performance and IOPS for your needs in addition to the best capacity and protection level you can get on that array.




However, you could also consider setting up all of your volumes with thin provisioning, and allocate a max capacity for each volume that give you plenty of room to grow (even if that means over allocating to some degree). Start with a RAID10, get your full production environment in place, and then use the SAN Headquarters software provided by Equallogic to measure your performance (feel free to contact support or a technical sales rep for more info on this - they're usually very helpful). If your IOPS on individual drives is sitting below 100 even at your peak utilization, then you can easily get away with converting to a RAID6 to gain some extra capacity. The catch to this is that you cannot convert back from a RAID6 to a RAID10 without performing a factory reset on the array (which is only realistic in large multi-member environments) so make sure to do your research before making the switch.



Summary
RAID type recommendation: RAID 6 (verify this w/ Dell after having your capacity needs evaluated)
Volumes: 4TB volume for your database + 3+ volumes (perhaps 2TB in size) for VMFS datastores (multiples recommended for various performance reasons), all with thin provisioning enabled



Note 1: RAID10 on this array would give you just under 6TiB of actual usable capacity, while RAID6 would give you just over 10TiB (possibly a touch lower for each after space taken up by array's metadata)



Note 2: These recommendations are all assuming you don't plan to make much use of the replication or snapshot features. If you do, you'll need to take the additional space requirements into consideration as well (making RAID6 an even more favorable option)



hp - SmartArray P400 Create Logical Drive Erase Data?




Does create logical drive on a Smart Array P400 erase existing data?



I have a disk that was from a RAID1 set on a IBM server. It's currently plugged in a HP server in order for me to read the data. Since P400 does not have JBOD feature, I am wondering if I just need to create a logical drive (e.g. RAID0) in order for the OS to recognize it.



Thank you!


Answer



This will not work.



What are you expecting this to do? The IBM controller and the HP Smart Array controller have different on-disk RAID formats. As there is no passthrough or JBOD mode available on the HP controller, creating any logical drive, even a RAID0, will delete the existing data on the drive.


anti virus - Does anyone have references/metrics on the value of installing antivirus software - especially on servers?

I strongly dislike antivirus software. In my opinion, the av software behaves much like a virus. The recent Symantec incident of actually causing server crashes, resource use, software interference, and user safety bias are each very problematic.



If I have locked down servers behind a firewall, with admins following security protocol (no surfing, no downloads, etc.). What benefit would I have from installing antivirus software on these machines? I must install something for insurance purposes...



When i researched AV products a few years ago, the coverage was 95% at best - and these are of known security issues. That means that the best AV protection is vulnerable to thousands of known viruses and worms.




Every single infection I have encountered has been on a machine with AV software on it. The user always says - but I have antivirus software...



Can anyone provide metrics on the utility of av software on servers that will make me feel better about having to do it?

security - Is it a bad idea to give our COO a copy of the production db?



We're a small company with only a handful of employees. Our non-technical COO has asked that we setup a copy of the production database for him to use with a service like Chartio, because our (admittingly poor) admin dashboard doesn't meet his needs.



The COO wants to use a friend (a contractor) to help integrate the Chartio service with our database.



All our passwords are encrypted, but emails and names are stored in clear text.
We handle payment processing through a 3rd party provider so we don't store any sensitive financial information except transaction totals.




Is this a bad idea, or am I being overly cautious?



PS: We'd be setting up the database server ourselves, so we'd have control over the firewall and such.


Answer



In the end, the company directors are there to make "big picture" decisions on behalf of the company, so I don't think I'd be prepared to put my foot down about doing something that's merely unwise (though I wouldn't do something I believed to be illegal - in many jurisdictions, "following superior orders" won't protect you against the consequences of breaking the law).



If it were me, I'd want him to instruct me in writing to give him the copy, and preferably in that instruction to confirm that he was aware of my concerns about that database leaving the company, but was instructing me to proceed nonetheless.


Saturday, May 20, 2017

Should I use SW RAID / setting up RAID on HP Proliant server with Ubuntu



I have looked at a number of messages regarding software vs. hardware RAID here on SF but can't figure out which one I should use. The server I'm installing is HP Proliant ML330 G6 and the RAID controller on that is HP Smart Array B110i SATA RAID Controller (RAID 0/1/0+1). Is this a real HW RAID or should I just skip it and use the SW RAID?



The box has three disks, one 250GB and two 1.5TB disks. The smaller one I thought will have the /boot, / and /swap partitions and the two bigger ones will be on RAID1. How do I actually set this up? I tried using the tools that came with the server (StartSmart, as well as the ROM configuration utility) and configured the two disks as a RAID array, but the Ubuntun installer still sees them as two separate disks.



Should I just first install Ubuntu on the small disk and then configure the RAID array or should I somehow be able to configure the RAID array so that the two disks are shown as one to the installer?



Answer



Unless you enabled the battery backed cache and have a cache at all (not all HP SmartArrays come with that anymore, which boggles the mind) then I don't see much difference between the two. Software RAID is ever so slightly more fragile in extremity, but for nearly all running should perform just as fast. If not faster.


PEBKAC failure on Raid 1 array (striped) installation

Some time ago, I attempted to upgrade a RAID 1 array to a RAID 10 array by adding two more disks to the original two that lived in an IBM xSeries 346. In the RAID card's bios, I told it to grow the array onto the new disks, and for whatever brain-glitch reason I checked the "striped" box. This just striped the old RAID 1 array across the new disks without upgrading it to RAID 10. The RAID card reports that the RAID level is "1E".



Naturally, I was quite dismayed to find that the array was no bigger than when I started.



Is there any way I can fix this? Can I pull the new drives (with the intention of formatting them and starting the upgrade again) without destroying the array?

centos - What is Slab, SReclaimable and SUnreclaim? What does Slab solve and how does it behave under load?

I have a server with 32 gigabytes of ram, with htop reporting 29 gigs used, with 4 gigs of swap in use as well.



cat /proc/meminfo returns around 13 gigabytes of memory listed as SReclaiable.




What is Slab? When was it added? What does it mean it is reclaimable, when does this happen? Why am I still stepping onto swap space if there's 13 gigs free?



I've found the slab allocator docs to be WAY too Unixy for someone just looking for an answer, so I'm hoping someone could provide a reasonable rundown of what it is and how it behaves.



By the way, it's a CentOS 3.10.0-123.13.2.el7.x86_64 in this case.



PS. I realize the question may be imperfect in many ways, but I'd love for a practical guide to slab surface somewhere. Maybe we could start here?

ubuntu 12.04 - Run fail2ban without banning



It is possible to run fail2ban in some kind of "simulation mode" so it does not ban but log somewhere who it would have ban?



Running fail2ban on Ubuntu 12.04.


Answer




If you look in jail.conf you'll see a line that says this:



# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overriden globally or per
# section within jail.local file
banaction = iptables-multiport


All the actions are in /etc/action.d/




You could make one that just sends mail.. there are quite a few actions there already that may do what you want.


Friday, May 19, 2017

networking - Best practice on forwarding traffic to VMs inside ESXI based on domain name requested?

Lets say I have a server with two ip addresses. There are 100 VMs on this server each that hosts a website whose domain names use one of the two IPs in their A records.




What's the best way to route a request coming into ESXI to the appropriate VM depending on which domain name the end user visited?

smart - Is my Hard Drive Failing?



I just tried to run a test on my hdd and it doesn't want to complete a self test. Here is the result:



smartctl --attributes --log=selftest /dev/sda
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)

=== START OF READ SMART DATA SECTION ===

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 697
3 Spin_Up_Time 0x0027 206 160 021 Pre-fail Always - 691
4 Start_Stop_Count 0x0032 074 074 000 Old_age Always - 26734
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 28
9 Power_On_Hours 0x0032 090 090 000 Old_age Always - 7432
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0

11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 097 097 000 Old_age Always - 3186
191 G-Sense_Error_Rate 0x0032 001 001 000 Old_age Always - 20473
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 84
193 Load_Cycle_Count 0x0032 051 051 000 Old_age Always - 447630
194 Temperature_Celsius 0x0022 113 099 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 16
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0

200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: read failure 90% 7432 92290592
# 2 Conveyance offline Completed: read failure 90% 7432 92290596
# 3 Conveyance offline Completed: read failure 90% 7432 92290592
# 4 Short offline Completed: read failure 90% 7431 92290596
# 5 Extended offline Completed: read failure 90% 7431 92290592



So is this disk failing?


Answer



Your drive is very happy to do a self-test; from the summary, it has done more than five of them in the past hour. And all of them have failed, early on in the test, with read errors.



Yes, this hard drive is failing. As the famous Google Labs report said (though I can't put my hand on a link to it at the moment), if smartctl says your drive is failing, it probably is (I paraphrase).



Edit: don't try to save it. Get all the data off it, and replace it.


Thursday, May 18, 2017

fedora - Could someone point me at some basic "setting up linux at home type" docs?

I've spent a fair amount of time using various *nix systems, but don't have a lot of experience in setting them up from scratch (unlike Windows, which I can more or less do in my sleep at this point, much to my dismay - I keep waking up and discovering I've set up a new box. Anyway.)



Looking to shore up this hole in my education, I've been playing around with getting a linux machine up and running on my home LAN of late. (Using Fedora 10 due to some advice I got out in meatspace.) While the install went fine, its that first batch of configuration that has me somewhat puzzled. The usual combo of google and sweat has got the thing more or less working, but are they any decent tutorials out there for someone who kinda knows what they're doing in the big picture, but doesn't know what to do with his Linux box on day 2?




Edited to add: wow! This is all great stuff! (Keep it coming?) Thanks, y'all!

Wednesday, May 17, 2017

How do I keep MySQL from ever increasing it's disk space usage when used with puppet-dashboard?



The setup



We have a Debian Linux set up with MySQL v5.1.73 (innoDB storage engine) and puppet-dashboard version 1.2.23. As you probably guessed, puppet-dashboard is using MySQL as its backend.



Also, it shouldn't be relevant but this a VMware virtual machine on vSphere 5.5.




The problem



The problem is that, despite the number of puppet nodes and run frequency staying relatively the same, the disk space used by MySQL keeps on increasing in a disturbing fashion to the point where it is now becoming an issue.



The following graph illustrates the issue.



disk space goes down



We have put in place the two cron jobs that should allow disk space to be freed. They are the following and both run daily :





  • rake RAILS_ENV=production db:raw:optimize

  • rake RAILS_ENV=production reports:prune:orphaned upto=3 unit=mon



The drops you can see in the graph are the cron jobs running and eating up more space trying to free some space.



MySQL binary logs are not enabled. 95% of the disk space used on this server is located in the /var/lib/mysql/dashboard_production which is the directory where the MySQL data is stored.




We have had this issue before with a different application (Zabbix monitoring) and had to dump the DB and re-import in order to free up space. This was a very painful process and not a very elegant solution but it ultimately worked.



Is there any way we can reclaim this disk space ? What we can do this stop this behavior ?



Edit 1



We are indeed using innoDB and we are not using configuration directive "innodb_file_per_table".



As requested by Felix, the output of the command is the following :




+----------------------+-------------------+-------------+
| table_schema | table_name | data_length |
+----------------------+-------------------+-------------+
| dashboard_production | resource_statuses | 39730544640 |
| dashboard_production | metrics | 643825664 |
| dashboard_production | report_logs | 448675840 |
| dashboard_production | timeline_events | 65634304 |
| dashboard_production | reports | 50937856 |
| dashboard_production | resource_events | 38338560 |
| glpidb | glpi_crontasklogs | 21204608 |

| ocsweb | softwares | 8912896 |
| ocsweb | deploy | 5044208 |
| phpipam | logs | 1269584 |
+----------------------+-------------------+-------------+


Also, I will be trying the reports:prune task without the "orphaned" option as mentionned as well as the other alternatives and will keep this question updated.



Edit 2




I ran the reports:prune rake task and, despite deleting 230000 reports, it kept on eating more space... I will therefore move on to the other options.



enter image description here



The solution



After deleting two thirds of the entries in the database, it only freed up 200MB of disk space which is senseless. We ended up dumping the content and re-importing it taking care to enable "innodb_file_per_table".



We will just have to wait and see if this fixes the solution in the long term but it seems to be the case for the moment.


Answer




I found this article which seems to address the issue pretty well



http://ximunix.blogspot.co.uk/2014/01/howto-cleanup-puppet-reports-and-db.html



posted by Ximena Cardinali



The short story is start deleting reports in small batches and then reclaim the space from MySQL







HOWTO Cleanup Puppet Reports and DB



If the database for Puppet Dashboard is using several GB and getting larger everyday, this is a way to get some of the space back.



There are two rake jobs you should be running everyday as part of daily maintenance for Puppet Dashboard.



cd /usr/share/puppet-dashboard
env RAILS_ENV=production rake reports:prune upto=5 unit=day
env RAILS_ENV=production rake reports:prune:orphaned



You can change the RAILS_ENV and number of day (day), weeks (wk), months (mon), etc to match your system and its needs.




  1. Stop incoming reports:



    cd /path/to/puppet-dashboard



    env RAILS_ENV=production script/delayed_job -p dashboard -m stop


  2. Start deleting reports in small batches





Keep working your way in towards the length of time you want to keep reports for. The reason for this is Innodb tables have poor performance when deleting more than 10k rows at a time. If you try to deleting a few hundred thousand rows, it will timeout and you'll have to break it up into smaller deletes anyway. Also the Ruby rake process will use probably use all your RAM and likely get killed off by the kernel before it finishes. Something like this progression should work for most people, but if you have many months of data you may want to start with a month or two of your earliest records. In our case, we are keeping just 2 weeks reports (14 days).



env RAILS_ENV=production rake reports:prune upto=6 unit=mon
env RAILS_ENV=production rake reports:prune upto=4 unit=mon
env RAILS_ENV=production rake reports:prune upto=2 unit=mon
env RAILS_ENV=production rake reports:prune upto=3 unit=wk
env RAILS_ENV=production rake reports:prune upto=1 unit=wk
env RAILS_ENV=production rake reports:prune upto=5 unit=day




  1. Determine the best method to reclaim space from MySQL



There are two methods to reclaim space depending on how MySQL was configured. Run this command to determine if "innodb_file_per_table" is enabled. It should be set to "ON" if it is.
NOTE: I recommend to use innodb on your MySQL for cases like this one.



mysqladmin variables -u root -p | grep innodb_file_per_table



You can also do a listing of the database to see if there are larger data files. The table most likely to be large is resource_statuses.ibd.



ls -lah /var/lib/mysql/dashboard_production
...
-rw-rw---- 1 mysql mysql 8.9K Jan 08 12:50 resource_statuses.frm
-rw-rw---- 1 mysql mysql 15G Jan 08 12:50 resource_statuses.ibd
...




  1. Reclaiming space the easy way



If MySQL was configured with innodb_file_per_table and your Dashoard DB shows that your data is in large table files, do the following:



mysql -u root -p
use puppet_dashboard;
OPTIMIZE TABLE resource_statuses;



This will create a new table based on the current data and copy it into place. If you do a listing while this is in progress you should see something like this:



-rw-rw---- 1 mysql mysql       8.9K Jan  08 12:50 resource_statuses.frm
-rw-rw---- 1 mysql mysql 15G Jan 08 12:50 resource_statuses.ibd
-rw-rw---- 1 mysql mysql 8.9K Jan 08 12:50 #sql-379_415.frm
-rw-rw---- 1 mysql mysql 238M Jan 08 12:51 #sql-379_415.ibd



And when it finished it'll copy the tmp file into place. In this case we went from 15GB to 708MB.



-rw-rw---- 1 mysql mysql 8.9K Jan 08 13:01 resource_statuses.frm
-rw-rw---- 1 mysql mysql 708M Jan 08 13:03 resource_statuses.ibd



  1. Reclaiming space the hard way




If your system was not configured with innodb_file_per_table or all the current data resides in a large ibdata file, the only way to reclaim space is to wipe the entire installation and reimport all the data.
The overall method should be something like: First configure innodb_file_per_table, dump all the databases, then stop Mysql, delete /var/lib/mysql, run mysql_install_db to create /var/lib/mysql again, start MySQL, and finally reimport the data. There will be no need to the optimize steps because of the data import.




  1. Finally, Restart the delayed_job:



    cd /path/to/puppet-dashboard



    env RAILS_ENV=production script/delayed_job -p dashboard -n 2 -m start


  2. Daily Reports Cleanup and DB Maintenance:





For a daily Reports Cleanup you can create a simple BASH script who search the Reports on /var/lib/puppet/reports by time (mtime +14 in our case), remove them and then cleanup the DB with (upto=2 unit=wk) and set it in your crontab.
An example of the script can be:



#!/bin/bash
REPORTS=`find /var/lib/puppet/reports -type f -mtime +14`
for i in $REPORTS; do rm -f $i; done

cd /usr/share/puppet-dashboardenv RAILS_ENV=production rake reports:prune upto=2 unit=wk


Tuesday, May 16, 2017

ddos - Slow SYN flood: preventing SYN amplification?



a (linux) webserver I administer (on a VPS) for a low volume website has just been subjected to about 5 SYN requests/second on port 80, with no other traffic coming from the remote host.
It was more of a drip than a flood, but it went on for something like an hour after I first noticed it. I don't know if this was an amplification attack or not.



I have no desire to be part of that sort of thing, and in any case, all those half-opened connections clogging up netstat were annoying, even if there was no real harm done. (SYN-cookies are enabled, the link was a long way from saturated, and apache was coping, though I think memory usage was up and responses a bit slower than normal.)



So, this one, I 'mitigated' by manually blocking the spoofed / actual
originating IP address, and then kept an eye on it with tcpdump, which is how
I knew they'd given up. I'd like an automatic solution...




As it's a public-facing webserver, there are SUPPOSED to be connections! And
some pages have a lot of pictures/CSS files, etc. on them. I therefore fully expect that genuine users will far exceeding the 5 'attack' requests per second, and that therefore things like



iptables -A INPUT -m state --state NEW -m limit --limit 4/second --limit-burst 20 ....



will just catch out genuine users but barely catch the attack, if at all.



Other than something crude like a script counting SYN_RCVD entries in the output of netstat every so often and then feeding the results into a log for fail2ban to respond to, is there an effective solution to drop packets if the cumulative total of half-open connections from a given IP exceeds some number? I can't see anything in iptables, but maybe I'm missing something.




EDIT: Forgot to say, the VPS is openVZ-based (built on RHEL6), so the kernel I'm stick with is 2.6.32 at the moment. Answers that work on kernels that ancient would be great!



EDIT2: Conntrack timeouts, as requested:



net.netfilter.nf_conntrack_generic_timeout = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 30
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 15
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 30
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 15

net.netfilter.nf_conntrack_tcp_timeout_last_ack = 10
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_tcp_timeout_close = 5
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 60
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 60
net.netfilter.nf_conntrack_udp_timeout = 10
net.netfilter.nf_conntrack_udp_timeout_stream = 60
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_icmp_timeout = 10
net.netfilter.nf_conntrack_events_retry_timeout = 15

net.ipv6.nf_conntrack_frag6_timeout = 4194


But as I wrote above, I'm not so much interested in impact on the server (so far it's been minimal) as blocking script-kiddies and the like, just
like when they try to see if I've got phpmyAdmin, a dlink-router, (etc) that's about the last packet of theirs I get.


Answer



Yes, you can do that Kernel settings (e.g. setting timeouts via conntrack module) in combination with iptables limits and SYNPROXY.



Basically what synproxy does is to check if a full TLS connection is established and if not drop the request. It has a much better performance than other solutions that came before (hashlimits etc.).




And you're right, a limit of 4 connections/second will block a lot of legitimate connections, around 60-80/s is more realistic. For your use case synproxy is definitely the way to go.



When playing around with DDoS protection with built in Linux tools earlier this year myself I found a great anti-DoS guide here: https://javapipe.com/ddos/blog/iptables-ddos-protection/



This guide basically is a how-to to all of the things I mentioned above.
It explains what every rule does, why you should or shouldn't implement it, how it works, how it affects performance and tells you if it's optional, a really great guide.


linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...