So I'm currently trying to get a script working but it's behaving differently when I run it manually than when I run it from crontab. Basically, I have a reverse ssh tunnel set up from one server to another, and in order to verify that my tunnel is up I:
- SSH from server A to server B
- Wget a test url from on server A from server B
- if Wget succeeds, I disconnect and do nothing
- if Wget fails, I disconnect and restart the tunnel
I know there are more elegant ways to verify ssh tunnels (like autossh and ServerKeepAlive), but for both policy and redundancy issues, I have to do things this way. Anyways, here's the script:
from __future__ import print_function
from __future__ import absolute_import
import os, sys, subprocess, logging, pexpect
COMMAND_PROMPT = '[#$] '
TERMINAL_PROMPT = '(?1)terminal type\?'
TERMINAL_TYPE = 'vt100'
SSH_NEWKEY = '(?i)are you sure you want to continue connecting'
SERVERS = [{address':'192.168.100.10', 'connString':'ssh user@192.168.100.10', 'testGet':'wget http://192.168.100.11/test.html -t 1 -T 10', 'tunnel':'start_tunnel'}, {address':'192.168.100.12', 'connString':'ssh user@192.168.100.12', 'testGet':'wget http://192.168.100.13/test.html -t 1 -T 10', 'tunnel':'start_tunnel2'}]
def main():
global COMMAND_PROMPT, TERMINAL_PROMPT, TERMINAL_TYPE, SSH_NEWKEY, SERVERS
#set up logging
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
handler = logging.FileHandler('/home/user/tunnelTest.log')
formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s: %(message)s')
handler.setFormatter(formatter)
log.addHandler(handler)
for x in SERVERS:
#connect to server
child = pexpect.spawn(x['connString'])
i = child.expect([pexpect.TIMEOUT, SSH_NEWKEY, COMMAND_PROMPT, '(?i)password'])
if i == 0: #Timeout
log.debug('ERROR! Could not log in to ' + x['address'] + ' ...')
sys.exit(1)
if i = 1: #No key cached
child.sendline('yes')
child.expect(COMMAND_PROMPT)
log.debug('Connected to ' + x['address'] + '...')
if i = 2: #Good to go
log.debug('Connected to ' + x['address'] + '...')
pass
#Housecleaning
child.sendline('cd /tmp')
child.expect(COMMAND_LINE)
child.sendline('rm -r test.html')
child.expect(COMMAND_LINE)
log.debug('Testing service using ' + x['testGet'] + ' ...')
child.sendline(x['testGet'])
child.expect(COMMAND_PROMPT)
if 'saved' in child.before.lower():
log.debug('Tunnel working, nothing to do here!')
log.debug('Disconnecting from remote host ' + x['address'] + '...')
child.sendline('exit')
else:
log.error('Tunnel down!')
log.debug('Disconnecting from remote host ' + x['address'] + ' and restarting tunnel')
child.sendline('exit')
subprocess.call(['start',x['tunnel']])
log.debug('Autossh tunnel restarted')
if __name__ == "__main__":
main()
My crontab entry is as follows:
0,30 * * * * python /home/user/tunnelTest.py
So yeah -- this script runs fine when I do it manually (sudo python tunnelTest.py) and also runs fine on crontab unless a tunnel is down. When a tunnel is down, I get the "Tunnel down!" and "Disconnecting from remote host 192.168.100.10 and restarting tunnel" messages in my log, but the script seems to die there. The tunnel doesn't restart, and I get no messages in my log until the start of the next scheduled run.
The start_tunnel script is in /etc/init, the testTunnel.py script is in /home/user, the testTunnel.log file is in /home/user/logs, and I ran crontab -e as root.
Any insight into this matter would be greatly appreciated.
Thanks!
