Monday, December 2, 2019

cron - Python script succeeds manually but fails on crontab

So I'm currently trying to get a script working but it's behaving differently when I run it manually than when I run it from crontab. Basically, I have a reverse ssh tunnel set up from one server to another, and in order to verify that my tunnel is up I:





  • SSH from server A to server B

  • Wget a test url from on server A from server B

  • if Wget succeeds, I disconnect and do nothing

  • if Wget fails, I disconnect and restart the tunnel



I know there are more elegant ways to verify ssh tunnels (like autossh and ServerKeepAlive), but for both policy and redundancy issues, I have to do things this way. Anyways, here's the script:




from __future__ import print_function
from __future__ import absolute_import

import os, sys, subprocess, logging, pexpect

COMMAND_PROMPT = '[#$] '
TERMINAL_PROMPT = '(?1)terminal type\?'
TERMINAL_TYPE = 'vt100'
SSH_NEWKEY = '(?i)are you sure you want to continue connecting'
SERVERS = [{address':'192.168.100.10', 'connString':'ssh user@192.168.100.10', 'testGet':'wget http://192.168.100.11/test.html -t 1 -T 10', 'tunnel':'start_tunnel'}, {address':'192.168.100.12', 'connString':'ssh user@192.168.100.12', 'testGet':'wget http://192.168.100.13/test.html -t 1 -T 10', 'tunnel':'start_tunnel2'}]


def main():

global COMMAND_PROMPT, TERMINAL_PROMPT, TERMINAL_TYPE, SSH_NEWKEY, SERVERS

#set up logging
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
handler = logging.FileHandler('/home/user/tunnelTest.log')
formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s: %(message)s')

handler.setFormatter(formatter)
log.addHandler(handler)


for x in SERVERS:

#connect to server
child = pexpect.spawn(x['connString'])
i = child.expect([pexpect.TIMEOUT, SSH_NEWKEY, COMMAND_PROMPT, '(?i)password'])
if i == 0: #Timeout

log.debug('ERROR! Could not log in to ' + x['address'] + ' ...')
sys.exit(1)
if i = 1: #No key cached
child.sendline('yes')
child.expect(COMMAND_PROMPT)
log.debug('Connected to ' + x['address'] + '...')
if i = 2: #Good to go
log.debug('Connected to ' + x['address'] + '...')
pass


#Housecleaning
child.sendline('cd /tmp')
child.expect(COMMAND_LINE)
child.sendline('rm -r test.html')
child.expect(COMMAND_LINE)

log.debug('Testing service using ' + x['testGet'] + ' ...')
child.sendline(x['testGet'])
child.expect(COMMAND_PROMPT)
if 'saved' in child.before.lower():

log.debug('Tunnel working, nothing to do here!')
log.debug('Disconnecting from remote host ' + x['address'] + '...')
child.sendline('exit')
else:
log.error('Tunnel down!')
log.debug('Disconnecting from remote host ' + x['address'] + ' and restarting tunnel')
child.sendline('exit')
subprocess.call(['start',x['tunnel']])
log.debug('Autossh tunnel restarted')


if __name__ == "__main__":
main()


My crontab entry is as follows:



0,30 * * * * python /home/user/tunnelTest.py


So yeah -- this script runs fine when I do it manually (sudo python tunnelTest.py) and also runs fine on crontab unless a tunnel is down. When a tunnel is down, I get the "Tunnel down!" and "Disconnecting from remote host 192.168.100.10 and restarting tunnel" messages in my log, but the script seems to die there. The tunnel doesn't restart, and I get no messages in my log until the start of the next scheduled run.




The start_tunnel script is in /etc/init, the testTunnel.py script is in /home/user, the testTunnel.log file is in /home/user/logs, and I ran crontab -e as root.



Any insight into this matter would be greatly appreciated.



Thanks!

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...