Wednesday, September 26, 2018

centos7 - Cron daily running twice



On my CentOS 7 server, errors generated by a nightly backup script that should have run fine caused me to check on my cron activity. I discovered that cron.daily is running twice - here is the relevant section of /var/log/cron after I removed my backup script to see if it was somehow causing the problem:



Oct 10 02:28:01 mail CROND[1750]: (root) CMD (run-parts /etc/cron.hourly)
Oct 10 02:28:01 mail run-parts(/etc/cron.hourly)[1750]: starting 0anacron

Oct 10 02:28:01 mail anacron[1759]: Anacron started on 2017-10-10
Oct 10 02:28:01 mail run-parts(/etc/cron.hourly)[1761]: finished 0anacron
Oct 10 02:28:01 mail anacron[1759]: Normal exit (0 jobs run)
Oct 10 02:30:01 mail CROND[1766]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 02:40:01 mail CROND[1847]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 02:50:01 mail CROND[1936]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 03:00:01 mail CROND[2032]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 03:10:01 mail CROND[2148]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 03:17:01 mail CROND[2223]: (root) CMD (run-parts /etc/cron.daily)
Oct 10 03:17:01 mail run-parts(/etc/cron.daily)[2223]: starting kizunademo

Oct 10 03:17:02 mail run-parts(/etc/cron.daily)[2259]: finished kizunademo
Oct 10 03:17:02 mail run-parts(/etc/cron.daily)[2223]: starting logrotate
Oct 10 03:17:02 mail run-parts(/etc/cron.daily)[2266]: finished logrotate
Oct 10 03:17:02 mail run-parts(/etc/cron.daily)[2223]: starting man-db.cron
Oct 10 03:17:02 mail run-parts(/etc/cron.daily)[2277]: finished man-db.cron
Oct 10 03:20:01 mail CROND[2288]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 03:28:01 mail CROND[2367]: (root) CMD (run-parts /etc/cron.hourly)
Oct 10 03:28:01 mail run-parts(/etc/cron.hourly)[2367]: starting 0anacron
Oct 10 03:28:01 mail anacron[2376]: Anacron started on 2017-10-10
Oct 10 03:28:01 mail run-parts(/etc/cron.hourly)[2378]: finished 0anacron

Oct 10 03:28:01 mail anacron[2376]: Will run job `cron.daily' in 35 min.
Oct 10 03:28:01 mail anacron[2376]: Jobs will be executed sequentially
Oct 10 03:30:01 mail CROND[2381]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 03:40:01 mail CROND[2462]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 03:50:02 mail CROND[2547]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 04:00:01 mail CROND[2670]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 10 04:03:01 mail anacron[2376]: Job `cron.daily' started
Oct 10 04:03:01 mail run-parts(/etc/cron.daily)[2685]: starting kizunademo
Oct 10 04:03:02 mail run-parts(/etc/cron.daily)[2721]: finished kizunademo
Oct 10 04:03:02 mail run-parts(/etc/cron.daily)[2685]: starting logrotate

Oct 10 04:03:02 mail run-parts(/etc/cron.daily)[2728]: finished logrotate
Oct 10 04:03:02 mail run-parts(/etc/cron.daily)[2685]: starting man-db.cron
Oct 10 04:03:03 mail run-parts(/etc/cron.daily)[2739]: finished man-db.cron
Oct 10 04:03:03 mail anacron[2376]: Job `cron.daily' terminated
Oct 10 04:03:03 mail anacron[2376]: Normal exit (1 job run)


Why is cron.daily running twice? As you can see, the log contains some entries related to the second run that aren't present for the first run: two lines announcing the upcoming run, and two more lines saying that it terminated with a normal exit. The first run simply ran the scripts with no extra fanfare. I assume that means something, but I don't know what.



I checked everything I could think of for doubles of something. I'm pretty sure I've read every similar thread on the subject, so compare with the following before calling this a duplicate question. Based on Why is cron running twice? I checked for extra processes - the complete output of ps aux | grep cron is as follows, so there is only one process:




root      9383  0.0  0.2 112672  2340 pts/0    S+   15:18   0:00 grep --color=auto cron
root 25624 0.0 0.0 126248 320 ? Ss Sep30 0:02 /usr/sbin/crond -n


Based on Cron jobs running twice - Ubuntu server 12.04 I also checked crontab -l -u root, which said no crontab for root.



And here is my /etc/crontab:



SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=my@email.com

28 * * * * root run-parts /etc/cron.hourly
17 3 * * * root run-parts /etc/cron.daily
44 2 * * 0 root run-parts /etc/cron.weekly
8 2 7 * * root run-parts /etc/cron.monthly


Thoughts?




EDIT (9 months after this discussion had fallen silent):



Comment today from Marin Velikov made me aware that there is an anacrontab file (I know it's silly, but it hadn't even occurred to me). Here is its contents:



SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45

# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly


So that's the cause. But why would the system be configured to run both? I assume someone smarter than me set it up this way, so I hesitate to muck with it before knowing the purpose. And if I should indeed get rid of the entries in either crontab or anacrontab, which one is best? Anacron is apparently the more sophisticated tool, but it seems weird/wrong to empty one's crontab. Am I just too old-school?



Answer




Why is cron.daily running twice?




crond is running it once:



Oct 10 03:17:01 mail CROND[2223]: (root) CMD (run-parts /etc/cron.daily)



anacron is running it once:



Oct 10 04:03:01 mail anacron[2376]: Job `cron.daily' started


crond started anacron, that is why you didn't see a process for it:



Oct 10 03:28:01 mail CROND[2367]: (root) CMD (run-parts /etc/cron.hourly)
Oct 10 03:28:01 mail run-parts(/etc/cron.hourly)[2367]: starting 0anacron
Oct 10 03:28:01 mail anacron[2376]: Anacron started on 2017-10-10

Oct 10 03:28:01 mail run-parts(/etc/cron.hourly)[2378]: finished 0anacron
Oct 10 03:28:01 mail anacron[2376]: Will run job `cron.daily' in 35 min.
Oct 10 03:28:01 mail anacron[2376]: Jobs will be executed sequentially

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...