Adding a scheduled task (cron job) to your environment can be done via your Dashboard. The task will be added to your application master instance under the crontab of either your deploy user or the root user (you choose which) and will only run on the application master by default. This avoids any potential adverse effects of a task accidentally running multiple times on different instances.
Adding a new job
To add a new cron job
- On the Environment page, click Crontabs
- You will then be presented with the Scheduled Jobs page
- Give the job a name for reference
- Enter the command to run (see notes on commands)
- Choose which user’s crontab to add this to
- Define a schedule (if you are familiar with cron syntax, you may prefer to use the advanced interface)
- Click Create
- Return to your Dashboard and click the Update Instances or Deploy button as prompted
Notes about commands
One of the most common uses for the crontab is to schedule the running of rake tasks or Rails runner scripts. When setting up such a task, you need to be aware of the following points:
- rake tasks and runners need to be run from your application root
- the cron environment does not have the same variables available as your standard shell (i.e.: RAILS_ENV will not be defined)
With that in mind, when adding such a task under cron, you will need to add your command like one of the examples below.
Example rake task
cd /data/appname/current && RAILS_ENV=production rake some:task
Example Rails runner task
cd /data/appname/current && ./script/runner -e production 'SomeClass.method'
or for Rails 3:
cd /data/appname/current && bundle exec rails runner -e production 'SomeClass.method'
Advanced usage
Running crons on different instances
As mentioned at the start of this document, the default setup for scheduled jobs is to only configure them on the app master to avoid inadvertently running duplicate jobs. If you want to run cron jobs on other instances, then you need to create a custom chef recipe to add the jobs to the crontabs of different instances.
A common requirement is to have cron jobs running on a utility instance, to achieve this you might write code like the following inside a cookbook recipe file:
if node[:name] == 'instance_name'
cron "task_name" do
minute '*/15'
user 'deploy'
command "cd /data/appname/current && rails runner -e production 'SomeModel.some_method'"
end
end
Above, instance_name would be the name you gave your utility instance when you created it. The task_name is just a label for the job. (similar to how you would give the job a name via the Dashboard). Note that the minute method is using the cron syntax to specify ‘every 15 minutes’ - you can also specify ‘hour’ and ‘day’ within the code block.
Lockrun
Any frequently scheduled jobs can sometimes potentially run longer than their scheduled window allows. This can cause multiple runs of the same task to overlap. For example, if you have a rake task that runs every five minutes and one of these runs takes fifteen minutes to complete, cron will happily fire up another two runs of this task once their scheduled time comes round. This can cause various effects, dependent on what the task does, but most often causes spikes in server load and memory consumption. This problem tends to slow down the running of subsequent tasks, causing more overruns and compounding the issue.
To solve this common problem, we recommend the use of lockrun. Lockrun is not installed by default so you will need to install it as outlined in Add Unix packages to your application. Update: On stack version stable-v2-1.1.432 +, stable-v4-2.0.87 + and all stable-v5, lockrun v20120508 is available by default.
After you have lockrun installed on your instance(s), you can then use it in your cron jobs by writing the commands like this:
/usr/bin/lockrun --lockfile=/tmp/jobname.lockrun -- sh -c "cd /data/appname/current && rake RAILS_ENV=production some:task"
Reminder: If you're using Bundler in your project, use bundle exec:
/usr/bin/lockrun --lockfile=/tmp/jobname.lockrun -- sh -c "cd /data/appname/current && bundle exec rake RAILS_ENV=production some:task"
Troubleshooting cron issues
Is cron running?
If you find that none of your cron jobs appear to be running, first check that cron is actually running with the following command:
/etc/init.d/vixie-cron status
If this command states that cron is not running, it may require restarting with this command:
/etc/init.d/vixie-cron restart
Are your scripts working?
If you can run the scripts successfully manually, but they fail when running under cron, you’re going to need to see what output comes out of the command when run under cron.
By default, cron will try and send any output from jobs via email to the user that ran the job. As our instances do not have a mailserver running by default, the sending of these messages will fail and the output ends up in a dead.letter file. These files are located in the home directory of the user that ran the job. For example, you will find output of jobs run by root in /root/dead.letter (use sudo to read this file) and the output of jobs by your deploy user would be in /home/deploy/dead.letter (assuming deploy is the name of this user).
Note that dead.letter can contain output from multiple jobs and does not have any timestamps in place, so you may wish to modify the script that is being called by cron to output some extra information if you are trying to debug a problem. You can also delete this file and it will be recreated as necessary when new output is emailed from cron.
If you have feedback or questions about this page, add a comment below. If you need help, submit a ticket with Engine Yard Support.
It's better to run the bundled rake version:
/usr/bin/lockrun --lockfile=/tmp/jobname.lockrun -- sh -c "cd /data/appname/current && ey_bundler_binstubs/rake RAILS_ENV=production some:task"
I created a crontab:
cd /data/appname/current && RAILS_ENV=production rake some:task
But my task was never executed.
Then I tried to execute it directly from the terminal:
RAILS_ENV=production rake some:task
-bash: rake: command not found
So in fact you should add "bundle exec" :
RAILS_ENV=production bundle exec rake some:task
And then it's working.
Either the documentation should be updated or maybe something is missing in my environment ?
Thanks for your comment, Alex. Running rake tasks can differ based on your setup. We do have some customers that have legacy Rails 2.x apps on their machines, so they usually run without the use of bundler. Though as you've pointed out, any bundler-enabled project, be it a Rails app or even a bare-bones Rack/Sinatra app, will need to run bundle exec ... to properly alter the gem path and find the right version of rake and associated gems.
We'll see what we can do about a documentation update. Thanks again for your comment.
A HUGE note of warning to folks who have been running EY for a while: your cron scheduling runs against UTC. It used to be that EY instances ran as Pacific. This one just bit me in the ass. Here's an article I was pointed to with a recipe that changes instance timezone: https://support.cloud.engineyard.com/entries/21016508-set-the-time-zone-for-an-instance
Thanks for pointing that out, Yuval. The reason for this was that we had two AMIs in service at one point in time (still do in some cases). The original AMI was based on PST for a timezone. So when our engineering team came back to update that AMI, the decision was made to make the default time zone UTC so that we can better standardize and internationalize the product as a whole. So when you boot new instances, you're likely going to have your time zone set to UTC by default from now on.
This is most likely to affect people who have had instances that have been running a very long time, kill them off and then spin up new ones, or copy chef recipes to different environments with new instances. A best practice I would recommend would be to use this chef recipe: https://github.com/engineyard/ey-cloud-recipes/tree/master/cookbooks/timezone in all your environments to force the time zone to be what you want (I recommend UTC, and even though that's the default it can't hurt to have the chef recipe overwrite whatever is there to begin with just in case of edge cases) and then calculate offsets as needed in your application and/or when setting up cron jobs.
Thanks J. I used that very recipe, as directed by Ralph, and everything seems honky dorry. Except I set the timezone to US/Pacific to prevent human calculation errors in my cron scheduling. As a side effect my Rails logs are also easier to follow now. I rebuilt my cluster late last year so it must have happened then. Might be worth a mailout?
It would be good if the above documentation conveyed how to know whether a scheduled cronjob has run. The following is the solution I found:
1 - SSH into Master App server
2 - Run the following command: sudo cat /var/log/syslog
Hey Aaron,
Thanks for the note, you are totally right, and I've opened a ticket with our docs team to update this page. :)
-Tasha
Due to the timezone being UTC, we need to make a better schedule for the default database backup which runs as root, however there is no way to do this from the dashboard and when editing with the crontab command you see a warning that the changes will be overwritten. (presumably when you apply environment changes) It's not clear where to go to make this simple change stick. Do I need to get into chef and all that?
Hi Steven,
You're right, those cron jobs are put there by our main chef run and will be overwritten when you hit apply or upgrade. If you'd like to change the time or frequency beyond what the UI offers, you will need to create some custom chef. More information is available at https://support.cloud.engineyard.com/entries/20996681-Back-Up-the-Database.
We also have some chef recipes ready to go that may help.
Changing the timezone on your instances: https://github.com/engineyard/ey-cloud-recipes/tree/master/cookbooks/timezone
Adding cron jobs: https://support.cloud.engineyard.com/entries/21016333-Add-Cron-Jobs
Essentially, you can just copy the existing cronjob for backing up and modify the run times to your needs.
Hope that helps, feel free to open up a support ticket if you need anything else!
I'm also having an issue with crontabs. I've created one:
cd /data/myapp/current && RAILS_ENV=production bundle exec rake banner_import:full_with_cleanup
The command runs when I paste it into the console but it appears to be erroring out in the crontab for some reason. I'm guessing there are errors based on this from dead.letter:
/bin/sh: bundle: command not found
Any suggestions?
Hello Sean,
Could you change that cronjob to
cd /data/myapp/current && RAILS_ENV=production /usr/local/bin/bundle exec rake banner_import:full_with_cleanup
Awesome, thanks for the speedy response!
I'll give it a try :)
Hi!
I've created a cron job as root, but nothing is happened. Can anyone help me? I really appreciate the help asap. Thanks.
cd /data/nearshoreconnect/current && bundle exec rails runner -e production 'Company.destroy_companies_cancel_subscription'
Hi Kirk --
I've opened you a support ticket so one of our support engineers can assist you.
--Tasha
Hi Tasha, thank you for the response. Two questions, how much it would take to answer? and How he will send me the answer?
-- Kirk
Hi Kirk,
A member of our team has already updated your ticket requesting some additional details about the environment. If you have not received messages associated with this ticket or are unable to view it in the ticketing system please reach out to our support team in the #engineyard IRC channel (irc.freenode.net) or give our support line a call at 866.518.9273 so we can assist you further.
Thank you,
Tyler
Also please note that if you're on the stable-v4 stack (can see this from your Edit Environment page), make sure you're running on the latest version - we recently fixed a bug that didn't include the location of bundle in the path.
How does one delete a crontab if you've created one using chef? I'm basically looking to clean up some of my cron actions, move them to different machines but I want it automated.
To remove a cron job from a particular instance in a chef recipe you can refer to it by name, like so:
remove "foo" cronjob from an instance named "bar".
if node[:name] == 'bar'
cron "foo" do
action :delete
end
end
An alternative approach would be to make chef the authoritative source for all cron jobs on a given instance, then delete them all and recreate them as desired.
Is it possible or good to make cron jobs against to "Filesystem Check Warning" ,
https://support.cloud.engineyard.com/hc/en-us/articles/115004832867-Filesystem-Health-Checks
and run this task twice a year ?
Please give me some clue :)
Thank you Engine Yard
Hello fsurucu,
Even if it is possible, it is not advisable to run the healthchecks on such a margin. Because in case of a failure, we can never guess that it would just fail close-by to those days. The main aim of the healthchecks is to give you a good understanding of the health status of the key volumes of the instance, such that you can get them replaced before an issue occurs.
In case you need to disable the health check you may utilize https://support.cloud.engineyard.com/hc/en-us/articles/115004832867-Filesystem-Health-Checks#disabling-the-check
Pasan