Knowledge Base/Engine Yard Cloud Documentation/Customize

Add Cron Jobs

Engine Yard
posted this on February 16, 2012 10:02 AM

Updated: November 28th, 2012

Adding a scheduled task (cron job) to your environment can be done via your Dashboard. The task will be added to your application master instance under the crontab of either your deploy user or the root user (you choose which) and will only run on the application master by default. This avoids any potential adverse effects of a task accidentally running multiple times on different instances.

Adding a new job

To add a new cron job

  • On the Environment page, click Crontabs
  • You will then be presented with the Scheduled Jobs page
  • Give the job a name for reference
  • Enter the command to run (see notes on commands)
  • Choose which user’s crontab to add this to
  • Define a schedule (if you are familiar with cron syntax, you may prefer to use the advanced interface)
  • Click Create
  • Return to your Dashboard and click the Update Instances or Deploy button as prompted

Notes about commands

One of the most common uses for the crontab is to schedule the running of rake tasks or Rails runner scripts. When setting up such a task, you need to be aware of the following points:

  • rake tasks and runners need to be run from your application root
  • the cron environment does not have the same variables available as your standard shell (i.e.: RAILS_ENV will not be defined)

With that in mind, when adding such a task under cron, you will need to add your command like one of the examples below.

Example rake task

cd /data/appname/current && RAILS_ENV=production rake some:task 

Example Rails runner task

cd /data/appname/current && ./script/runner -e production 'SomeClass.method' 

or for Rails 3:

cd /data/appname/current && bundle exec rails runner -e production 'SomeClass.method' 

Advanced usage

Running crons on different instances

As mentioned at the start of this document, the default setup for scheduled jobs is to only configure them on the app master to avoid inadvertently running duplicate jobs. If you want to run cron jobs on other instances, then you need to create a custom chef recipe to add the jobs to the crontabs of different instances.

A common requirement is to have cron jobs running on a utility instance, to achieve this you might write code like the following inside a cookbook recipe file:

if node[:name] == 'instance_name' 
cron "task_name" do
minute '*/15'
user 'deploy'
command "cd /data/appname/current && rails runner -e production 'SomeModel.some_method'"
end
end

Above, instance_name would be the name you gave your utility instance when you created it. The task_name is just a label for the job. (similar to how you would give the job a name via the Dashboard). Note that the minute method is using the cron syntax to specify ‘every 15 minutes’ - you can also specify ‘hour’ and ‘day’ within the code block.

Lockrun

Any frequently scheduled jobs can sometimes potentially run longer than their scheduled window allows. This can cause multiple runs of the same task to overlap. For example, if you have a rake task that runs every five minutes and one of these runs takes fifteen minutes to complete, cron will happily fire up another two runs of this task once their scheduled time comes round. This can cause various effects, dependent on what the task does, but most often causes spikes in server load and memory consumption. This problem tends to slow down the running of subsequent tasks, causing more overruns and compounding the issue.

To solve this common problem, we recommend the use of lockrun. Lockrun is not installed by default so you will need to install it as outlined in Add Unix packages to your application.

After you have lockrun installed on your instance(s), you can then use it in your cron jobs by writing the commands like this:

/usr/bin/lockrun --lockfile=/tmp/jobname.lockrun -- sh -c "cd /data/appname/current && rake RAILS_ENV=production some:task"

Reminder: If you're using Bundler in your project, use bundle exec:

/usr/bin/lockrun --lockfile=/tmp/jobname.lockrun -- sh -c "cd /data/appname/current && bundle exec rake RAILS_ENV=production some:task"

Troubleshooting cron issues

Is cron running?

If you find that none of your cron jobs appear to be running, first check that cron is actually running with the following command:

/etc/init.d/vixie-cron status 

If this command states that cron is not running, it may require restarting with this command:

/etc/init.d/vixie-cron restart 

Are your scripts working?

If you can run the scripts successfully manually, but they fail when running under cron, you’re going to need to see what output comes out of the command when run under cron.

By default, cron will try and send any output from jobs via email to the user that ran the job. As our instances do not have a mailserver running by default, the sending of these messages will fail and the output ends up in a dead.letter file. These files are located in the home directory of the user that ran the job. For example, you will find output of jobs run by root in /root/dead.letter (use sudo to read this file) and the output of jobs by your deploy user would be in /home/deploy/dead.letter (assuming deploy is the name of this user).

Note that dead.letter can contain output from multiple jobs and does not have any timestamps in place, so you may wish to modify the script that is being called by cron to output some extra information if you are trying to debug a problem. You can also delete this file and it will be recreated as necessary when new output is emailed from cron.


If you have feedback or questions about this page, add a comment below. If you need help, submit a ticket with Engine Yard Support.

 

Comments

User photo
Alexander Greim
benztownbranding

It's better to run the bundled rake version:

/usr/bin/lockrun --lockfile=/tmp/jobname.lockrun -- sh -c "cd /data/appname/current && ey_bundler_binstubs/rake RAILS_ENV=production some:task"
September 02, 2012 10:23 AM
User photo
Alex klein
DEX

I created a crontab:

cd /data/appname/current && RAILS_ENV=production rake some:task 

But my task was never executed.

Then I tried to execute it directly from the terminal:

RAILS_ENV=production rake some:task 
-bash: rake: command not found

 

So in fact you should add "bundle exec" :

RAILS_ENV=production bundle exec rake some:task 

And then it's working.

Either the documentation should be updated or maybe something is missing in my environment ?

November 26, 2012 04:33 PM
User photo
J. Austin Hughey
Engine Yard Inc.

Thanks for your comment, Alex. Running rake tasks can differ based on your setup. We do have some customers that have legacy Rails 2.x apps on their machines, so they usually run without the use of bundler. Though as you've pointed out, any bundler-enabled project, be it a Rails app or even a bare-bones Rack/Sinatra app, will need to run bundle exec ... to properly alter the gem path and find the right version of rake and associated gems.

We'll see what we can do about a documentation update. Thanks again for your comment.

November 26, 2012 04:43 PM
User photo
Yuval Kordov
Giftr

A HUGE note of warning to folks who have been running EY for a while: your cron scheduling runs against UTC. It used to be that EY instances ran as Pacific. This one just bit me in the ass. Here's an article I was pointed to with a recipe that changes instance timezone: https://support.cloud.engineyard.com/entries/21016508-set-the-time-zone-for-an-instance

January 15, 2013 02:39 PM
User photo
J. Austin Hughey
Engine Yard Inc.

Thanks for pointing that out, Yuval. The reason for this was that we had two AMIs in service at one point in time (still do in some cases). The original AMI was based on PST for a timezone. So when our engineering team came back to update that AMI, the decision was made to make the default time zone UTC so that we can better standardize and internationalize the product as a whole. So when you boot new instances, you're likely going to have your time zone set to UTC by default from now on.

This is most likely to affect people who have had instances that have been running a very long time, kill them off and then spin up new ones, or copy chef recipes to different environments with new instances. A best practice I would recommend would be to use this chef recipe: https://github.com/engineyard/ey-cloud-recipes/tree/master/cookbook... in all your environments to force the time zone to be what you want (I recommend UTC, and even though that's the default it can't hurt to have the chef recipe overwrite whatever is there to begin with just in case of edge cases) and then calculate offsets as needed in your application and/or when setting up cron jobs.

January 15, 2013 02:58 PM
User photo
Yuval Kordov
Giftr

Thanks J. I used that very recipe, as directed by Ralph, and everything seems honky dorry. Except I set the timezone to US/Pacific to prevent human calculation errors in my cron scheduling. As a side effect my Rails logs are also easier to follow now. I rebuilt my cluster late last year so it must have happened then. Might be worth a mailout?

January 15, 2013 03:05 PM
User photo
Aaron Bartell
vpsa

It would be good if the above documentation conveyed how to know whether a scheduled cronjob has run.  The following is the solution I found:

1 - SSH into Master App server

2 - Run the following command: sudo cat /var/log/syslog

April 04, 2013 12:19 PM
User photo
Tasha Drew
Engine Yard Inc.

Hey Aaron, 

Thanks for the note, you are totally right, and I've opened a ticket with our docs team to update this page. :) 

-Tasha

April 04, 2013 10:12 PM
User photo
Steven Talcott Smith
App2Saas-AELOGICA

Due to the timezone being UTC, we need to make a better schedule for the default database backup which runs as root, however there is no way to do this from the dashboard and when editing with the crontab command you see a warning that the changes will be overwritten.  (presumably when you apply environment changes)    It's not clear where to go to make this simple change stick.  Do I need to get into chef and all that?

November 25, 2013 05:44 PM
User photo
John Yerhot
Engine Yard Inc.

Hi Steven,

You're right, those cron jobs are put there by our main chef run and will be overwritten when you hit apply or upgrade.   If you'd like to change the time or frequency beyond what the UI offers, you will need to create some custom chef.  More information is available at https://support.cloud.engineyard.com/entries/20996681-Back-Up-the-D....  

We also have some chef recipes ready to go that may help.  

Changing the timezone on your instances: https://github.com/engineyard/ey-cloud-recipes/tree/master/cookbook...

Adding cron jobs: https://support.cloud.engineyard.com/entries/21016333-Add-Cron-Jobs

Essentially, you can just copy the existing cronjob for backing up and modify the run times to your needs.

Hope that helps, feel free to open up a support ticket if you need anything else!

November 26, 2013 09:41 AM
User photo
Sean Marcia
Daniel-Collier

I'm also having an issue with crontabs. I've created one:

cd /data/myapp/current && RAILS_ENV=production bundle exec rake banner_import:full_with_cleanup

The command runs when I paste it into the console but it appears to be erroring out in the crontab for some reason. I'm guessing there are errors based on this from dead.letter:

/bin/sh: bundle: command not found

 Any suggestions?

 

 

December 05, 2013 09:18 AM
User photo
Ralph Bankston
Engine Yard Inc.

Hello Sean,

 

Could you change that cronjob to 

cd /data/myapp/current && RAILS_ENV=production /usr/local/bin/bundle exec rake banner_import:full_with_cleanup
December 05, 2013 09:20 AM
User photo
Sean Marcia
Daniel-Collier

Awesome, thanks for the speedy response!

I'll give it a try :)

December 05, 2013 09:21 AM
User photo
Kirk Laughlin
Nearshoreconnect

Hi!

I've created a cron job as root, but nothing is happened. Can anyone help me? I really appreciate the help asap. Thanks.

cd /data/nearshoreconnect/current && bundle exec rails runner -e production 'Company.destroy_companies_cancel_subscription'
January 30, 2014 02:22 PM
User photo
Tasha Drew
Engine Yard Inc.

Hi Kirk --

I've opened you a support ticket so one of our support engineers can assist you. 

--Tasha 

January 30, 2014 02:25 PM
User photo
Kirk Laughlin
Nearshoreconnect

Hi Tasha, thank you for the response. Two questions, how much it would take to answer? and How he will send me the answer?

-- Kirk

January 31, 2014 07:28 AM
User photo
Tyler Poland
Engine Yard Inc.

Hi Kirk,

A member of our team has already updated your ticket requesting some additional details about the environment. If you have not received messages associated with this ticket or are unable to view it in the ticketing system please reach out to our support team in the #engineyard IRC channel (irc.freenode.net) or give our support line a call at 866.518.9273 so we can assist you further.

Thank you,

Tyler

January 31, 2014 08:19 AM
User photo
Dennis Bell
Engine Yard Inc.

Also please note that if you're on the stable-v4 stack (can see this from your Edit Environment page), make sure you're running on the latest version -  we recently fixed a bug that didn't include the location of bundle in the path.

January 31, 2014 09:43 AM
User photo
Influitive Corp.
infl

How does one delete a crontab if you've created one using chef? I'm basically looking to clean up some of my cron actions, move them to different machines but I want it automated.

February 04, 2014 01:11 PM
User photo
John Dalton
Engine Yard Inc.

To remove a cron job from a particular instance in a chef recipe you can refer to it by name, like so:

# remove "foo" cronjob from an instance named "bar".
if node[:name] == 'bar'
  cron "foo" do
    action :delete
  end
end

An alternative approach would be to make chef the authoritative source for all cron jobs on a given instance, then delete them all and recreate them as desired.

February 04, 2014 03:08 PM