Knowledge Base/Engine Yard Cloud Documentation/Customize

Customize Your Environment with Chef Recipes

Engine Yard
posted this on February 16, 2012 10:02 AM

Updated: October 23rd, 2013

You can use Chef recipes to customize your Engine Yard Cloud environments. You can edit cookbooks that are provided by Engine Yard or you can make your own cookbooks.

This page describes:

Set up the Chef environment

In order to be able to develop Chef recipes that build (and rebuild each time you start an instance) your environment customizations, you need to install the engineyard gem in your local environment on your development machine.

Even if you already have the engineyard gem installed, it is good practice to run "install engineyard" to make sure that you have the latest version of the gem.

To install the engineyard gem

  1. Type:

    $ sudo gem install engineyard 
  2. When prompted, enter the password for your Engine Yard account.

Clone the ey-cloud-recipes repository

To work with the ey-cloud-recipes repository, you need to fork and clone it to your development environment on your local machine. Clone a local copy of the ey-cloud-recipes repository in the directory that you'll work in when writing custom Chef recipes for your environment (the same environment where you installed the engineyard gem).

Here is the standard procedure for cloning a GitHub repository:

To fork and clone the ey-cloud-recipes repository

  1. Browse to the ey-cloud-recipes site on GitHub.
  2. Click Fork to fork the repository.
    This creates a fork under your user account on GitHub.
  3. Copy the URL of your forked repository to your clipboard.
  4. Clone the repository to your local development machine:
    Note: Unless you are using submodules in git, do not put the ey-cloud-recipes repository in the same directory as your application repository. By default, git does not support nesting.

About updating the ey-cloud-recipes repository

From time to time, you might want to or need to refresh your ey-cloud-recipes repository to keep up with changes made by Engine Yard. A good reason to refresh the repository is if something isn't working correctly or if there is a new feature that you want to take advantage of. You can review the GitHub Commit History for the repository to find out about recent changes.

Important! Be sure to test carefully after updating the repository and before applying new Chef recipes to a production environment.

About the file structure of ey-cloud-recipes

Become familiar with the file structure of the ey-cloud-recipes repository.

Base files and directories

README.md 
Rakefile
cookbooks/
main/
attributes/
definitions/
libraries/
recipes/

Cookbooks directory

In the cookbooks directory are directories of the self-contained recipes for various "components" (such as Sphinx, MongoDB, Redis, Varnish) that you can enable and customize for your environment.

Each directory under a cookbook has a number of sub-directories. On this page, we use the Sphinx recipe as the example:

cookbooks/ 
sphinx/
files/
default/
sphinx.logrotate
recipes/
default.rb
templates/
default/
sphinx.monitrc.erb
sphinx.yml.erb

Recipes directory

For each cookbook `recipes/default.rb is the main definition file that prescribes how Chef performs each of its actions to achieve the customization. View the Sphinx example.

Files directory

In the Sphinx cookbook, the recipes/default.rb creates a remote_file resource (that's a Chef term). That resource is found in the files/default/sphinx.logrotate location and corresponds to the source value in the code block below.

remote_file "/etc/logrotate.d/sphinx" do 
owner "root"
group "root"
mode 0755
source "sphinx.logrotate"
backup false
action :create
end

The sphinx.logrotate is a file that has no variables. You use the templates and ERB to insert the variable data into the sphinx.logrotate file.

Templates directory

Also in the the Sphinx cookbook, the recipes/default.rb creates a template and passes variables to the template.

In the recipes/default.rb file:

template "/etc/monit.d/sphinx.#{app_name}.monitrc" do 
source "sphinx.monitrc.erb"
owner node[:owner_name]
group node[:owner_name]
mode 0644
variables({
:app_name => app_name,
:user => node[:owner_name],
:flavor => flavor
})
end

The variables above are passed to the template (sphinx.monitrc.erb), and Chef renders the static file (/etc/monit.d/sphinx.myapp.monitrc).

In the sphinx.monitrc.erb file:

check process sphinx_<%= @app_name %>_3312 
with pidfile /var/run/sphinx/<%= @app_name %>.pid
start program = "/engineyard/bin/<%= @flavor %>_searchd <%= @app_name %> start" as uid <%= @user %> and gid <%= @user %>
stop program = "/engineyard/bin/<%= @flavor %>_searchd <%= @app_name %> stop" as uid <%= @user %> and gid <%= @user %>
group sphinx_<%= @app_name %>

Turn on a cookbook

In order to turn on a cookbook and have that set of recipes run when you deploy your application, you need to uncomment the recipe in the cookbooks/main/recipes/default.rb file.

To turn on an existing cookbook

  1. Select the cookbook that you want to use.

  2. In your local environment, open cookbooks/main/recipes/default.rb for editing.

  3. Uncomment the "require_recipe" line for the recipe that you want to turn on.

  4. Follow any additional instructions in the comments.
    For example, for Sphinx, you need to edit cookbooks/sphinx/recipes/default.rb; for details, see Implement full text search with Sphinx on Engine Yard Cloud.

  5. Save cookbooks/main/recipes/default.rb and commit your changes locally and push them to your remote repository.

  6. Use the ey recipes commands to upload and apply the recipes:

    ey recipes upload -e environment_name 
    ey recipes apply -e environment_name

Create a cookbook

If there isn't a ready-made cookbook to suit your needs, you can create your own cookbook from scratch.

The procedure below shows how to create a new Chef recipe called nginx_logrotate. This recipe changes the retention time of Nginx logs from 30 days (the default) to 60 days.

To generate a new Chef recipe:

  1. In your local environment, in the ey-cloud-recipes repository directory, type:

    $ rake new_cookbook COOKBOOK=nginx_logrotate 

    This creates the file: cookbooks/nginx_logrotate/recipes/default.rb.

  2. Edit cookbooks/nginx_logrotate/recipes/default.rb to add the following code:

    remote_file "/etc/logrotate.d/nginx" do 
    owner "root"
    group "root"
    mode 0755
    source "nginx.logrotate"
    backup false
    action :create
    end
  3. Create a file called files/default/nginx.logrotate with the following content:

    /var/log/engineyard/nginx/*.log { 
    daily
    missingok
    compress
    rotate 60
    dateext
    notifempty
    sharedscripts
    extension gz
    postrotate
    [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
    endscript
    }
  4. Edit cookbook/main/recipes/default.rb to enable the recipe:

     require_recipe "nginx_logrotate" 
  5. Test the syntax of your new recipe:

     $ rake test 
  6. Commit your changes locally and push them to your remote repository.

  7. Use the ey recipes commands to upload and apply the recipes:

        ey recipes upload -e environment_name 
    ey recipes apply -e environment_name

Report to the dashboard from custom recipes

You can have messages appear when your custom Chef recipes run. These appear on the Environment page under the "Instances" heading; and allow you to see when the custom portions of your Chef recipes are running. These messages also appear in the chef.custom.log file. (See Report to a log file from custom Chef recipes below.)

To report to the dashboard from custom Chef recipes

  1. Edit the Ruby file (for example, recipes/default.rb) file with code like this:

    ey_cloud_report "recipe_name" do 
    message "message text"
    end

    Where more message text is the message that you want to appear on dashboard when that part of the code is executed.

    For example:

    ey_cloud_report "nginx" do 
    message "custom logrotate for nginx"
    end

There is an example of the ey_cloud_report method in the Sphinx recipe.

Report to a log file from custom Chef recipes

Custom Chef recipes are logged to /var/log/chef.custom.log. (Default Engine Yard Cloud recipes are logged to /var/log/chef.main.log.)

To report to a log file from custom Chef recipes

  1. Edit the RB file (for example, recipes/default.rb) file with code like this:

    Chef::Log.info "message text" 
    For example:
    Chef::Log.info "Doing step 1." 
    writes a line like this:
    [Sun, 22 Jan 2012 22:29:00 +0000] INFO: Doing step 1. 

    to the /var/log/chef.custom.log file.

Specify which instance roles run a recipe

In a clustered environment, you have multiple instances, each instance playing a different role. In most cases, you want the recipe to run on only one type of instance; for example, to run on the application master, but not on the application slave, database, or utility instances.

To specify which instance (role) runs a recipe

  1. Add an if statement around the recipe code:
    if node[:instance_role] == 'instance_role' 
    Where instance_role is one of the following:
    • app_master (for the application master)
    • app (for an application slave)
    • solo (for a single instance)
    • db_master (for a database master)
    • db_slave (for a database slave)
    • util (for a utility instance)

Examples

In the ssh_tunnel recipe, to have the recipe run on only the application master in a clustered environment:

  if node[:instance_role] == 'app_master' 
...
end

However, to have the same recipe run in either a clustered environment or a single-instance environment, write the condition this way:

  if ['app_master', 'solo'].include?(node[:instance_role])
...
end

About utility servers

If you want a recipe to run on a particular utility server, you can specify it by name instead of by instance role.

For example,

  if node[:name] == 'myutility' 
...
end

Where myutility is the name of a utility instance.

More information

For more information about...See...
the engineyard gem                                                                          Engine Yard CLI User Guide.
the ey recipes commands Engine Yard CLI User Guide.
custom Chef recipes Custom Chef Recipes examples & best practices by Tim Littlemore.
running custom Chef recipes during deploy Custom Chef Recipes During Deploy blog by Dr. Nic Williams.

If you have feedback or questions about this page, add a comment below. If you need help, submit a ticket with Engine Yard Support.

 

Comments

User photo
Jesse Cooke
trial-account-110893566

I had a very simple recipe that was missing main/attributes/recipe.rb, and my chef recipes would not run. That file is needed by chef to run, and should include the following 3 lines:

 

recipes 'main'

owner_name @attributes[:users].first[:username]

owner_pass @attributes[:users].first[:password]

May 04, 2012 10:19 AM
User photo
Shai Rosenfeld
Engine Yard Inc.

It should be in ey-cloud-recipes, which is the repo for our example recipes: recipe.rb

May 10, 2012 01:53 PM
User photo
Keri Meredith
Engine Yard Inc.

Hi Jesse, thanks for your input. The missing recipe.rb file is part of the Clone the ey-cloud-recipes repository section above.

As Shai mentioned, you can find recipe.rb in ey-cloud-recipes on Github.

Best practice is to clone it and add custom recipes after that. kjm

May 21, 2012 04:54 PM
User photo
Petteri Räty
experq

I wonder why the instructions tell you to push your changes to github when that is not required for ey recipes to work. It's of course a form of backup and contribution to the general community but people should be told it's their choice.

June 09, 2012 03:00 AM
User photo
Marcus Malmberg
cko-adwall

May I suggest that you add, to the guide, that one need to have the recipe.rb file containing:

recipes 'main'
owner_name @attributes[:users].first[:username]
owner_pass @attributes[:users].first[:password]

in order to be able to run any recipe at all. I read Jesse's comment earlier but didn't realize that it was needed to run any recipes at all, and thought it was related to something with his code. 

I'm aware that this guide want you to use ey-cloud-recipes, but I was only interested in a simple recipe that would create a swapfile and it felt superfluous to clone a git repo to accomplish that. Since it's so easy to create the needed folder structure and files manually I'd love to see a short section about that in the guide as well, since I imagine that I'm not alone in this matter :)

July 05, 2012 02:36 AM
User photo
Jason Fowler
ACL US Ltd

There is a typo under Cookbooks Directory:  "teplates" should be "templates".  Mmmm..

teplates/ 
October 23, 2013 10:18 AM
User photo
Diana Lam
Engine Yard Inc.

Thanks for catching that Jason! Typo has been fixed.

October 23, 2013 10:29 AM