Knowledge Base/Engine Yard Cloud Documentation/Manage your Instances

Application Master Takeover

Engine Yard
posted this on February 16, 2012 10:03 AM

Updated: April 14th, 2014

Before you get started, you might like to check out this short video on High Availability.

Takeover is the Engine Yard failover process for recovering from failure of an application master instance.

Notes:

  • Takeover requires that you have at least one application slave in your environment.
  • The environment's application takeover preference determines the method in which a takeover occurs. For example, it's possible to disable automated takeovers; in this case you need to manually initiate app takeovers instead.

If your application master has failed and takeover is happening, you:

  • Receive automated email notification from Engine Yard.
  • See a takeover message for the environment on your dashboard.
    Takeover message

Takeover occurs when Engine Yard detects that your application master is unable to reliably respond to requests. For example, this can happen because of an Amazon EC2 issue or because the instance froze. If the instance does not recover within a short time, Engine Yard does the following:

  • Terminates the problem instance.
  • Promotes an app slave to master, using the method configured by the environment's application takeover preference.
  • If your environment does NOT have elastic IP (EIP) addresses on your app slaves, Engine Yard assigns the old master’s IP address to the new master. If you have EIP addresses on your app slaves, see EIP Addressing.
  • Replaces the app slave instance that was promoted. (The new application slave uses the same version of the stack as the other instances in that environment.)
  • Deletes the old app master (unless you have configured to detach and create a utility instance instead; see app master takeover preference for more information).

EIP Addressing

If you have EIP addresses on your app slaves and an app slave is promoted to an app master, the new app master will retain the IP address of the promoted app slave. Below is an example of this EIP assignment.

Important: If your domain points to the EIP address, this addressing assignment will crash your application because the EIP does not point to any instances.

Before Promotion

App master IP address: 108.20.182.83

App slave IP address: 51.19.102.138

After Promotion

App master IP address: 51.19.102.138

The Engine Yard dashboard shows 108.20.182.83 as available and unused.

Action required to apply cron jobs and custom Chef recipes

The new application master is the same as the old one with two important exceptions:

  • Cron jobs are not set up.
  • Custom Chef recipes are not applied.

To apply cron jobs and custom Chef recipes to the new application master

  1. In your dashboard, click the environment name.
  2. Click Apply.
    This applies/re-applies configuration, including cron jobs and custom Chef recipes as appropriate, to all instances in the environment.

Be prepared

Do the following to prepare your environment in case of application master failover.

To prepare your environment

  1. Do keep a spare application server.
    Make sure that your cluster has one spare application server. For example, if you need three application servers to serve the everyday traffic, put four application servers in your cluster. Without a spare, your site might fail or slow down under load during the takeover.

  2. Don’t keep important data on the application server alone.
    For example, if your application stores user-generated content, consider an online storage web service (such as Amazon S3).

Manual takeover

In some situations you might want to manually initiate a takeover. For example, if you simulate a takeover, you can test your recovery processes.

To manually trigger a takeover

From the application slave, you need to send a notification to stonith several times, which indicates a failed connection to the application master. The instance you run this on will attempt to become the new application master.

i-abcd1234 ~ # for i in $(seq 1 6); do stonith notify; done

When the takeover is complete, you will be in the post-takeover state and need to perform the steps mentioned in Action required to apply cron jobs and custom Chef recipes.

More information

For more information about...See...
Using online storage web services (such as Amazon S3) with applications                   Use CarrierWave (and optionally fog) to upload and store files      
Frozen instances Deal with frozen or crashed instances

If you have feedback or questions about this page, add a comment below. If you need help, submit a ticket with Engine Yard Support.

 

Comments

User photo
Willian Kis Camara
inmov

Hello

I have one doubt about this takeover.

I would like to know how the takeouver knows that the app is down?

Is this configured on takeover?

Regards,

November 05, 2012 09:25 AM
User photo
Tyler Poland
Engine Yard Inc.

Hi Willian,

The application instances run a service called Stonith which checks connectivity to the application master. By default this check runs against the url `https://[internal_AppMaster_hostname]/haproxy/monitor`. In the event this check fails 6 times in a row the first Application Slave to complete notification to the provisioning server will be promoted to the Application Master role.

Thank you,

Tyler

February 11, 2013 08:26 AM
User photo
Daniel Nelson
Populr

Can you explain why cronjobs are not set up on and custom Chef recipes are not applied to the newly promoted instance? Without cronjobs and custom recipes, our environment is not fully functional. Can we configure out instances to automatically  run "Apply" when takeover happens?

January 10, 2014 10:52 AM
User photo
Tasha Drew
Engine Yard Inc.

Hi Daniel, 

Someone from Support may be able to give you some practical advice in the short term, but I've filed a ticket with the engineering team to add this functionality as an option, and added a feature request to our feature request forum to track the work: 

https://support.cloud.engineyard.com/entries/37911908-Configure-ins...

Cheers, Tasha 

January 10, 2014 01:52 PM