Follow

Replace Degraded Instances

Updated:

Read this page if you have received notification from Engine Yard that one of your instances is degraded and that you need to terminate the instance and boot another.

Engine Yard gets notification of degraded instances from Amazon and forwards the message so that you can take action.

When you get one of these notifications, it might state the time that the underlying hardware will be taken offline. If there is no time specified, replace the degraded instance as soon as you can.

If you don’t replace the degraded instance in time, it might become frozen. For information about frozen instances, see Fix frozen or crashed instances.

What is a degraded instance?

An instance is degraded when the host hardware that the instance is running on fails. The hardware failure is not catastrophic. However, Amazon needs to shut down the host to carry out maintenance. When this maintenance happens, all instances on the host are terminated.

To replace a degraded instance

If the degraded instance is:

  • A single-instance (or solo) environment, then terminate and rebuild the whole environment (See ELT on notes below).
  • An application slave, then terminate the specific degraded instance by clicking Terminate icon on the dashboard, and then add a new instance (See ITR on notes below).
  • An application master, then promote an app slave to be app master.
  • A database replica, then terminate the specific degraded instance by clicking Terminate icon on the dashboard, and then add a new instance (See ITR on notes below).
  • A database master, then promote a database replica to be database master.
  • A utility instance, then terminate the specific degraded instance by clicking Terminate icon on the dashboard, and then add a new instance, using the most recent snapshot (See ITR on notes below).

Notes:

  • You must know the difference between the "Environment Level Terminate" (ELT) and the "Instance Terminate reference" (ITR). The first one is identified by the button "Terminate" located at the top of the environment's page, above its name. The second one is identified by a "Terminate" link next to each instance ID.
  • In some cases (and depending on your Support level), Engine Yard might initiate an instance takeover on your behalf. In that case, Engine Yard Support will contact you.
  • The environment's settings for takeover preference and failed app master behavior can also affect the way app takeovers occur.

More information

For more information about...See...
Frozen or crashed instances Fix Frozen or Crashed Instances
Rebuilding an environment Rebuild an Environment
Promoting an app slave Promote an Application Slave
App master takeover Application Master Takeover
Promoting a db replica Promote a Database Replica

If you have feedback or questions about this page, add a comment below. If you need help, submit a ticket with Engine Yard Support.

Was this article helpful?
3 out of 3 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    shobhab shrestha

    for "A utility instance, then terminate the specific degraded instance by clicking Terminate icon on the dashboard, and then add a new instance, using the most recent snapshot."
    my Question: I have "redis and sidekiq for backgroung job" running in my utility instance. Is my email sending and other task running in my utility instance affected during the time I terminate and add a new instance?

  • Avatar
    Daniel Valfre

    Hi Shobhab, yes, the tasks will stop when the instance is terminated, and then be started in the new instance when it comes up. If your site relies on Redis then we will recommend to setup the maintenance page while the util instance is getting replaced.

Powered by Zendesk