Developer Center

Kevin French Mar 18 3 News and Notes / Security Updates

March 18, 2015: 11:30am PDT (6:30pm UTC) - 

Engine Yard is aware of the recently announced vulnerability in the OpenSSL protocol. The corrected versions are 1.0.2a, 1.0.1m, 1.0.0r and 0.9.8zf. The announcement’s high-risk vulnerability pertains only to 1.0.2, which is not applicable to Engine Yard’s application stacks, and therefore poses no risk. However, versions 1.0.1k, 1.0.0p, and 0.9.8zd are applicable as medium-severity risk. The following CVEs are involved, but detailed information has not been publicly disclosed at this time:

CVE-2015-0209

CVE-2015-0285

CVE-2015-0288

CVE-2015-0291

Once OpenSSL has released further information and the recommended patches, we will begin the process of reviewing, testing, and integrating.

We will update this Known Issue once more information is available.

 

March 23, 2015: 4:48pm PT

Engine Yard has updated all infrastructure-related environments with the OpenSSL patches. Furthermore, customers can now update their environment(s) with the recently released hotfix. Please see our release notes here for instructions and further details.

Important details about the hotfix

+ Services that use OpenSSL will need to be manually restarted after the Cloud dashboard upgrade is completed. Alternatively, a full restart of all instances will negate the need to follow the remaining notes in this list.
+ Nginx, ntpd, postgres and most other services can be restarted using their respective init scripts (i.e. `/etc/init.d/<NAME> restart`).
+ Restarting Nginx will restart Passenger (and the Rack or Rails processes it runs). PHP-FPM, Unicorn, and Puma processes may also need to be restarted if they use SSL for outgoing connection.
+ Restarting monit can be done via `monit quit; sleep 5; monit summary`.
+ Background job processes can either be manually restarted or instead simply redeploy the relevant application(s).
+ Database environments should have slaves restarted before the master.

Customers with any questions or concerns for this update should reach out to our Application Support engineers via a support ticket.

 

Kevin French Mar 1 1 News and Notes / Engine Yard Known Issues

March 1, 2015 11:40am PST (7:40pm UTC): Amazon has informed us of a maintenance event that will affect a percentage of our customers’ instances. The window for this event begins on March 3rd at 12:00AM UTC through March 10th at 11:00 PM UTC. Information on this maintenance reboot can be obtained from Amazon here: https://aws.amazon.com/premiumsupport/maintenance-2015-03/ 

These instances will be rebooted by Amazon and unavailable for a few minutes inside the maintenance window. During that time, the underlying hosts will have the necessary maintenance performed on them. Affected instances are intended to return to normal operation following the reboot. 

Engine Yard has received a complete list of affected instances from Amazon and will contact our affected customers individually by means of a support ticket to Cloud account owners.  These tickets will contain a link that, when logged into your Engine Yard Cloud Dashboard, allows you to view the affected instance IDs, the environment details and their scheduled reboot window.

If you do not receive a proactive ticket, but want to check and see if you are affected, logging in to your cloud dashboard and going to: https://cloud.engineyard.com/instance-reboots will show if any instances are affected or not. 

We recommend subscribing to this “Known Issue” page (https://support.cloud.engineyard.com/entries/89523998-Amazon-reboots-due-to-Xen-vulnerability), and our status page (http://engineyard.statuspage.io/), which can generate notifications as updated information becomes available.

Should you have any specific questions on how your system is affected, please submit a support ticket and our Application Support staff will happily assist.

 

Updates: 

March 2, 2015 - 8:55am PST (4:55pm UTC) - 

AWS has announced that they will be able to update most of the affected instances without requiring a reboot. The majority of our customers will no longer face any reboots. Please continue to check https://cloud.engineyard.com/instance-reboots for an updated list. Due to this change, you may notice previously affected instances are no longer listed, or that your list is much smaller. 

 

March 10, 2015 - 7:15am PDT (2:15pm UTC) - 

This morning, AWS has confirmed the completion of the reboots for all affected instances. With the finalization of of this reboot process, we will be marking this event as resolved. It was our pleasure to support you during this time and welcome any support tickets with questions or comments you may have! 

 

 

Frequently Asked Questions (FAQs)

Listed below are common questions regarding the event, our recommendations for each, and any source material containing additional information.

 

Q: What do I need to do after the reboot to ensure my services run reliably?

A: Be sure to click ‘Apply’ in the Cloud dashboard once your instance is rebooted.  In some situations it is possible your instance will be initiated as a ‘quick run’ in which cookbooks are not also run during the boot process.

 

Q: Does this event affect me?  Where can I see what instances are included in the reboots?

A: Please take a look at Engine Yard’s Instance Reboots page on our Cloud dashboard for a list of what instances need reboots you have access to. Newly booted instances can be affected to. It is our recommendation to check the link periodically to see if any instances in your account populate the report.

 

Q: Will I keep the same IP address?

A: You will keep the same assigned IPs. New IPs are only assigned when an instance is stopped then started, not rebooted.

 

Q: Why couldn’t I have more notice?  Will I have to reboot again for another imminent maintenance event?

A: We notified our customers as soon as we were made aware of this event.  This event is to apply emergency maintenance to Amazon instances that are vulnerable to the Xen vulnerability.

 

Q: Is there any way to opt out?  What if I switch to new instances?

A: No, this event applies to a large group of Amazon instances.  Newer instances may not be affected, but the list of those instance types is not being released. 

 

Q: What if I reboot now?  Will I avoid the reboot later?  

A: This will not avoid the maintenance window.

 

Q: After the reboot my services did not restart automatically, why?

A: After the reboot you must click "Apply" on the environment.   This is not automatic. The Stable-v4/2012.11 instances may run cookbooks on reboots but there is an issue where it may be a quick run only. Without the “Apply” being clicked, HAProxy, Nginx, MySQL and Postgres may not automatically start.

 

Q: How can I minimize the impact this event will have on my operations?  

A: We suggest that you ensure any custom services or applications are properly configured to start automatically after a reboot.  Also consider adding additional instances for your critical environments across multiple availability zones (AZs). Amazon may perform this maintenance on more than one AZ a day, but at different times. 

Severin Discher Jan 27 5 News and Notes / Security Updates

February 10, 2015: 1:45pm PT

A patch for v2 (2009a) is now available via the methods described in the February 2nd update.  

This concludes the planned development work to mitigate risk for CVE-2015-0235.  If you have any questions on how to apply the patch to your system, please do not hesitate to inform an Application Support engineer via our ticketing system.

February 6, 2015: 11:36am PT

The v2 (2009a) patch is still progressing through the QA process and is not yet ready for distribution, however we expect a release shortly.  

February 2, 2015: 3:20pm PT

The patch is fully ready for customers on 2012.11 to apply to their environments. Customers on 2009a will have a release later this week.

Once the Upgrade button on your environment’s dashboard becomes available, please click it to receive the patch. However, this will not purge the vulnerability from your environment(s). Once the upgrade process is complete, perform one of the following (listed in recommended preference order):

Simple, quick, with some downtime -Terminate and rebuild the environment using the most recent snapshot to fully purge the vulnerable software from all involved instances.

Maximized uptime -

* Cycle Application slave and utility instances (add new instance, then remove the older one, for each existing instance)
* Promote one Application slave to master
* Remove all existing DB slaves, making note of any names
* Add new DB slave, and wait for it to catch up to master
* Promote DB slave to master (this is the only step that results in downtime)
* Add back in DB slaves, using the previous names if applicable

NOTE: This requires a cluster setup -- those using single server setup will need to use one of the other methods.

Where speed is the most important - Disable takeover functionality, then manually reboot all instances simultaneously within the environment.
* Schedule a time where a short duration of downtime can be tolerated
* Edit environment and change the Takeover Preference to Disabled
* Save the environment change, then click Apply.
* Wait for Apply to complete.
* Log into each instance and reboot simultaneously: sudo shutdown -r now
* Wait for the instances to reboot and confirm your app is working
* If there are any issues after the reboot, click apply and re-deploy your app
* Open a support ticket if you need further assistance
* Finally, edit the environment to restore your Takeover Preference to its previous value, save, then click Apply.

If absolute minimum downtime required and fully comfortable with linux process management - Manually kill and restart selective processes running the old software by checking for them using `lsof | grep DEL.*lib.*\.so | awk '{print $2, $1}' | sort -un`

We understand these options are delicate to your operations. If you have questions, issues, or are not comfortable performing the above steps then please let us know via a support request. Our Application Support engineers are available to assist however possible.

January 29, 2015: 1:50pm PT

We have prepared a patch for manual installation, however the Cloud dashboard’s ‘Upgrade’ button can be invoked early next week to automatically install the patch for you.  We advise waiting until this function is ready for use.  This will need to be applied to all existing environments.

For assistance with the manual installation to your environment(s), please open or update your support ticket and an Application Support engineer will be happy to help.  

January 28, 2015: 4:16pm PT

A patch has been prepared and will be ready via the Cloud ‘Upgrade’ button tomorrow after the remaining preparation work has been completed.  We are satisfied with the testing results and the minimal changes to glibc.  

Please note due to the significant impact glibc has on the operating system, customer environments will need to be fully restarted shortly after the upgrade in order to ensure all vulnerable processes and services have been ended and brought up on the patched version.

Further instructions will follow once the patch is deployed.

January 28, 2015: 6:12am PT

Our engineers have made significant progress testing the patch and preparing a deployment plan. We expect to have another update on this event within the next couple hours.

January 27, 2015: 6:40pm PT

We are still working on a tested patch to mitigate this vulnerability. We will continue to update this article as work progresses internally.

January 27, 2015: 2:30pm PT

We are aware of a new vulnerability regarding glibc’s gethostbyname function (CVE-2015-0235).Many involved functions and local services are affected.  We are researching and testing these reports to determine the severity and scope of impact within Engine Yard’s infrastructure and hosted services.  


We will update this post again as details come available.  We recommend subscribing to this feed via the blue ‘Subscribe’ button once logged into Engine Yard’s Developer Center to receive email notifications of updates.  

Zhen Yin Jan 5 News and Notes / Announcements

We are happy to announce that Amazon S3 is now available on the Engine Yard platform.

You can now easily setup AWS S3 cloud storage buckets for use with your Engine Yard application. Simply log into the Engine Yard console and select "Cloud Storage" from the "Tools" drop down. From here you can setup storage buckets and configure users and permissions for all of our supported AWS regions. All charges for usage of AWS S3 will be added to your invoice under the "AWS Other Services". For more information, please see the Knowledge Base document here

 

Jamie Miller May 3, 2013 3 News and Notes / Announcements

We are happy to announce that Engine Yard Cloud customers can now file tickets directly from the Engine Yard Cloud dashboard. 

By clicking on the "File a Ticket" link,  in the right-hand corner, a ticket submission screen will now pop up. 

Screen_Shot_2013-05-01_at_9.23.15_AM.png

You will be able to file your ticket here, and then continue back to your dashboard, without having to navigate to the ticketing system directly. 

Screen_Shot_2013-05-01_at_9.24.21_AM.png

Upon completion of the form, you will receive an email confirmation with your ticket link.  

Note: Don't forget to search our documentation, too! Your question may already have an answer. 

 

Overview | Recent