February 10, 2015: 1:45pm PT
A patch for v2 (2009a) is now available via the methods described in the February 2nd update.
This concludes the planned development work to mitigate risk for CVE-2015-0235. If you have any questions on how to apply the patch to your system, please do not hesitate to inform an Application Support engineer via our ticketing system.
February 6, 2015: 11:36am PT
The v2 (2009a) patch is still progressing through the QA process and is not yet ready for distribution, however we expect a release shortly.
February 2, 2015: 3:20pm PT
The patch is fully ready for customers on 2012.11 to apply to their environments. Customers on 2009a will have a release later this week.
Once the Upgrade button on your environment’s dashboard becomes available, please click it to receive the patch. However, this will not purge the vulnerability from your environment(s). Once the upgrade process is complete, perform one of the following (listed in recommended preference order):
Simple, quick, with some downtime -Terminate and rebuild the environment using the most recent snapshot to fully purge the vulnerable software from all involved instances.
Maximized uptime -
* Cycle Application slave and utility instances (add new instance, then remove the older one, for each existing instance)
* Promote one Application slave to master
* Remove all existing DB slaves, making note of any names
* Add new DB slave, and wait for it to catch up to master
* Promote DB slave to master (this is the only step that results in downtime)
* Add back in DB slaves, using the previous names if applicable
NOTE: This requires a cluster setup -- those using single server setup will need to use one of the other methods.
Where speed is the most important - Disable takeover functionality, then manually reboot all instances simultaneously within the environment.
* Schedule a time where a short duration of downtime can be tolerated
* Edit environment and change the Takeover Preference to Disabled
* Save the environment change, then click Apply.
* Wait for Apply to complete.
* Log into each instance and reboot simultaneously: sudo shutdown -r now
* Wait for the instances to reboot and confirm your app is working
* If there are any issues after the reboot, click apply and re-deploy your app
* Open a support ticket if you need further assistance
* Finally, edit the environment to restore your Takeover Preference to its previous value, save, then click Apply.
If absolute minimum downtime required and fully comfortable with linux process management - Manually kill and restart selective processes running the old software by checking for them using `lsof | grep DEL.*lib.*\.so | awk '{print $2, $1}' | sort -un`
We understand these options are delicate to your operations. If you have questions, issues, or are not comfortable performing the above steps then please let us know via a support request. Our Application Support engineers are available to assist however possible.
January 29, 2015: 1:50pm PT
We have prepared a patch for manual installation, however the Cloud dashboard’s ‘Upgrade’ button can be invoked early next week to automatically install the patch for you. We advise waiting until this function is ready for use. This will need to be applied to all existing environments.
For assistance with the manual installation to your environment(s), please open or update your support ticket and an Application Support engineer will be happy to help.
January 28, 2015: 4:16pm PT
A patch has been prepared and will be ready via the Cloud ‘Upgrade’ button tomorrow after the remaining preparation work has been completed. We are satisfied with the testing results and the minimal changes to glibc.
Please note due to the significant impact glibc has on the operating system, customer environments will need to be fully restarted shortly after the upgrade in order to ensure all vulnerable processes and services have been ended and brought up on the patched version.
Further instructions will follow once the patch is deployed.
January 28, 2015: 6:12am PT
Our engineers have made significant progress testing the patch and preparing a deployment plan. We expect to have another update on this event within the next couple hours.
January 27, 2015: 6:40pm PT
We are still working on a tested patch to mitigate this vulnerability. We will continue to update this article as work progresses internally.
January 27, 2015: 2:30pm PT
We are aware of a new vulnerability regarding glibc’s gethostbyname function (CVE-2015-0235).Many involved functions and local services are affected. We are researching and testing these reports to determine the severity and scope of impact within Engine Yard’s infrastructure and hosted services.
We will update this post again as details come available. We recommend subscribing to this feed via the blue ‘Subscribe’ button once logged into Engine Yard’s Developer Center to receive email notifications of updates.
Severin - does this vulnerability affect environments in the New UI as well? And will the patch next week be available through the Cluster Update mechanism in the New UI as well?
Matt - Excellent question! Any environments built from the new UI are running a version of Ubuntu which is **not** applicable to this vulnerability; there is no need for you to apply updates or reboot your instances.
Severin just to be clear, as of this week if we manually do the upgrade for cloud instances via the upgrade button in the EY cloud GUI will the fix be applied at this point?
Chris, I've just updated the KI to address your question and include more information. Let us know if you have anything else we can assist with!
For some reason, no email came through for the latest update to this article. Last one received was on the 6th ("patch is still progressing through the QA process").