Engine Yard utilizes Phusion Passenger Standalone in conjunction with nginx on Stack V5 (16.06) of the platform. Customers have the option to choose between Passenger5, Puma and Unicorn but this article is focusing on Phusion Passenger Standalone explaining the interconnection between processes and providing useful information during troubleshooting sessions.
Engine Yard's stock nginx
Stock nginx acts as traffic entry point and it's listening on port 8081 by default. There is no SSL handling beyond this step. SSL termination is handled by HA Proxy (by default) on V5, however, SSL termination can also take place on ELB/ALB.
The control script of this server resides in /etc/init.d/nginx and its PID file is /var/run/nginx.pid. There is no monitoring of this server, therefore no monit entry exists.
The relevant processes can be seen below:
root 22319 0.0 0.4 77416 4404 ? Ss 10:47 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
deploy 22321 0.0 0.8 80820 8440 ? S 10:47 0:00 \_ nginx: worker process
deploy 22322 0.0 0.8 80820 8440 ? S 10:47 0:00 \_ nginx: worker process
deploy 22323 0.0 0.8 80820 8440 ? S 10:47 0:00 \_ nginx: worker process
deploy 22324 0.0 0.8 80820 8440 ? S 10:47 0:00 \_ nginx: worker process
deploy 22325 0.0 0.8 80820 8440 ? S 10:47 0:00 \_ nginx: worker process
deploy 22326 0.0 0.8 80820 8440 ? S 10:47 0:00 \_ nginx: worker process
Passenger's nginx + worker
Also known as WebHelper on Passenger's terminology
This process is monitored by monit via an Engine Yard provided monit script. It resides under /etc/monit.d/passenger5_<your-app>.monitrc and its PID file is /data/<APP>/shared/pids/passenger.8000.pid
Relevant processes can be seen below:
deploy 21028 0.0 0.0 30888 868 ? Ss 10:47 0:00 nginx: master process /home/deploy/.passenger/support-binaries/5.1.8/nginx-1.12.1 -c /tmp/passenger-standalone.oyteap/nginx.conf -p /tmp/passenger-standalone.oyteap deploy 21031 0.0 0.4 32596 4512 ? S 10:47 0:00 \_ nginx: worker process
These workers are the final destination of a user's request. They are indirectly controlled and monitored by monit, therefore the commands used to control nginx above will affect passenger's workers.
There is an additional script called "passenger-monitor" which monitors passenger application instances and gracefully restarts any monitored instances exceeding the configured (real) memory limit and kills any orphaned instances. The memory allowance is set using worker allocation guide and any change on it has to be handled by Engine Yard personnel.
Relevant processes can be seen below:
deploy 1845 0.0 9.9 523708 101040 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1854 0.0 9.7 523376 99028 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1863 0.0 9.7 523472 99040 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1872 0.0 9.7 523568 99052 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1881 0.0 9.7 523664 99064 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1890 0.0 9.7 523760 99076 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
A useful command is passenger-status, sample output below (correlate the PIDs below with the output from "ps" command above):
ip-10-255-244-175 ~ # passenger-status
Version : 5.1.8
Date : 2018-08-30 08:27:32 +0000
Instance: MRdfhDjJ (nginx/1.12.1 Phusion_Passenger/5.1.8)
----------- General information -----------
Max pool size : 6
App groups : 1
Processes : 6
Requests in top-level queue : 0
----------- Application groups -----------
App root: /data/todo_ddalianis/current
Requests in queue: 0
* PID: 1845 Sessions: 0 Processed: 1 Uptime: 24m 51s CPU: 0% Memory : 47M Last used: 24m 51s ago
* PID: 1854 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 40M Last used: 24m 51s ago
* PID: 1863 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 33M Last used: 24m 51s ago
* PID: 1872 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 30M Last used: 24m 51s ago
* PID: 1881 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 30M Last used: 24m 51s ago
* PID: 1890 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 30M Last used: 24m 51s ago
Additional processes to consider during troubleshooting
Apart from the nginx and passenger processes analyzed above, there are additional nginx/passenger related processes that one can see using "ps" command. A brief explanation of each one is below.
PassengerAgent is responsible for creating and clearing (on graceful shutdown) of passenger's temporary directory. This is a temporary directory containing files that will be used during the life time of this Phusion Passenger instance.
The relevant process can be seen below and it remains in interruptible sleep (Ss state) while passenger is running.
deploy 21038 0.0 0.1 23536 2004 ? Ss 10:47 0:00 /home/deploy/.passenger/support-binaries/5.1.8/PassengerAgent temp-dir-toucher /tmp/passenger-standalone.oyteap --cleanup --daemonize --pid-file /tmp/passenger-standalone.oyteap/temp_dir_toucher.pid --log-file /data/todo_ddalianis/shared/log/passenger.8000.log
Passenger watchdog, core & ust-router
Passenger's watchdog, core and ust-router are key elements of Phusion Passenger, therefore we will be quoting official documentation here:
deploy 21008 0.0 0.4 367264 5044 ? Ssl 10:47 0:00 Passenger watchdog
deploy 21011 0.1 0.9 587580 10112 ? Sl 10:47 0:07 \_ Passenger core
deploy 21017 0.0 0.6 375692 6132 ? Sl 10:47 0:00 \_ Passenger ust-router
The core and the UstRouter contain complex logic, so they could contain bugs which could crash them. So as a safety measure, they are both monitored by the Watchdog. If either of them crash, they are restarted by the Watchdog. This setup seeks to ensure that the system stays up, no matter what.
The Passenger core is where most of the processing is done. The core keeps track of which application processes currently exist, and using load balancing rules, determines which process a request should be forwarded to. The core also takes care of application spawning: if it determines that having more application processes is necessary or beneficial, then it will make that happen. Process spawning is subject to user-configured limits: the core will never spawn more processes than a user-configured maximum.
The core also has monitoring and statistics gathering capabilities. It constantly keeps track of applications' memory usage, how many requests they’ve handled, etc. This information can later be queried from administration tools. And if an application process crashes, the core restarts it.
The core cooperates with the UstRouter. This latter is responsible for sending data to Union Station, a monitoring web service. If you didn’t explicitly tell Phusion Passenger to send data to Union Station, then the UstRouter sits idle and does not consume resources.
Ust-router is not utilized by Engine Yard.
More information on Phusion Passenger can be found in the project's official Design & Architecture documentation.
Engine Yard utilizes collectd / rrdtool in order to monitor instances. Performance graphs - which can be viewed on environment's dashboard - are being served by a dedicated nginx server listening on port 8989. This sometimes causes confusion that's why we are documenting its existence below.
Monit is also responsible for keeping this collectd nginx running. Querying monit using "sudo monit summary" should show nginx running:
ip-10-255-242-19 ~ # monit summary
Process 'collectd_httpd' Running
Process 'collectd_fcgi' Running
The associated nginx processes can be seen below (notice that this nginx master process is running as root):
root 22536 0.0 0.3 77304 3412 ? Ss 10:47 0:00 nginx: master process /usr/sbin/nginx -c /etc/collectd-httpd/collectd-httpd.conf
deploy 22539 0.0 0.5 77304 5640 ? S 10:47 0:00 \_ nginx: worker process