Engine Yard Developer Center

Passenger + nginx on Engine Yard's stack V5 (16.06)

Engine Yard utilizes Phusion Passenger Standalone in conjunction with nginx on Stack V5 (16.06) of the platform. Customers have the option to choose between Passenger5, Puma and Unicorn but this article is focusing on Phusion Passenger Standalone explaining the interconnection between processes and providing useful information during troubleshooting sessions.

passenger5_nginx_stackv5.png 

Engine Yard's stock nginx

Stock nginx acts as traffic entry point and it's listening on port 8081 by default. There is no SSL handling beyond this step. SSL termination is handled by HA Proxy (by default) on V5, however, SSL termination can also take place on ELB/ALB.

The control script of this server resides in /etc/init.d/nginx and its PID file is /var/run/nginx.pid. There is no monitoring of this server, therefore no monit entry exists.

The relevant processes can be seen below:

root     22319  0.0  0.4  77416  4404 ?        Ss   10:47   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
deploy   22321  0.0  0.8  80820  8440 ?        S    10:47   0:00  \_ nginx: worker process
deploy   22322  0.0  0.8  80820  8440 ?        S    10:47   0:00  \_ nginx: worker process
deploy   22323  0.0  0.8  80820  8440 ?        S    10:47   0:00  \_ nginx: worker process
deploy   22324  0.0  0.8  80820  8440 ?        S    10:47   0:00  \_ nginx: worker process
deploy   22325  0.0  0.8  80820  8440 ?        S    10:47   0:00  \_ nginx: worker process
deploy   22326  0.0  0.8  80820  8440 ?        S    10:47   0:00  \_ nginx: worker process

 

Passenger's nginx + worker

Also known as WebHelper on Passenger's terminology

This process is monitored by monit via an Engine Yard provided monit script. It resides under /etc/monit.d/passenger5_<your-app>.monitrc and its PID file is /data/<APP>/shared/pids/passenger.8000.pid 

Relevant processes can be seen below: 

deploy   21028  0.0  0.0  30888   868 ?        Ss   10:47   0:00 nginx: master process /home/deploy/.passenger/support-binaries/5.1.8/nginx-1.12.1 -c /tmp/passenger-standalone.oyteap/nginx.conf -p /tmp/passenger-standalone.oyteap
deploy   21031  0.0  0.4  32596  4512 ?        S    10:47   0:00  \_ nginx: worker process

 

Passenger workers

These workers are the final destination of a user's request. They are indirectly controlled and monitored by monit, therefore the commands used to control nginx above will affect passenger's workers. 

There is an additional script called "passenger-monitor" which monitors passenger application instances and gracefully restarts any monitored instances exceeding the configured (real) memory limit and kills any orphaned instances. The memory allowance is set using worker allocation guide and any change on it has to be handled by Engine Yard personnel.

Relevant processes can be seen below:

deploy 1845 0.0 9.9 523708 101040 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1854 0.0 9.7 523376 99028 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1863 0.0 9.7 523472 99040 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1872 0.0 9.7 523568 99052 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1881 0.0 9.7 523664 99064 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)
deploy 1890 0.0 9.7 523760 99076 ? Sl 08:02 0:00 Passenger RubyApp: /data/todo_ddalianis/current/public (production)

A useful command is passenger-status, sample output below (correlate the PIDs below with the output from "ps" command above):

ip-10-255-244-175 ~ # passenger-status
Version : 5.1.8
Date : 2018-08-30 08:27:32 +0000
Instance: MRdfhDjJ (nginx/1.12.1 Phusion_Passenger/5.1.8)
----------- General information -----------
Max pool size : 6
App groups : 1
Processes : 6
Requests in top-level queue : 0
----------- Application groups -----------
/data/todo_ddalianis/current/public (production):
App root: /data/todo_ddalianis/current
Requests in queue: 0
* PID: 1845 Sessions: 0 Processed: 1 Uptime: 24m 51s CPU: 0% Memory : 47M Last used: 24m 51s ago
* PID: 1854 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 40M Last used: 24m 51s ago
* PID: 1863 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 33M Last used: 24m 51s ago
* PID: 1872 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 30M Last used: 24m 51s ago
* PID: 1881 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 30M Last used: 24m 51s ago
* PID: 1890 Sessions: 0 Processed: 0 Uptime: 24m 51s CPU: 0% Memory : 30M Last used: 24m 51s ago

 

Additional processes to consider during troubleshooting

Apart from the nginx and passenger processes analyzed above, there are additional nginx/passenger related processes that one can see using "ps" command. A brief explanation of each one is below.

 

Passenger Agent

PassengerAgent is responsible for creating and clearing (on graceful shutdown) of passenger's temporary directory. This is a temporary directory containing files that will be used during the life time of this Phusion Passenger instance.

The relevant process can be seen below and it remains in interruptible sleep (Ss state) while passenger is running.

deploy   21038  0.0  0.1  23536  2004 ?        Ss   10:47   0:00 /home/deploy/.passenger/support-binaries/5.1.8/PassengerAgent temp-dir-toucher /tmp/passenger-standalone.oyteap --cleanup --daemonize --pid-file /tmp/passenger-standalone.oyteap/temp_dir_toucher.pid --log-file /data/todo_ddalianis/shared/log/passenger.8000.log

 

Passenger watchdog, core & ust-router

Passenger's watchdog, core and ust-router are key elements of Phusion Passenger, therefore we will be quoting official documentation here:

deploy   21008  0.0  0.4 367264  5044 ?        Ssl  10:47   0:00 Passenger watchdog
deploy   21011  0.1  0.9 587580 10112 ?        Sl   10:47   0:07  \_ Passenger core
deploy   21017  0.0  0.6 375692  6132 ?        Sl   10:47   0:00  \_ Passenger ust-router
 
 

Passenger watchdog

The core and the UstRouter contain complex logic, so they could contain bugs which could crash them. So as a safety measure, they are both monitored by the Watchdog. If either of them crash, they are restarted by the Watchdog. This setup seeks to ensure that the system stays up, no matter what.

 
 

Passenger core

The Passenger core is where most of the processing is done. The core keeps track of which application processes currently exist, and using load balancing rules, determines which process a request should be forwarded to. The core also takes care of application spawning: if it determines that having more application processes is necessary or beneficial, then it will make that happen. Process spawning is subject to user-configured limits: the core will never spawn more processes than a user-configured maximum.

The core also has monitoring and statistics gathering capabilities. It constantly keeps track of applications' memory usage, how many requests they’ve handled, etc. This information can later be queried from administration tools. And if an application process crashes, the core restarts it.

ust-router

The core cooperates with the UstRouter. This latter is responsible for sending data to Union Station, a monitoring web service. If you didn’t explicitly tell Phusion Passenger to send data to Union Station, then the UstRouter sits idle and does not consume resources. 

Ust-router is not utilized by Engine Yard.

 
 

More information on Phusion Passenger can be found in the project's official Design & Architecture documentation.

 

Collectd

Engine Yard utilizes collectd / rrdtool in order to monitor instances. Performance graphs  - which can be viewed on environment's dashboard - are being served by a dedicated nginx server listening on port 8989. This sometimes causes confusion that's why we are documenting its existence below.

Monit is also responsible for keeping this collectd nginx running. Querying monit using "sudo monit summary" should show nginx running: 

ip-10-255-242-19 ~ # monit summary
Process 'collectd_httpd' Running
Process 'collectd_fcgi' Running

The associated nginx processes can be seen below (notice that this nginx master process is running as root):

root     22536  0.0  0.3  77304  3412 ?        Ss   10:47   0:00 nginx: master process /usr/sbin/nginx -c /etc/collectd-httpd/collectd-httpd.conf
deploy   22539  0.0  0.5  77304  5640 ?        S    10:47   0:00  \_ nginx: worker process
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Please sign in to leave a comment.

Powered by Zendesk