实践 Nginx 平滑升级流程及失败处理方案

环境阿里云 Centos 6.5 nginx/1.10.3

查看当前Nginx 版本及编译信息 :

[root@xxx nginx]#/usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.3
built by gcc 6.1.0 (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_ssl_module --with-pcre=/usr/src/make/software/pcre-8.38 --add-module=/usr/src/make/software/redis2-nginx-module-master --add-module=../ngx_devel_kit-0.3.0 --add-module=../lua-nginx-module-0.10.8

挑选最新版本nginx (个人使用不考虑稳定性, 使用jishu)

[root@xxx src]#wget http://nginx.org/download/nginx-1.13.9.tar.gz
--2018-03-02 10:57:14--  http://nginx.org/download/nginx-1.13.9.tar.gz
Resolving nginx.org... 206.251.255.63, 95.211.80.227, 2606:7100:1:69::3f, ...
Connecting to nginx.org|206.251.255.63|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 994802 (971K) [application/octet-stream]
Saving to: “nginx-1.13.9.tar.gz”

100%[======================================================================================================================================================================================================================================>] 994,802     84.8K/s   in 16s     

2018-03-02 10:57:32 (59.1 KB/s) - “nginx-1.13.9.tar.gz” saved [994802/994802]

个人习惯安装软件在 /usr/local/src

[root@xxx src]#pwd
/usr/local/src

解压

[root@xxx src]#tar zxvf nginx-1.13.9.tar.gz
[root@xxx src]#cd nginx-1.13.9
[root@xxx nginx-1.13.9]#ll
auto  CHANGES  CHANGES.ru  conf  configure  contrib  html  LICENSE  man  README  src

开始编译老套路中间有插曲编译参数错误了所以生成了Makefile 文件使用 make clean 清理编译文件

[root@xxx nginx-1.13.9]#make clean
rm -rf Makefile objs

开始编译参数

./configure  --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_ssl_module --with-pcre=/usr/src/make/software/pcre-8.38 --add-module=/usr/src/make/software/redis2-nginx-module-master --add-module=../ngx_devel_kit-0.3.0 --add-module=../lua-nginx-module-0.10.8

沮丧又出问题了缺少第三方模块 ngx_devel_kit 和lua 从网上查来安装教程

下载ngx_devel_kit（NDK）模块：https://github.com/simpl/ngx_devel_kit/tags，不需要安装

wget https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1rc1.tar.gz
tar -xzvf v0.3.1rc1.tar.gz

下载最新的lua-nginx-module 模块：https://github.com/openresty/lua-nginx-module/tags，不需要安装

wget https://github.com/openresty/lua-nginx-module/archive/v0.10.12rc2.tar.gz
tar -xzvf v0.10.12rc2.tar.gz

重新编译

./configure  --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_ssl_module  --with-pcre=/usr/src/make/software/pcre-8.38 --add-module=/usr/src/make/software/redis2-nginx-module-master --add-module=/usr/local/src/ngx_devel_kit-0.3.1rc1 --add-module=/usr/local/src/lua-nginx-module-0.10.12rc2

Configuration summary
  + using PCRE library: /usr/src/make/software/pcre-8.38
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

终于ok了可以开始 make 了平滑升级记住只能make 千万别 make install

[root@xxx nginx-1.13.9]#make
make -f objs/Makefile
make[1]: Entering directory `/usr/local/src/nginx-1.13.9'
cd /usr/src/make/software/pcre-8.38 \
    && if [ -f Makefile ]; then make distclean; fi \
    && CC="cc" CFLAGS="-O2 -fomit-frame-pointer -pipe " \
    ./configure --disable-shared
/bin/sh: line 0: cd: /usr/src/make/software/pcre-8.38: No such file or directory
make[1]: *** [/usr/src/make/software/pcre-8.38/Makefile] Error 1
make[1]: Leaving directory `/usr/local/src/nginx-1.13.9'
make: *** [build] Error 2

哦抛错了不要慌看看什么问题

[root@xxx nginx-1.13.9]#cd /usr/src/make/software/pcre-8.38 
-bash: cd: /usr/src/make/software/pcre-8.38: No such file or directory 没有'-'

#把准备好的pcre包解压
[root@xxx software]#tar zxf pcre-8.38.tar.gz

在make 正在运行中我的服务器配置太低如果你是多核 make -j 加速编译速度

make[1]: Leaving directory `/usr/local/src/nginx-1.13.9'

终于编译完了 . 查看objs 目录下生成了新的nginx

[root@xxx nginx-1.13.9]#ls objs/
addon  autoconf.err  Makefile  nginx  nginx.8  ngx_auto_config.h  ngx_auto_headers.h  ngx_modules.c  ngx_modules.o  src

保存并替换nginx

[root@xxx nginx-1.13.9]#mv /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx-20180302
[root@xxx nginx-1.13.9]#cp objs/nginx /usr/local/nginx/sbin/nginx

执行升级失效通知 kill -user2 通知主进程失败正常到这步就结束了但通知主进程失败未生成

/usr/local/nginx/logs/nginx.pid.oldbin 文件老nginx 进程在垂死挣扎

[root@xxx nginx-1.13.9]#make upgrade
/usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
kill -USR2 `cat /usr/local/nginx/logs/nginx.pid`
sleep 1
test -f /usr/local/nginx/logs/nginx.pid.oldbin
make: *** [upgrade] Error 1
You have mail in /var/spool/mail/root

手动试下还是没用中午了吃饭去回来再看看进程死没死

[root@xxx nginx-1.13.9]#kill -USR2 `cat /usr/local/nginx/logs/nginx.pid`
[root@xxxx nginx-1.13.9]#test -f /usr/local/nginx/logs/nginx.pid.oldbin && echo OK!

介绍下`Nginx`的信号控制

TERM,INT 快速关闭
QUIT 从容关闭
HUP 平滑重启，重新加载配置文件
USR1 重新打开日志文件，在切割日志时用途较大
USR2 平滑升级克执行程序
WINCH 从容关闭工作进程

出现异常 USR2 失效了.

最终也没有成功做到平滑升级.

推荐失败后解决方案

先写个简单的监控脚本查看服务器是否断连

#!/bin/bash

while [ 0 ]
    do
        curl -I 'http://localhost:80/'
        sleep 1
    done

尝试多次nginx -s reload 进程ID仍未改变.

采用方案

kill -WINCH #先关闭工作进程后观察脚本输出停止了
kill -HUP #平滑重启观察脚本重新输出但并未做到更新
kill -QUIT #从容关闭观察脚本输出连接错误

`nginx` #重启成功

HTTP/1.1 200 OK
Server: nginx/1.13.9
Date: Fri, 02 Mar 2018 08:41:53 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

总结: 本次平滑升级失败 . 失败原因受个人能力问题未找到.

后记: 模块加载错了我又重新编译安装错误仍然存在. kill -USR2 无效未生成 nginx.pid.oldbin

查看当前Nginx 版本及编译信息 :

介绍下Nginx的信号控制

采用方案

nginx #重启成功

介绍下`Nginx`的信号控制

`nginx` #重启成功