1, Nginx reverse proxy usually only does seven layers of load balancing, but now my cluster needs FTP as the file system to upload and download web files, so it needs to be a TCP "four layers proxy". If nginx wants to achieve four layers of load balancing, it needs to add a "- with stream" module.
2, Environment software version preparation
System: CentOS Linux release 7.2.1511 (Core)
Software: Nginx-1.18
3, Installation and compilation environment
yum install -y pcre-devel zlib zlib-devel gcc gcc-c++ make
4, Check the nginx version and installed modules. There is no -- with stream
nginx -V nginx version: nginx/1.18.0 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --prefix=/usr/local/nginx --user=nginx --group=nginx --with-http_stub_status_module --with-http_gzip_static_module --http-proxy-temp-path=/usr/local/nginx/proxy --http-fastcgi-temp-path=/usr/local/nginx/fcgi --http-uwsgi-temp-path=/usr/local/nginx/uwsgi --http-scgi-temp-path=/usr/local/nginx/scgi --with-pcre --with-http_ssl_module
5, My cluster has already done nginx proxy, so I just need to smoothly add the "- with stream" module and switch to the nginx-1.18 source directory for operation
cd /usr/local/nginx/sbin/ #First back up the nginx startup file and shut down the nignx service [root@RS2 sbin]# cp nginx nginx.old [root@RS2 sbin]#systemctl stop nginx [root@RS2 sbin]#cd /root/app/nginx-1.18.0 [root@RS2 nginx-1.18.0]#./configure \ --prefix=/usr/local/nginx \ --user=nginx \ --group=nginx \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-proxy-temp-path=/usr/local/nginx/proxy \ --http-fastcgi-temp-path=/usr/local/nginx/fcgi \ --http-uwsgi-temp-path=/usr/local/nginx/uwsgi \ --http-scgi-temp-path=/usr/local/nginx/scgi \ --with-pcre \ --with-http_ssl_module \ --with-stream [root@RS2 nginx-1.18.0]#make [root@RS2 nginx-1.18.0]cp ./objs/nginx /usr/local/nginx/sbin/nginx
6, vsftpd server configuration will not be released during the installation process. Use yum. Virtual user connection for ftp
[root@serverA vsftpd]# cat vsftpd.conf anonymous_enable=NO local_enable=YES write_enable=YES local_umask=022 dirmessage_enable=YES xferlog_enable=YES connect_from_port_20=YES xferlog_std_format=YES listen=YES listen_ipv6=NO guest_enable=YES guest_username=virtftp pasv_enable=YES pasv_promiscuous=YES #pasv_address=192.168.1.13 pasv_address=193.168.0.128 pasv_min_port=8000 pasv_max_port=8002 max_clients=50 max_per_ip=3 allow_writeable_chroot=YES pam_service_name=vsftpd.vu user_config_dir=/etc/vsftpd/ftp_user userlist_enable=YES tcp_wrappers=YES
pasv_ There are potential safety hazards when promiscuous is closed;
In the actual networking situation, the source IP information can be reserved during Nginx forwarding, but the client and server cannot access it directly, so they can only give up the reservation of the source IP information.
7, Nginx_ In proxy configuration, the stream module is a four-tier load balancing module at the same level as the http module. At first, I didn't know. I mistakenly placed the stream in the http module, resulting in the following error reports
[root@RS2 conf.d]# nginx -t nginx: [emerg] "stream" directive is not allowed here in /usr/local/nginx/conf.d/ftp.conf:1 nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
8, After knowing the reason, we can write the stream in nginx.conf
[root@RS2 nginx]# vim /usr/local/nginx/conf/nginx.conf stream { upstream ftp { server 192.168.1.10:21; } server { listen 1100; #Failed to retry proxy_next_upstream on; proxy_next_upstream_timeout 0; proxy_next_upstream_tries 0; #Timeout configuration proxy_connect_timeout 1s; proxy_timeout 10m; #Speed limit configuration proxy_upload_rate 10240k; proxy_download_rate 20480k; #Upstream server proxy_pass ftp; } upstream ftp_1 { server 192.168.1.10:8000; } server { listen 8000; proxy_pass ftp_1; } upstream ftp_2 { server 192.168.1.10:8001; } server { listen 8001; proxy_pass ftp_2; } upstream ftp_3 { server 192.168.1.10:8002; } server { listen 8002; proxy_pass ftp_3; }
Here is ftp_1,ftp_2,ftp_3. The specific control of data connection (such as speed limit) is not configured
9, Restart the service and start testing the connection without reporting an error. I use the physical machine windows10 to access it and use the web3 host to test the connection
1.windows10
2.web3. To connect, you also need to add a network card in web3, which is and nginx_proxy is the IP address of the same external network segment. Here, it is regarded as an external network host and simulates the external network host to connect to the ftp server
[root@serverC ~]# ftp 193.168.0.128 1100 Connected to 193.168.0.128 (193.168.0.128). 220 (vsFTPd 3.0.2) Name (193.168.0.128:root): ftpadmin 331 Please specify the password. Password: 230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 227 Entering Passive Mode (193,168,0,128,31,66). 150 Here comes the directory listing. -rw-r--r-- 1 0 0 0 Oct 22 08:10 22 -rw-r--r-- 1 0 0 25 Oct 06 02:54 index.html -rw-r--r-- 1 0 0 45 Oct 06 03:07 index.php -rwxr-xr-x 1 0 0 400 Oct 17 08:03 inotifyrsync.sh -rw------- 1 0 0 33642 Oct 23 06:00 nohup.out drwxr-xr-x 12 0 0 4096 Oct 06 03:07 phpMyAdmin drwxr-xr-x 5 1006 1006 4096 Oct 22 21:41 wordpress 226 Directory send OK. ftp>
10, In this way, Nginx can reverse proxy FTP, but it also has disadvantages
There are the following disadvantages:
- Whether the control connection and data connection come from the same connection cannot be verified, and there are potential safety hazards;
- In FTP passive mode, when the range of data ports is wide, it is troublesome to add configuration to Nginx;
- The Nginx agent needs to open more ports, and the security settings such as iptables and firewall are complex.