When envoy is used as the front-end agent, the acquisition of user IP is very important. Generally, the way to obtain IP is different. They are obtained through the X-Forward-For, X-Real-IP or Remote addr attributes in the Header, but what if you ensure that the IP that envoy can obtain is the real user IP? Continue to decrypt this article!

Concept description

• Remote Address
  This is the real address of the client obtained during the TCP connection between nginx and the client. The Remote Address cannot be forged because it requires three handshakes to establish a TCP connection. If the source IP is forged, the TCP connection cannot be established, and there will be no subsequent HTTP requests.
  Generally, when Envoy is the outermost agent, this IP is the real IP client IP

• X-Real-IP
  Is a custom header. X-Real-Ip is usually used by HTTP proxy to represent the IP of the device that generates TCP connection with it. This device may be another proxy or a real requester. X-Real-Ip does not belong to any standard at present. Any custom header can be agreed between the agent and the Web application to pass this information.

• X-Forwarded-For
  X-forward-for is an extension header. HTTP/1.1 (RFC 2616) protocol has no definition of it. It was first introduced by Squid, a cache proxy software, to represent the real IP of the HTTP requester. Now it has become a de facto standard, widely used by major HTTP proxy, load balancing and other forwarding services, and written into RFC 7239 (Forwarded HTTP Extension) standard. In general, X-Forwarded-For can be forged and rewritten using CDN

How to get real IP in Envoy

In Envoy, the client IP is configured as follows:
use_remote_address: the default value is false, set to true, the real remote address of the client connection is used, and false is x-forwarded-for
skip_xff_append: if set to true, the remote address will not be attached to x-forwarded-for
request_headers_to_add add request header
request_headers_to_remove deletes a request header

Preparation of experimental environment configuration

admin:
  access_log_path: /dev/null
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: listener_80
    address:
      socket_address: { address: 0.0.0.0, port_value: 80 }
    access_log:
   
    filter_chains:
    - filters:
      - name: envoy_http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          access_log:
          - name: envoy.listener.accesslog
            typed_config: 
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /var/log/envoy.log
              log_format: 
                text_format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"
          http_filters:
          - name: envoy.filters.http.router
          use_remote_address: true
          skip_xff_append: false
          xff_num_trusted_hops: 0
          stat_prefix: local_route
          codec_type: AUTO
          route_config:
            name: local_route
            #request_headers_to_remove: "X-Forwarded-For"
            request_headers_to_add:
              header: 
                key: "X-Forwarded-For"
                value: "%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%"
                #value: "%REQ(REMOTE_ADDR)%"
              append: true
            virtual_hosts:
            - name: split_traffic
              domains: [ "*" ]
              routes:
              - match: 
                  prefix: "/"
                route:
                  cluster: version_v1
                  request_mirror_policies:
                    cluster: version_v2
                    runtime_fraction:
                      default_value:
                        numerator: 10
                        denominator: HUNDRED
                      runtime_key: routing.request_mirror.version
  clusters:
  - name: version_v1
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: version_v1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: version1, port_value: 90 }
    health_checks:
      timeout: 3s
      interval: 30s
      unhealthy_threshold: 2
      healthy_threshold: 2
      http_health_check:
        path: /ping
        expected_statuses: { start: 200, end: 201 }
  - name: version_v2
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: version_v2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: version2, port_value: 90 }
    health_checks:
      timeout: 3s
      interval: 30s
      unhealthy_threshold: 2
      healthy_threshold: 2
      http_health_check:
        path: /ping
        expected_statuses: { start: 200, end: 201 }

docker-compose

version: '3'
services:
  envoy:
    image: envoyproxy/envoy-alpine:v1.15-latest
    environment: 
    - ENVOY_UID=0
    ports:
    - 80:80
    - 443:443
    - 82:9901
    volumes:
    - ./envoy.yaml:/etc/envoy/envoy.yaml
    networks:
      envoymesh:
        aliases:
        - envoy
    depends_on:
    - webserver1
    - webserver2
    - webserver3
    - webserver4
  
  webserver1:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - version1
    environment: 
    - VERSION=v1
    - COLORFUL=blue
    expose:
    - 90
  webserver2:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - version1
    environment: 
    - VERSION=v1
    - COLORFUL=blue
    expose:
    - 90
  webserver3:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - version2
    environment: 
    - VERSION=v2
    - COLORFUL=red
    expose:
    - 90
  webserver4:
    image: sealloong/envoy-end:latest
    environment: 
    - VERSION=v2
    - COLORFUL=red
    networks:
      envoymesh:
        aliases:
        - version2
    expose:
    - 90
networks:
  envoymesh: {}

External environment when Envoy is actually used as agent

Environment 1: client communicates directly with Envoy

When a normal request is made, the client IP can be normally obtained here. In fact, the value of envy is x-forward-for

Backend log

After forging or rewriting x-forward-for, it is actually the forged value obtained.

When Envoy directly acts as an outer agent, the following parameters can be used. No matter how forged, the corresponding parameters can be obtained.

name: local_route
request_headers_to_remove: "X-Forwarded-For"  # If x-forward-for is a forged value, you can delete it,
request_headers_to_add:   # This value needs to be added because it needs to be passed to the back end after deletion
  header: 
    key: "X-Forwarded-For"
    value: "%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%"  # Get remote_addr, this value cannot be forged. It is an Envoy variable, indicating that the real IP of the downstream host does not add a port, that is, remote_addr no port
  append: true  # Is the surface value appended or overridden

You can see that the real ip obtained by envoy is not a forged request

Environment 2: Envoy has an agent in the front section (no CDN)

In this environment, there are agents on the front end, such as f5 and nginx. Remote cannot be used in this case_ The IP of the front-end agent obtained by addr in this way is not the real IP

f5 or nginx exist in the front end. You can configure irule in f5 to pass real remote_addr, which is replaced by the real client IP, and the front-end proxy rewrites the configuration. You can customize the value.

request_headers_to_remove: "X-Forwarded-For"
request_headers_to_add:
  header: 
    key: "X-Forwarded-For"
    value: "%REQ(custom_header)%"

Environment 3: Envoy has an agent (single CDN) in the front section

In this environment, there are agents in the front end and CDN is used. The way to obtain the customer's real IP for each CDN manufacturer is different. Here, you need to find the CDN manufacturer and find the method to obtain the real IP. Follow step 2.

give an example:
Alicloud cdn method for obtaining real IP
Method for accelerating music to obtain real IP

Environment 4: Envoy has agents (multiple CDN s) in the front segment

Due to the bandwidth, price, usage scenario and other factors of each CDN, multiple CDNs may be used in practice; For example, CDN acceleration is used under normal circumstances, and CDN with high security defense is switched in case of attack. Generally, the price of accelerated CDN is much cheaper than that with defense.

Enovy is to be updated here, and the back-end application can normally obtain IP according to the http header of the CDN

Environment 5: internal agent

No special requirements, no configuration required