Preface

Because of business needs, nginx is often used as a reverse proxy server to provide http web services, rtmp and hls streaming media services. Although the configuration and use of nginx are well understood, little is known about how nginx is implemented, but it is unknown. As a c++ programmer over 20 years of age, it is obviously a bit of a problem. Starting from this issue, we will update the reading documents of nginx source code from time to time, one is a summary of our own learning, the other is also hope to give you some different interpretations, so that you can understand nginx source code more easily

Tip: The following is the main body of this article. The following cases can be used as reference.

1. Basic concepts of memory pools

The memory pool is a Memory The allocation method is also known as fixed-size-block allocation. Usually we are used to directly apply for memory allocation using API s such as new and malloc. The disadvantage of this is that due to the variable size of the memory blocks being applied for, a large amount of memory fragmentation can occur and performance will be degraded when used frequently. Memory pools are reserved for allocation of a certain number of equally sized blocks of memory before they are actually used. When there is a new memory requirement, a part of the memory block is separated from the memory pool. If there is not enough memory block to continue requesting new memory, then the memory blocks are concatenated and managed in a chain table. A significant advantage of this is that the efficiency of memory allocation is improved.

2. nginx data type monitoring

ngix itself provides a number of custom types whose naming structure rule is NGX (prefix)_ Type keyword_ t (typedef), mainly including basic type, advanced type, communication type, component type, etc.

       

1. The underlying types include ngx_int_t, ngx_string_t, ngx_list_t, ngx_buf_t, ngx_table_elt_t, ngx_chain_t and so on,
2. Some of the more complex advanced types, including ngx_queue_t, ngx_array_t, ngx_rbtree_t, ngx_radix_tree_t, ngx_hash_t and so on.
3. Since nginx itself is a framework for multiprocess collaboration, there are also some types of process communication, including ngx_shm_t, ngx_channel_t, ngx_single_t, ngx_shmtx_t, spin locks, atomic operations, and so on.
4. Another is more complex component types, including ngx_pool_t, ngx_thread_pool_t, ngx_slab_t, ngx_connection_t.

3. Definition of nginx memory pool related structures

Nginx has a complete set of memory pool definitions, applications and operation procedures. Let's first look at the definition of nginx memory pool, which mainly includes memory pool data management structure (ngx_pool_data_t), memory pool data structure (ngx_pool_t), large memory block data structure (ngx_pool_large_t), memory release processing structure (ngx_pool_cleanup_t), etc.

1. Memory pool data management structure

* ngx_ Pool_ Data_ The T structure manages small memory blocks (the size of the requested memory is smaller than the max in the ngx_pool_s structure)

typedef struct {
    u_char               *last; //Where the current memory allocation ends, the next available remaining memory start address
    u_char               *end;  //End location of memory pool
    ngx_pool_t           *next;  //Chain list of memory blocks contained in the entire memory pool, pointing to the next memory block
    ngx_uint_t            failed;   //Number of failed allocations that do not meet the requirements
} ngx_pool_data_t;

• Each request for small memory adjusts the last pointer value once. The adjusted value is last + the size of the last request and is not greater than the end.
• End is the end of the memory block. If the new requested memory size plus last is greater than end, a new memory block needs to be requested and the next point is at the start of the new memory block.
• The value failed is a bit confusing and starts to understand that memory requests fail, which should actually be the count of memory used for new requests when there is not enough space left in the memory blocks, as detailed in the next section.
• The sizeof(ngx_pool_data_t) size behaves differently in different system architectures, 16 bytes in 32-bit systems and 32 bytes in 64-bit systems. (

2. Memory pool data structure

The primary responsibility is to manage the memory block as a whole, whether it is so-called large or small, while using the current pointer to optimize the efficiency of data queries in the case of multiple memory blocks. A large memory block is a list of memory chains whose contents are irregular in size greater than the max value that the variable largepoints to, and a small block is the ngx_of the rule that current points to Pool_ S Memory area header address

typedef struct ngx_pool_s            ngx_pool_t;
struct ngx_pool_s{
    ngx_pool_data_t       d;    //data block
    size_t                max;  //Maximum available value of memory pool block s
    ngx_pool_t           *current;  //Address of current memory pool block
    ngx_chain_t          *chain;    //The pointer is attached to a ngx_chain_t-structure
    ngx_pool_large_t     *large;    //Address in bulk, that is, memory requests in bulk that exceed max
    ngx_pool_cleanup_t   *cleanup;  //callback operation function to free memory pool
    ngx_log_t            *log;      //log information
};

• First clarify ngx_pool_s and ngx_pool_t is one thing
• The structure variable d is a member variable of the memory pool data management structure (ngx_pool_data_t), which has been explained above and is not covered in detail.
• The unsigned reshaping Max is the size of the valid memory area available and used. The memory size of the join request is 1024, the max is 1024-sizeof(ngx_pool_t). Under 32 systems, the size of sizeof(ngx_pool_t) is 40 bytes, of which 16 bytes is the size of sizeof(ngx_pool_data_t), and ngx_pool_t is the size of sizeof sizeof(ngx_pool_data_t). Pool_ Data_ T contains four variables plus ngx_ Pool_ T itself has six variables, exactly 10 variables, each 4 bytes and 10 exactly 40 bytes. For 32-bit cases, max=1024-40=884, while for 64-bit cases, sizeof(ngx_pool_t) takes 80 and bytes, for 64-bit cases, max=1024-40=844.
• The pointer current points to its starting position when the memory block is less than 6. If there are more than 6 memory blocks, such as n existing memory blocks, the current points to the starting position of the n-6 memory blocks. The purpose of this is to improve the efficiency of searching by not traversing from the beginning when looking for memory variables. The reason for 6 is explained in detail below.
• The pointer chain can be interpreted as a temporary cache, which normally points to a buff buffer when in practice.
• The pointer larges to a memory chain table of irregular size that is larger than the max value, unlike the standard memory block that exists for large memory blocks, which is the memory size of the max tag, the valid area is between the last of the structure variable D and the memory address area that the end pointer points to, which is d.end-d.last.
• The pointer cleanup points to the callback operation function that frees the memory pool.

3. Large Memory Block Data Structure

typedef struct ngx_pool_large_s  ngx_pool_large_t;
struct ngx_pool_large_s {
    ngx_pool_large_t     *next;
    void                 *alloc;
};

* ngx_pool_large_s itself is simpler, a standard chain table, a pointer to the next node and a pointer to the starting address of the memory area requested by this node.

4. Memory Release Processing Structure

typedef void (*ngx_pool_cleanup_pt)(void *data);
typedef struct ngx_pool_cleanup_s  ngx_pool_cleanup_t;
struct ngx_pool_cleanup_s {
    //Is a function pointer to a function that releases the corresponding resources of the data. The only parameter to this function is data
    ngx_pool_cleanup_pt   handler;
    
    //Point to the data to be cleared 
    void                 *data;
    ngx_pool_cleanup_t   *next;
};

4. Operation of memory pool

The operations of the memory pool mainly include creating, using, resetting and releasing, and the main interface functions are as follows

1. Memory pool creation

ngx_pool_t * ngx_create_pool(size_t size, ngx_log_t *log)
{
    ngx_pool_t  *p;
	
    //Request size bytes of memory with 16-bit alignment
    p = ngx_memalign(NGX_POOL_ALIGNMENT, size, log);
    if (p == NULL) {
        return NULL;
    }
    //Skip sizeof(ngx_pool_t) from a new request for memory where p->d.last is the starting address for the memory area that can be used
    p->d.last = (u_char *) p + sizeof(ngx_pool_t);
    //Maximum boundary of memory region for this request
    
    p->d.end = (u_char *) p + size;
    
    //Starting position of next memory block
    p->d.next = NULL;
    
    //The number of times that memory usage does not meet the next time (0 this time, 1 next time, count starts at least twice) and must be reapplied
    p->d.failed = 0;

    size = size - sizeof(ngx_pool_t);
	
    //Set the effective memory area size if it is larger than NGX_MAX_ALLOC_FROM_POOL, NGX_MAX_ALLOC_FROM_POOL, size if less than
    p->max = (size < NGX_MAX_ALLOC_FROM_POOL) ? size : NGX_MAX_ALLOC_FROM_POOL;

    p->current = p;
    p->chain = NULL;
    p->large = NULL;
    p->cleanup = NULL;
    p->log = log;

    return p;
}

Ngx_ Create_ The main pool process consists of two steps

1. Specifies the size of the memory block in the requested memory pool. And through ngx_memalign actually applies for the memory region, which is NGX_POOL_ALIGNMENT alignment, NGX_POOL_ALIGNMENT must be a power of 2, aligned at 8 bytes on 32-bit systems, 16 bytes on 64-bit systems, and nginx's default NGX_POOL_ALIGNMENT is 16, ngx_memalign also calls posix_further Memalign function for memory request, posix_ Exceptional results returned by memalign NGX_if EINVAL POOL_ ALIGNMENT is not a power of 2, if ENOMEM means insufficient memory. If the system does not support posix_memalign, the system will call memalign to request memory.
2. Memory and ngx_for new applications after successful applications Pool_ T Pointer to associate and initialize ngx_pool_t object, ngx_pool_t takes up the first 40 bytes (32 bits) or 80 bytes (64 bits) of the newly requested memory area, followed by the actual available memory area. Take a 64-bit system for example, create a memory pool object with 1024 bytes of size for a memory block and initialize the memory snapshot as follows:

* About NGX_MAX_ALLOC_FROM_POOL is very verbose. The nginx source defines it as #define NGX_MAX_ALLOC_FROM_POOL (ngx_pagesize - 1), where ngx_ Page size refers to the size of the memory page. In the x86(32/64 bit) architecture, the size of the memory page is 4096 bytes, or 4K. The purpose of this is to request a block size of no more than 4096 bytes, which will increase the hit rate of the memory cache when requesting memory to be concentrated in one memory page.

2. Use of memory pools

If the required memory is less than pool->max, apply for a small memory block, otherwise apply for a large memory block

void * ngx_palloc(ngx_pool_t *pool, size_t size)
{
    //If the required memory is less than pool->max, apply for a small memory block, otherwise apply for a large memory block
    #if !(NGX_DEBUG_PALLOC)
    {
        if (size <= pool->max) {	
        return ngx_palloc_small(pool, size, 1);
    }
    #endif

    return ngx_palloc_large(pool, size);
}

3. Application for small memory blocks

//Small Memory Block Request
static ngx_inline void * ngx_palloc_small(ngx_pool_t *pool, size_t size, ngx_uint_t align)
{
    u_char      *m;
    ngx_pool_t  *p;
	
	//Record the current pointer of the first memory block in the memory pool
    p = pool->current;

    do {
        m = p->d.last;
		
		//Alignment is guaranteed if alignment is required
        if (align) {
            m = ngx_align_ptr(m, NGX_ALIGNMENT);
        }
		//If the remaining memory space (p->d.end-m) is larger than the required memory space, return the starting address and modify the last direction to reduce the size of the remaining memory space.
        if ((size_t) (p->d.end - m) >= size) {
            p->d.last = m + size;

            return m;
        }
		//Current memory block is too small to meet the memory request, swap to another memory block
        p = p->d.next;

    } while (p);
	//If you have traversed all the nodes that have been requested or if you have not found available memory space, request a new memory block
    return ngx_palloc_block(pool, size);
}

static void *ngx_palloc_block(ngx_pool_t *pool, size_t size)
{
    u_char      *m;
    size_t       psize;
    ngx_pool_t  *p, *new;
	//Requests for a new memory block are made according to the size of the first block requested, which may be a value specified by 4095 or another size.
    psize = (size_t) (pool->d.end - (u_char *) pool);
	//Request memory with 16-bit alignment
    m = ngx_memalign(NGX_POOL_ALIGNMENT, psize, pool->log);
    if (m == NULL) {
        return NULL;
    }
	//Initializing a new requested memory block actually has some value except for the initialization structure of the first memory block. The later value is actually not too big and wastes a little memory space.
    new = (ngx_pool_t *) m;

    new->d.end = m + psize;
    new->d.next = NULL;
    new->d.failed = 0;
    m += sizeof(ngx_pool_data_t);
    m = ngx_align_ptr(m, NGX_ALIGNMENT);
    new->d.last = m + size;
	
    //Loop start sets d.failed value to correct pool->current value
    //The value of p->d.next for the first entry is null, so a for loop can only start if the memory pool has more than two memory blocks. That is, you start requesting the third memory block to enter the for loop, because the first one is ngx_ Creat_ Created by the poot function, the second is the first time you enter ngx_ Palloc_ The block function is created and a chain list of p->d.next is initially constructed at the end of the function, but the number of times does not satisfy the for loop entry condition, and the second time enters ngx_ Palloc_ The block function opens the for loop. The conditional statement p->d.failed++>4 requires that the value of p->d.failed be 5, p->d.failed++ be 6 before entering pool->current = p->d.next, that is, adding the first time that does not satisfy the cycle requirements and requires six consecutive entries into ngx_ Palloc_ The block function can only change the value of pool->current, where the first ngx_is included in the memory pool Creat_ A total of 7 memory blocks created by poot via ngx_ The change in value of memalign block, p->d.failed from the first to the most recently applied block is 6, 4, 3, 2, 1, 0, 0, p->next points to p->d.failed of 4, but the loop does not end at this time, and subsequent p->d.failed continues to add 1 until ngx_ Palloc_ The p->d.failed values for all memory blocks at the end of the block function execution are 6, 5, 4, 3, 2, 1, and 0, respectively.
    for (p = pool->current; p->d.next; p = p->d.next) {
        if (p->d.failed++ > 4) {
            pool->current = p->d.next;
        }
    }

    p->d.next = new;

    return m;
}

Following the program's processing logic, a memory snapshot of nine memory blocks after eight scales is shown below:

4. Application for Large Memory Block

static void *
ngx_palloc_large(ngx_pool_t *pool, size_t size)
{
    void              *p;
    ngx_uint_t         n;
    ngx_pool_large_t  *large;

    p = ngx_alloc(size, pool->log);
    if (p == NULL) {
        return NULL;
    }

    n = 0;
    //In ngx_ Pool_ Find idle ngx_in largechain of large memory nodes in t Pool_ Larger node. If you find ngx_that hangs a large amount of memory on this node Pool_ Larger queue
    for (large = pool->large; large; large = large->next) {
        if (large->alloc == NULL) {
            large->alloc = p;
            return p;
        }
        //Find the number of free nodes no more than five times. Abandon if more than five nodes fail to find an idle node
        if (n++ > 3) {
            break;
        }
    }
    //If no idle largenode is found more than five times, create a new ngx_pool_large_t
    large = ngx_palloc_small(pool, sizeof(ngx_pool_large_t), 1);
    if (large == NULL) {
        ngx_free(p);
        return NULL;
    }

    large->alloc = p;
    large->next = pool->large;
    pool->large = large;

    return p;
}

5. Release of memory pool

void ngx_destroy_pool(ngx_pool_t *pool)
{
    ngx_pool_t          *p, *n;
    ngx_pool_large_t    *l;
    ngx_pool_cleanup_t  *c;

    for (c = pool->cleanup; c; c = c->next) {
        if (c->handler) {
            ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0,
                           "run cleanup: %p", c);
            c->handler(c->data);
        }
    }

#if (NGX_DEBUG)

    /*
     * we could allocate the pool->log from this pool
     * so we cannot use this log while free()ing the pool
     */

    for (l = pool->large; l; l = l->next) {
        ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0, "free: %p", l->alloc);
    }

    for (p = pool, n = pool->d.next; /* void */; p = n, n = n->d.next) {
        ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, pool->log, 0,
                       "free: %p, unused: %uz", p, p->d.end - p->d.last);

        if (n == NULL) {
            break;
        }
    }

#endif
    //Focus on starting at this location, looping frees up all large memory blocks
    for (l = pool->large; l; l = l->next) {
        if (l->alloc) {
            ngx_free(l->alloc);
        }
    }
    //Loop all small memory blocks
    for (p = pool, n = pool->d.next; /* void */; p = n, n = n->d.next) {
        ngx_free(p);

        if (n == NULL) {
            break;
        }
    }
}

5. Code examples

1. Development environment

All nginx source code analysis is based on nginx version 1.12.1 and VirtualBox 6. 1 Deployed 64-bit ubuntu16.1 system, to avoid the impact of permissions, use the root account for all operations, basic and root way to log on centos is no different. The development tool uses vscode, installs remote ssh plug-in, makes editing, compiling and debugging more convenient remotely

2. Code Modification

In file src/core/ngx_ Palloc. Function ngx_of C Palloc_ Modification within the block to test changes in memory allocation

 for (p = pool->current; p->d.next; p = p->d.next) {
        if (p->d.failed++ > 4) {
            pool->current = p->d.next;
        }        
    }

    p->d.next = new;

    //Add a new test line
    for (p = pool; p->d.next; p = p->d.next) {
        printf("p->d.failed:%d\r\n",(int)p->d.failed);
    }
        printf("p->d.failed:%d\r\n",(int)p->d.failed);
    }
    printf("--------------------------\r\n");

The main thing is that the modifications here will need to recompile nginx before later examples will work. Ngx_is not used in another example Log_ T, in order to compile pass and decouple more modules, ngx_is involved in the program Log_ Annotating the contents of the error function does not work, as does recompiling nginx to generate a new one. o file only

3. Test demo

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "ngx_config.h"
#include "ngx_core.h"

#include "ngx_list.h"
#include "ngx_palloc.h"
#include "ngx_string.h"

#define N 10

void print_list(ngx_list_t *l)
{
    ngx_list_part_t *p = &(l->part);
    while(p)
    {
        for(int i = 0 ; i <  p->nelts ;i++)
        {
            char* data = ((ngx_str_t*)p->elts+i)->data;
            printf("%s\n",data);
        }

        p = p->next;
        printf("------------------\n");
    }
}

int main()
{
    ngx_pool_t *pool = NULL;
    pool = ngx_create_pool(5000,0);
    printf("ngx_pagesize 0 pool->max Be equal to:%ld\r\n",pool->max);
    ngx_destroy_pool(pool);

    ngx_pagesize = 4096;
    pool = ngx_create_pool(5000,0);
    printf("ngx_pagesize Equal to 4096 pool->max Be equal to:%ld\r\n",pool->max);
    ngx_list_t* l = ngx_list_create(pool,N,sizeof(ngx_str_t));
    for(int  i = 0;i < 80 ;i++)
    {
        ngx_str_t *ptr = ngx_list_push(l);
        char *buf = ngx_palloc(pool,512);
        sprintf(buf,"demo:%d\n",i+1);
        ptr->len = strlen(buf);
        ptr->data = buf;
    }

    //print_list(l);
    ngx_destroy_pool(pool);   
    
    return 1;
}

4. Compile commands

gcc -o pool ngx_pool_demo.c -I./core/ -I./os/unix/ -I../objs/ -I../objs/src/core ../objs/src/core/ngx_palloc.o ../objs/src/core/ngx_list.o ../objs/src/os/unix/ngx_alloc.o -g

5. Operational effectiveness