We know that to implement https, MQTT and other protocols, communication security is required, and the client must support tls certificates.

However, it seems that we don't need to pay attention to tls when we open the computer and mobile browser to visit https website every day?

The answer is that the browser manufacturer has helped us to be compatible, but when you want to achieve https access through embedded IOT devices or MCU, you need to deal with the problem of TLS certificate.

Because the open source code of ESP32 is relatively clear and concise, we will explain the principle today. Taking ESP32 as the hardware platform, other hardware is also a similar process. You can achieve similar results as long as you implement your own process and replace the underlying socket and other related interfaces.

Through the detailed explanation of the key code, this paper explains the implementation principle and process of supporting TLS certificate under ESP32. Taking this principle and process as a reference, users can realize TLS certificate support of other embedded hardware and single chip microcomputer, because the content and code of ESP certificate are visible from the source code, which can be well transplanted and imitated after understanding it.

ESP32 HTTPS demo

See \ examples \ protocols \ HTTPS for official examples_ Request, for example, three ways to support TLS:

    https_get_request_using_crt_bundle();
    https_get_request_using_cacert_buf();
    https_get_request_using_global_ca_store();

The first is the cert bundle method, which is our focus today. The latter two methods are to use the certificate specified by the user. The code is very simple and will not be expanded. Today we mainly introduce the method of cert bundle. After clarifying the cert bundle method, the latter two methods are clear, because the latter two are the simplification of cert bundle method, which simplifies the deserialization, creation, matching and verification of certificates.

Advantages and disadvantages of cert bundle

• Advantages: cert bundle has the advantage that it does not require users to pay attention to certificates and download certificates manually, and automatically supports almost all TLS root certificates in the world.

• Disadvantages: because there are more than 100 built-in certificates, the occupied space should be capitalized. After the actual code test and verification, it is about 60K larger.

ESP32 x509 Certificate Bundle

esp32 supports the implementation of x509 Certificate Bundle for TLS scenarios, which can be simply understood as x509 certificate collection.

For details on x509 Certificate Bundle, please refer to:

https://docs.espressif.com/projects/esp-idf/zh_CN/latest/esp32/api-reference/protocols/esp_crt_bundle.html .

The original text has been made very clear, so we won't repeat it. Today we only focus on the details of the source code.

Brief description of ESP32 TLS implementation process

1. cert bundle deserialization

2. Create an empty certificate

3. In the access callback of https, find the corresponding certificate according to cert name.

4. Verify the validity of the certificate

5. After the verification is passed, a link is established for tls based reading and writing.

6. Close the link and release the tls object

Source code analysis

The whole process is briefly described above. Next, we will analyze and explain the key source code for the above steps.

cert bundle deserialization

Outer call:

if (s_crt_bundle.crts == NULL) {
        ret = esp_crt_bundle_init(x509_crt_imported_bundle_bin_start);
    }

Valuepoint x509_ crt_ imported_ bundle_ bin_ The value corresponding to start is asm("_binary_x509_crt_bundle_start")

This asm object represents a piece of assembly code, which is used to return the data address of x509 object. This object is packaged into binary files through python tools. For the specific assembly content, please refer to x509 in the build directory_ crt_ bundle. S assembly file.

For the implementation of inner layer functions, see the notes for analysis:

static esp_err_t esp_crt_bundle_init(const uint8_t* x509_bundle){
    // Obtain the number of certificates according to the two bytes of the packet header
    s_crt_bundle.num_certs = (x509_bundle[0] << 8) | x509_bundle[1];
    s_crt_bundle.crts      = calloc(s_crt_bundle.num_certs, sizeof(x509_bundle));

    if (s_crt_bundle.crts == NULL) {
        ESP_LOGE(TAG, "Unable to allocate memory for bundle");
        return ESP_ERR_NO_MEM;
    }

    const uint8_t* cur_crt;
    cur_crt = x509_bundle + BUNDLE_HEADER_OFFSET;

    ESP_LOGW(TAG, "cert num:%d", s_crt_bundle.num_certs);

    // The content of each certificate is obtained by offsetting one by one according to the number
    for (int i = 0; i < s_crt_bundle.num_certs; i++) {
        s_crt_bundle.crts[i] = cur_crt;
        // The first 4 bytes of each certificate are length information
        size_t name_len = cur_crt[0] << 8 | cur_crt[1];
        size_t key_len  = cur_crt[2] << 8 | cur_crt[3];
        ESP_LOGW(TAG, "cert name len:%d,key len:%d", name_len, key_len);
        // Offset each certificate by length
        cur_crt = cur_crt + CRT_HEADER_OFFSET + name_len + key_len;
    }

    return ESP_OK;}

The purpose of this code is to deserialize the cert bundle address of the input parameter into memory, define {BUNDLE_HEADER_OFFSET,{name_len,key_len, name,key}} according to the data structure, and deserialize it into the cert structure array object s_crt_bundle.

As of the writing date of this article, the x509 cert bundle implemented by esp32 supports 135 root certificates, almost all certificates in the world.

Create an empty certificate

mbedtls_x509_crt_init(&s_dummy_crt);
mbedtls_ssl_conf_ca_chain(ssl_conf, &s_dummy_crt, NULL);

This empty certificate is mainly used to carry the server certificate characteristics obtained when accessing the server, which will be passed to mbedtls through callback_ ssl_ conf_ Verify function parameters.

Find and verify certificates

The code is as follows. See the notes for analysis:

int esp_crt_verify_callback(void* buf, mbedtls_x509_crt* crt, int depth, uint32_t* flags){
    mbedtls_x509_crt* child = crt;

    /* It's OK for a trusted cert to have a weak signature hash alg.
       as we already trust this certificate */
    uint32_t flags_filtered = *flags & ~(MBEDTLS_X509_BADCERT_BAD_MD);

    if (flags_filtered != MBEDTLS_X509_BADCERT_NOT_TRUSTED) {
        return 0;
    }

    if (s_crt_bundle.crts == NULL) {
        ESP_LOGE(TAG, "No certificates in bundle");
        return MBEDTLS_ERR_X509_FATAL_ERROR;
    }

    ESP_LOGD(TAG, "%d certificates in bundle", s_crt_bundle.num_certs);

    size_t name_len = 0;
    const uint8_t* crt_name;

    // start and end are index es of the certificate array. The binary search algorithm implemented here shows that the name of cert bundle array adds sorting features. For specific details, you need to check the implementation code of python packaging tool.
    bool crt_found = false;
    int start      = 0;
    int end        = s_crt_bundle.num_certs - 1;
    int middle     = (end - start) / 2;

    /* Look for the certificate using binary search on subject name */
    while (start <= end) {
        name_len = s_crt_bundle.crts[middle][0] << 8 | s_crt_bundle.crts[middle][1];
        crt_name = s_crt_bundle.crts[middle] + CRT_HEADER_OFFSET;

        int cmp_res = memcmp(child->issuer_raw.p, crt_name, name_len);
        if (cmp_res == 0) {
            crt_found = true;
            break;
        } else if (cmp_res < 0) {
            end = middle - 1;
        } else {
            start = middle + 1;
        }
        middle = (start + end) / 2;
    }
    // End of binary search

    // Verify the validity of the certificate
    int ret = MBEDTLS_ERR_X509_FATAL_ERROR;
    if (crt_found) {
        size_t key_len = s_crt_bundle.crts[middle][2] << 8 | s_crt_bundle.crts[middle][3];
        ret = esp_crt_check_signature(child, s_crt_bundle.crts[middle] + CRT_HEADER_OFFSET + name_len, key_len);
    }

    if (ret == 0) {
        ESP_LOGI(TAG, "Certificate validated");
        *flags = 0;
        return 0;
    }

    ESP_LOGE(TAG, "Failed to verify certificate");
    return MBEDTLS_ERR_X509_FATAL_ERROR;}

The code is very simple. As stated in the comments, it is a binary search process.

Verify the validity of the certificate

The relevant code has been commented in the above section, which is esp_crt_check_signature function. Verification details, interested friends can view the source code of this function.

TLS read / write process

Write request process:

do {
        ret = esp_tls_conn_write(tls, REQUEST + written_bytes, sizeof(REQUEST) - written_bytes);
        if (ret >= 0) {
            ESP_LOGI(TAG, "%d bytes written", ret);
            written_bytes += ret;
        } else if (ret != ESP_TLS_ERR_SSL_WANT_READ && ret != ESP_TLS_ERR_SSL_WANT_WRITE) {
            ESP_LOGE(TAG, "esp_tls_conn_write  returned: [0x%02X](%s)", ret, esp_err_to_name(ret));
            goto exit;
        }
    } while (written_bytes < sizeof(REQUEST));

esp_tls_conn_write writes circularly according to the length.

Read response process:

    do {
        len = sizeof(buf) - 1;
        bzero(buf, sizeof(buf));
        ret = esp_tls_conn_read(tls, (char*) buf, len);

        if (ret == ESP_TLS_ERR_SSL_WANT_WRITE || ret == ESP_TLS_ERR_SSL_WANT_READ) {
            continue;
        }

        if (ret < 0) {
            ESP_LOGE(TAG, "esp_tls_conn_read  returned [-0x%02X](%s)", -ret, esp_err_to_name(ret));
            break;
        }

        if (ret == 0) {
            ESP_LOGI(TAG, "connection closed");
            break;
        }

        len = ret;
        ESP_LOGD(TAG, "%d bytes read", len);
        /* Print response directly to stdout as it is read */
        for (int i = 0; i < len; i++) {
            putchar(buf[i]);
        }
        putchar('\n'); // JSON output doesn't have a newline at end
    } while (1);

esp_tls_conn_read loop reads until the link is closed and jumps out of the loop

Release tls object:

esp_tls_conn_delete(tls);

end

Well, the whole code process is very clean and clear. If you need to implement TLS certificate support in your own embedded system, you can also follow the above process to implement your own bundle and matching verification process, and you can customize and optimize it according to your actual needs.

I hope the above explanation can help you quickly understand and realize your TLS access function.

If you like the content, pay attention to forwarding and likes. Thank you!