1. Preface

Recently, a small partner left a message to me in the background, saying that he wrote an Api interface for uploading large files with Django. Now he wants to test the stability of interface concurrency locally and ask me if I have a good scheme

This article takes file upload as an example   Jmeter concurrent execution of Python on script

2. Python realizes file upload

There are three steps to upload large files:

• Get file information and number of slices

• Segment, slice and upload  - API

• File merge  - API

• File path parameterization

2-1 obtaining file information and number of slices

First, get the size of the file

Then, the total number of segments is obtained by using the preset slice size

Finally, get the file name and md5 value

import os
import math
import hashlib

def get_file_md5(self, file_path):
    """Gets the name of the file md5 value"""
    with open(file_path, 'rb') as f:
         data = f.read()
         return hashlib.md5(data).hexdigest()

def get_filename(self, filepath):
    """Gets the original name of the file"""
    # File name with suffix
    filename_with_suffix = os.path.basename(filepath)
    # file name
    filename = filename_with_suffix.split('.')[0]
    # Suffix
    suffix = filename_with_suffix.split('.')[-1]
    return filename_with_suffix, filename, suffix

def get_chunk_info(self, file_path):
    """Get segmentation information"""
    # Get total file size (bytes)
    file_total_size = os.path.getsize(file_path)
    print(file_total_size)

    # Total number of segments
    total_chunks_num = math.ceil(file_total_size / self.chunk_size)
    # File name (with suffix)
    filename = self.get_filename(file_path)[0]
    # md5 value of the file
    file_md5 = self.get_file_md5(file_path)
    return file_total_size, total_chunks_num, filename, file_md5

2-2 slicing and segmented uploading

Using the total number and size of segments, slice the file and call the segment file upload interface

import requests

def do_chunk_and_upload(self, file_path):
    """Process the files in sections and upload them"""
    file_total_size, total_chunks_num, filename, file_md5 = self.get_chunk_info(file_path)

    # ergodic
    for index in range(total_chunks_num):
        print('The first{}File upload'.format(index + 1))
        if index + 1 == total_chunks_num:
            partSize = file_total_size % chunk_size
        else:
            partSize = chunk_size

        # File offset
        offset = index * chunk_size

        # Generate slice id, starting from 1
        chunk_id = index + 1

        print('Start preparing to upload files')
        print("Slice id:", chunk_id, "File offset:", offset, ",Current slice size:", partSize, )

        # Segmented upload file
        self.__upload(offset, chunk_id, file_path, file_md5, filename, partSize, total_chunks_num)

def __upload(self, offset, chunk_id, file_path, file_md5, filename, partSize, total):
    """Upload files in batches"""
    url = 'http://**/file/brust/upload'
    params = {'chunk': chunk_id,
                'fileMD5': file_md5,
                'fileName': filename,
                'partSize': partSize,
                'total': total
                }
    # Read the binary data of the file according to the file path and offset
    current_file = open(file_path, 'rb')
    current_file.seek(offset)

    files = {'file': current_file.read(partSize)}
    resp = requests.post(url, params=params, files=files).text
    print(resp)

2-3 consolidated documents

Finally, the interface of merging files is called, and the large files are segmented into small files.

def merge_file(self, filepath):
        """merge"""
        url = 'http://**/file/brust/merge'
        file_total_size, total_chunks_num, filename, file_md5 = self.get_chunk_info(filepath)
        payload = json.dumps(
            {
                "fileMD5": file_md5,
                "chunkTotal": total_chunks_num,
                "fileName": filename
            }
        )
        print(payload)
        headers = {
            "Content-Type": "application/json"
        }
        resp = requests.post(url, headers=headers, data=payload).text
        print(resp)

2-4 file path parameterization

For concurrent execution, the file upload path is parameterized

# fileupload.py
...
if __name__ == '__main__':
    filepath = sys.argv[1]

    # Size of each slice (MB)
    chunk_size = 2 * 1024 * 1024

    fileApi = FileApi(chunk_size)
    # Segmented upload
    fileApi.do_chunk_and_upload(filepath)

    # merge
    fileApi.merge_file(filepath)

3. Jmeter concurrent execution

Before using Jmeter to create concurrent processes, we need to write batch scripts

When executing a batch script, you need to follow the file path

# cmd.bat

@echo off
set filepath=%1

python  C:\Users\xingag\Desktop\rpc_demo\fileupload.py %*

Then, create a new CSV file locally and write multiple file paths

# Prepare multiple file paths (csv)
C:\\Users\\xingag\\Desktop\\charles-proxy-4.6.1-win64.msi
C:\\Users\\xingag\\Desktop\\V2.0.pdf
C:\\Users\\xingag\\Desktop\\HBuilder1.zip
C:\\Users\\xingag\\Desktop\\HBuilder2.zip

Then, you can use Jmeter to create concurrent processes

The complete steps are as follows:

• Create a test plan and add a thread group below

  Here, the number of thread groups can be consistent with the number of files above

• Under thread group, add synchronization timer

  The number of simulated user groups in the synchronization timer is consistent with the number of parameters above

• Add CSV data file settings

  Point to the csv data file prepared above, set the file format to UTF-8 and the variable name to   file_path. Finally, set the thread sharing mode to "current thread group"

• Add debugging sampler to facilitate debugging

• Add OS process sampler

  Select the batch file created above, and set the command line parameter to ${file_path}

• Add view results

4. Finally

Run the Jmeter concurrent process created above, and you can view the results of concurrent uploaded files in the number of results

Of course, we can increase the number of concurrency to simulate real usage scenarios. We only need to modify the CSV data source and Jmeter parameters.