Summary of uploading and receiving egg files

Posted by korngold on Fri, 30 Aug 2019 10:03:19 +0200

There are two methods for egg to get uploaded files. One is to read files directly, and the other is to stream files.

file reading mode

Let's first look at how file is read, which needs to be configured in config

// config.defult.js
config.multipart = {
    mode: 'file'
  };

Control Layer Code

const  fs = require('fs')
const path = require('path')
const Controller = require('egg').Controller;

class HomeController extends Controller {
  async index() {
    const { ctx } = this;
   // console.log(ctx.request.body)
     let file = ctx.request.files[0] // File contains file name, file type, size, path and other information, you can print it to see.
    
     // read file
     let file = fs.readFileSync(file.filepath) //files[0] means to get the first file, which can be traversed if multiple files are uploaded from the front end.
     // Save the file to the specified location
     fs.writeFileSync(path.join('./', `uploadfile/test.png`), file)
    // ctx.cleanupRequestFiles()

   
    ctx.body = { code: 200, message: '', data: file.filename}
  }
}

Front-end upload file code (Vue+axios is used here)

<template>
  <div id="app">
    <img src="./assets/logo.png" @click="testClick">
    <input type="file" @change="upload" ref="fileid" multiple="multiple"/>
    <router-view/>
  </div>
</template>

<script>
import axios from 'axios'
export default {
  name: 'App',
  methods: {
    testClick() {
      console.log('ddd')
    },
    upload() {
      let file = this.$refs.fileid.files[0]
      console.log(file)
      let formData = new FormData()
      formData.append('file', file)

      axios({
        method: 'post',
        url: 'http://127.0.0.1:7001/fileupload',
        data: formData
      }).then(res => {
        console.log(res)
      })
    }
  }
}
</script>

After the above code runs, you can find the upload file in the upload file directory of the egg project.

Stream stream mode

The method of stream stream stream, single file can use getFileStream method to get file stream. Note that using stream stream stream method, we need to delete the multipart in the previous configuration. These two methods can not be used together, otherwise we will report an error.

const  fs = require('fs')
const path = require('path')
const querystring =require('querystring');
const sendToWormhole = require('stream-wormhole');
const Controller = require('egg').Controller;

class HomeController extends Controller {
  async index() {
    const { ctx } = this;

    let stream = await ctx.getFileStream()
    let filename = new Date().getTime() + stream.filename  // The stream object also contains basic information such as file name, size and so on.
 
    // Create File Write Path
    let target = path.join('./', `uploadfile/${filename}`)

    const result = await new Promise((resolve, reject) => {
      // Create a file write stream
      const remoteFileStrem = fs.createWriteStream(target)
      // Write streams in a pipeline fashion
      stream.pipe(remoteFileStrem)

      let errFlag 
      // Listening for error events
      remoteFileStrem.on('error', err => {
        errFlag = true
        // Stop writing
        sendToWormhole(stream)
        remoteFileStrem.destroy()
        console.log(err)
        reject(err)
      })
      
      // Listen for Write Completion Events
      remoteFileStrem.on('finish', () => {
        if (errFlag) return
        resolve({ filename, name: stream.fields.name })
      })
    })

    ctx.body = { code: 200, message: '', data: result }
  }
}

Front-end uploads multiple file codes

upload() {
      let files = this.$refs.fileid.files
      let formData = new FormData()
      // Traversing files
      for (let i = 0; i <files.length; i++) {
        let file = files[i]
        formData.append('file'+ i, file)
      }
     
      axios({
        method: 'post',
        url: 'http://127.0.0.1:7001/fileupload',
        data: formData
      }).then(res => {
        console.log(res)
      })
    }

getFileStream is the method used to upload a file. If multiple files are uploaded, this method can only get one file. Multi-file upload should use the multipart method.

const  fs = require('fs')
const path = require('path')
const querystring =require('querystring');
const Controller = require('egg').Controller;

class HomeController extends Controller {
  async index() {
    const { ctx } = this;

    const parts = ctx.multipart();
    let part;
     while ((part = await parts()) != null) {
      if (part.length) {
        // Processing other parameters
        console.log('field: ' + part[0]);
        console.log('value: ' + part[1]);
        console.log('valueTruncated: ' + part[2]);
        console.log('fieldnameTruncated: ' + part[3]);
      } else {
        if (!part.filename) {
          
          continue;
        }
        // otherwise, it's a stream
        console.log('field: ' + part.fieldname);
        console.log('filename: ' + part.filename);
        console.log('encoding: ' + part.encoding);
        console.log('mime: ' + part.mime);
        let writePath = path.join('./', `uploadfile/${ new Date().getTime() + part.filename}`)
        let writeStrem = fs.createWriteStream(writePath)
        await part.pipe(writeStrem)     
      }
    }

    ctx.body = { code: 200, message: '', data: result }
  }
}

Comparison of two ways

It's obvious that these two methods are much simpler to read file s, but what's the difference between them in terms of performance? How does egg work at the bottom?

Using file reading, we can get the file path, which is used for caching files. You can print it. Uploaded files can be found in this path.

That is to say, the way to read the file is to write the cached file in the server first, and then we read the cached file for operation. In the above file operation, the IO operation of file reading mode includes writing to the cache file, reading to the cache file and writing to the file, a total of three IO operations. The stream stream stream method does not cache the file, that is to say, IO operation only once, if not to write to the server but upload OSS and so on, there is no IO operation. So the effect and performance of these two methods need not be said much.

configuration option

File reading mode can configure the location of cached files and automatic cleaning time because of the file cache.

config.multipart = {
  mode: 'file',
  tmpdir: path.join(os.tmpdir(), 'egg-multipart-tmp', appInfo.name), // Configuration file cache directory
  cleanSchedule: {
   
    cron: '0 30 4 * * *',  // Automatic Clearance Time
  },
};

You can also use the cleanupRequestFiles() method in your code to clean it up directly. Of course, if you choose stream streaming, you don't need to worry about that.

Configuration file type and size

config.multipart = {
   whitelist: [   // Only upload png format is allowed
    '.png',
  ],
  fileSize: '5mb',  // Maximum 5mb  
};

Topics: node.js axios encoding Vue