Introduction to FastDFS and its architecture:

• FastDFS: it is an open source distributed file system implemented in C language. It manages files. Its functions include file storage, file synchronization, file access (file upload, file download), etc. It solves the problems of mass storage and load balancing. It is especially suitable for online services based on documents, such as photo album websites, video websites and so on.
• The FastDFS server has two roles: tracker and storage node.
  • tracker is used to track files, which is equivalent to an index of files.
  • The storage node is used to store files, including files and file attributes (meta data), which are saved to the storage server disk to complete all functions of file management: file storage, file synchronization and file access. The uploaded file is finally saved on storage, and the metadata information of the file is saved on Tracker. The load balancing of storage can be realized through Tracker.
  • Storage is generally built into a cluster. A Storage Cluster can be composed of multiple groups without communication between different groups. A group is equivalent to a small cluster. The group is composed of multiple storage servers. The storage servers in the group will synchronize files through connection to ensure high availability.

The steps for Centos7.x to install FastDFS are as follows:

• Install GCC environment: yum install gcc-c++
• Install dependent library libevent: yum -y install libevent
• Install the dependency library libfastcommon, copy the downloaded libfastcommon-1.0.36.tar.gz package to the / usr/local / directory, and then execute the following commands in sequence:

cd /usr/local

tar -zxvf libfastcommon-1.0.36.tar.gz

cd libfastcommon-1.0.36

./make.sh

./make.sh install

• Note: Tracker and Storage are the same installation package, that is, the fastdfs-5.11.tar.gz compressed package provided. Copy it to the / usr/local / directory, and then execute the following commands in sequence:

cd /usr/local

tar -zxvf fastdfs-5.11.tar.gz

cd fastdfs-5.11

./make.sh

./make.sh install

• After successful installation, execute the following command to copy the configuration file under the conf directory in the installation directory to the / etc/fdfs / Directory:

cd conf/

cp ./* /etc/fdfs/

• Next, enter the / etc/fdfs / directory to configure the parameters in several files. Note: the metadata storage directory should be created in advance (i.e. the following base_path)!
• vim tracker.conf, modify several parameter values:

port=22122

base_path=/opt/fastdfs #Here, I do not distinguish between tracker and storage, but store them in the same directory

http.server_port=443 #If https is not used for access, this parameter is ignored

• vim storage.conf, modify several parameter values:

base_path=/opt/fastdfs #Here, I do not distinguish between tracker and storage, but store them in the same directory

store_path0=/opt/fastdfs #The absolute path of the storage file is: $store_path0/data/

tracker_server=Server host extranet ip:22122 # Only one tracker server is used here

http.server_port=443 #If https is not used for access, this parameter is ignored

• Start Tracker and Storage services respectively:

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start

Start successful

• Next, install Nginx in two steps: ① install Nginx; ② Install fastdfs Nginx module under Storage.
• ① Download Nginx dependency package: wget http://nginx.org/download/nginx-1.17.0.tar.gz , copy and unzip the nginx-1.17.0.tar.gz compressed package to the / opt / directory. Before compiling and installing, you also need to install two dependencies, that is, execute the following commands in sequence. After installation, the default installation location is: / usr/local/nginx/sbin/nginx. Notes on Nginx tutorial: Click me to reach

tar -zxvf nginx-1.17.0.tar.gz

cd nginx-1.17.0

yum -y install pcre-devel

yum -y install openssl openssl-devel

./configure

make

make install

• ② Copy fastdfs-nginx-module-master.zip to the / usr/local directory, and then execute the following commands in sequence:

cd /usr/local

unzip fastdfs-nginx-module-master.zip

mv fastdfs-nginx-module-master fastdfs-nginx-module # Change folder name

cp /usr/local/fastdfs-nginx-module/src/mod_fastdfs.conf  /etc/fdfs/ # Set Mod_ Copy the fastdfs.conf configuration file to the / etc/fdfs / directory

• vim /etc/fdfs/mod_fastdfs.conf, modify several parameter values:

tracker_server=Server extranet ip:22122

url_have_group_name = true # Whether the url contains the group name. It is enabled by default

store_path0=/opt/fastdfs #The file storage directory is the same as before by default

• Next, go back to the unzipped directory of the nginx installation file downloaded in the first step / opt/nginx-1.17.0 /, execute the following command to reconfigure the compilation and installation, which is divided into two versions: http access version and https access version:
• http access version:

./configure --add-module=/usr/local/fastdfs-nginx-module/src

make

make install

• https access version. Note: back up the original nginx startup command in advance to prevent configuration errors for restore:

./configure --prefix=/usr/local/nginx/ --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_ssl_module --add-module=/usr/local/fastdfs-nginx-module/src

make

# If you are from the http version, you do not need to execute the following installation command. You only need to survive the new nginx startup item

make install 

# If it comes from the http version, you need to copy the newly existing nginx to the running directory, otherwise it will not be executed.

cp objs/nginx   /usr/local/nginx/sbin/nginx

• Verifying the Nginx configuration file:. / nginx -t

Verification successful

• After installation, modify the configuration file of nginx. For http access version, you can omit root /opt/fastdfs; This line, but if it's https access version, it's best to add it! Finally, start nginx:. / nginx.

Configure fastdfs file request forwarding

Force http forwarding to https

Simply configure ssl certificates

• Springboot + fastdfs actual combat case:
• Add pom dependency:

<dependency>

  <groupId>net.oschina.zcx7878</groupId>

  <artifactId>fastdfs-client-java</artifactId>

  <version>1.27.0.0</version>

</dependency>

• Add the FastDFS configuration file fastdfs-client.properties under the resources directory of the project, as follows:

fastdfs.connect_timeout_in_seconds=5

fastdfs.network_timeout_in_seconds=30

fastdfs.charset=UTF-8

fastdfs.http_anti_steal_token=false #Disable the anti-theft chain. The relevant configuration is in / etc/fdfs/http.conf

fastdfs.http_secret_key=FastDFS1234567890 # The default is the same as the configuration file. If the anti-theft chain is opened, the key must not be exposed. It is best to set it to be long and complex

fastdfs.http_tracker_http_port=80

fastdfs.tracker_servers=Server extranet ip:22122

fastdfs.connection_pool.enabled=true

fastdfs.connection_pool.max_count_per_entry=500

fastdfs.connection_pool.max_idle_time=3600

fastdfs.connection_pool.max_wait_time_in_ms=1000

• Add tool class: FastDFSUtil.java

import org.csource.common.MyException;

import org.csource.fastdfs.*;

import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;

import java.io.UnsupportedEncodingException;

import java.security.NoSuchAlgorithmException;

import java.time.Instant;

public class FastDFSUtil {

    private static StorageClient1 client1;

    static {

        try {

            ClientGlobal.initByProperties("fastdfs-client.properties");

            TrackerClient trackerClient = new TrackerClient();

            TrackerServer trackerServer = trackerClient.getConnection();

            client1 = new StorageClient1(trackerServer, null);

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    /**

     * Upload file
     */
    public static String uploadFile(MultipartFile file) throws IOException, MyException {
        String fileName = file.getOriginalFilename();
        //Returns the path uploaded to the server
        return client1.upload_file1(file.getBytes(), fileName.substring(fileName.lastIndexOf(".") + 1), null);
    }

    /**

     * Download File
     */
    public static byte[] downloadFile(String fileId) throws IOException, MyException {
        return client1.download_file1(fileId);
    }
    //test
    public static String uploadFile(String localFilePath) throws IOException, MyException {
        return client1.upload_file1(localFilePath, localFilePath.substring(localFilePath.lastIndexOf(".") + 1), null);
    }
    /**
     * Get the token to access the file and handle the global exception (when the anti-theft chain is turned on)
     */
    public static String getToken(String fileId) throws UnsupportedEncodingException, NoSuchAlgorithmException, MyException {
        int ts = (int) Instant.now().getEpochSecond();
        String subStr = fileId.substring(7); // Note that this address does not contain group. Make no mistake
        String token = ProtoCommon.getToken(subStr, ts, "FastDFS1234567890"); // FastDFS1234567890 is the previously configured parameter fastdfs.http_ secret_ Value of key
        StringBuilder sb = new StringBuilder();
        String IP = "http(s)://xx.xx.xx.x / "; / / server extranet ip. s in parentheses indicates whether http or https access is determined according to the previous configuration
        sb.append(IP);
        sb.append(fileId);
        sb.append("?token=").append(token);
        sb.append("&ts=").append(ts);
        return sb.toString();
    }
}

• The Controller layer http interface is written as follows:

//Upload file

@PostMapping("/uploadFile")

@ResponseBody

public R uploadFile(MultipartFile file) throws IOException, MyException {

    //Set the message type according to the extension: emoji/text/img/file/sys/whiteboard/video/audio

    String filePartName = FastDFSUtil.uploadFile(file);

    String filePath = nginxHost + filePartName; // nginxHost is the server domain name. Note that it is consistent with the previous setting of http or https access

    System.out.println("The file name on the server is:" + filePartName);

    return R.ok().data("filePath", filePath);

}

//Provide file download

@GetMapping("/downloadFile")

public void downloadFile(@RequestParam("fileId") String fileId,

                         @RequestParam("fileName") String fileName,

                         HttpServletResponse resp) {

    try {

        byte[] bytes = FastDFSUtil.downloadFile(fileId);

        resp.setCharacterEncoding("UTF-8");

        resp.setHeader("Content-disposition", "attachment;filename=" + URLEncoder.encode(fileName, "UTF-8"));

        ServletOutputStream outputStream = resp.getOutputStream();

        IOUtils.write(bytes, outputStream);

    } catch (Exception e) {

        e.printStackTrace();

    }

}

• Writing test of obtaining its access token through file fileId:

@Test

void testGetFileToken() throws UnsupportedEncodingException, NoSuchAlgorithmException, MyException {

   String fileId = "group1/M00/00/00/wKgxxxAgGBxxxxxsMM090.jpg";

   String fileUrl = FastDFSUtil.getToken(fileId);

   System.out.println(fileUrl);

}