FastDFS Download
Link: https://pan.baidu.com/s/1SbUdiVtkc1rmf6k4AVXn8g
Extraction code: 2959
1, Distributed file system overview?
1. Classification
1.1 general distributed file system
It corresponds to traditional local file systems (such as ext3, NTFS, etc.). Typical representatives: lustre, MooseFS
1.1.1 advantages
The standard file system operation mode has a low threshold for developers
1.1.2 disadvantages
The system is highly complex and needs to support several standard file operations, such as directory structure, file read-write permission, file lock, etc. Higher complexity
The overall performance of the system is reduced because it supports POSIX standard (representing Portable Operating System Interface of UNIX, which defines the interface standard that the operating system should provide for applications)
1.2 dedicated distributed file system
Based on the idea of google File System, the file cannot be modified after uploading. You need to use proprietary API s to access files, which can also be called distributed file storage services. Typical representatives: MogileFS, FastDFS, TFS.
1.2.1 advantages
The system has low complexity and does not need to support several standard file operations, such as directory structure, file read-write permission, file lock, etc. the system is relatively simple.
The overall performance of the system is high, because it does not need to support POSIX standard, the link of supporting POSIX introduction can be omitted, and the system is more efficient.
1.2.2 disadvantages
Proprietary API is adopted, which has a high threshold for developers (directly encapsulated into tool classes)
2 Google FS architecture
2.1 two roles
Nameserver (index server)
Storage server
2.2 architecture features
File modification is not supported.
The file is stored in blocks and requires an index server
Multiple copies of a file can be stored, and the storage servers to which a file is stored are usually dynamically allocated.
2, What is FastDFS?
FastDFS is a lightweight open source distributed file system. It started in April 2008. A lightweight distributed file system similar to google FS, implemented in pure C, supports UNIX systems such as Linux, FreeBSD and AIX.
It mainly solves the problems of large-capacity file storage and high concurrent access, and realizes load balancing during file access. A software based RAID (Redundant Arrays of Independent Drives) can be stored using a cheap IDE (Integrated Drive Electronics) hard disk. It also supports online capacity expansion of storage servers. Only one copy of the file supporting the same content is saved to save disk space.
FastDFS can only be accessed through the Client API and does not support POSIX access.
FastDFS is especially suitable for large and medium-sized websites to store resource files (such as pictures, documents, audio, video, etc.)
website:
FastDFS has no official website. However, the author Yu Qing (happy_fish100) is the moderator of FastDFS in chinaunix. And the content in the section will be updated from time to time.
http://bbs.chinaunix.net/
FastDFS software can be downloaded in sourceforge. The latest version is 5.08
3, FastDFS architecture
1. Architecture diagram
2. Role
Client: client. Projects written in the java language belong to the client.
Tracker Server: a tracking server, which is mainly used for scheduling and load balancing in access. The status information of group and storage server in the cluster is recorded in memory. It is the hub connecting the Client and storage server.
Storage Server: Storage Server. Files and file attributes (meta data) are saved to the Storage Server
3. Architecture interpretation
There are only two roles, tracker server and storage server, which do not need to store file index information.
All servers are peer-to-peer, and there is no master slave relationship.
The storage servers are grouped, and the files on the storage servers in the same group are exactly the same (RAID 1).
Storage servers in different groups do not communicate with each other.
The storage server actively reports status information to the tracker server, and the tracker servers will not communicate with each other.
4, FastDFS installation
1. Install FastDFS dependency
FastDFS is an application developed in C language. The installation must use the make, cmake, and gcc compilers.
#yum install -y make cmake gcc gcc-c++
2 upload and unzip libfastcommon master
Upload libfastcommon master to / usr/local/tmp. Libfastcommon is a common C function library extracted from FastDFS and FastDHT
Unzip libfastcommon master Since zip is a zip file, use the unzip command
#cd /usr/local/tmp
#unzip libfastcommon-master.zip
3 compile and install
libfastcommon does not provide the make command installation file. Shell scripts are used to perform compilation and installation. The shell script is make sh
Enter the extracted file
cd libfastcommon-master
Compile
./make.sh
Installation
./make.sh install
There is a fixed default installation location. In the / usr/lib64 and / usr/include/fastcommon directories
4 create a soft connection
Because the Lib directory set by the FastDFS main program is / usr/local/lib, you need to create a soft connection
ln -s /user/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
ln -s /usr/local/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
5 upload and unzip the FastDFS main program
Upload FastDFS_v5.08.tar.gz to / usr/local/tmp and decompress
cd /usr/local/tmp
tar zxf FastDFS_v5.08.tar.gz
6 compile and install FastDFS
Enter the extracted FastDFS file
cd FastDFS
Compile
./make.sh
Installation
./make.sh install
After installation, the location of FastDFS main program is
/ usr/bin location of the executable
Location of the / etc/fdfs configuration file
/ usr/bin location of main program code
/ usr/include/fastdfs contains the location of some plug-in groups
7 configure tracker
7.1 copying configuration files
Enter / etc/fdfs and copy the tracker configuration file
cd /etc/fdfs
cp tracker.conf.sample tracker.conf
7.2 creating a data directory
Create a directory to place tracker data
mkdir -p /usr/local/fastdfs/tracker
7.3 modifying configuration files
Modify tracker Conf set the tracker content storage directory
base_path=/usr/local/fastdfs/tracker
#vim tracker.conf
The default port 22122 does not need to be modified
7.4 start up service
service fdfs_trackerd start
After successful startup, the base in the configuration file_ The FastDFS service related data directory (data directory, logs directory) appears in the directory pointed to by path
7.5 viewing service operation status
service fdfs_trackerd status
If is running is displayed, it indicates normal operation.
7.6 turn off the firewall
service iptables stop
chkconfig iptables off
8. Configure storage
Storage can not be on the same server as the tracker. In the example, storage and tracker are installed on the same server.
8.1 copying configuration files
Go to / etc/fdfs and copy the storage configuration file
cd /etc/fdfs
cp storage.conf.sample storage.conf
8.2 create directory
Create two directories. base is used to store basic data and logs, and store is used to store uploaded data.
mkdir -p /usr/local/fastdfs/storage/base
mkdir -p /usr/local/fastdfs/storage/store
8.3 modifying configuration files
storage. The conf configuration file is used to describe the behavior of the storage service. The following modifications are required
vim /etc/fdfs/storage.conf
The configuration contents are as follows:
base_path=/usr/local/fastdfs/storage/base
store_path0=/usr/local/fastdfs/storage/store
tracker_server=tracker service IP: 22122
base_path - base path. The directory used to save the basic data content and log content of the storage server.
store_path0 - storage path. Is the directory used to save the files stored in FastDFS, which is the location where you want to create 256 * 256 subdirectories.
base_path and store_path0 can use the same directory.
tracker_server - tracks the location of the server. Is to track the IP and port of the server.
Start service
service fdfs_storaged start
After successful startup, the base in the configuration file_ The store in the configuration file of FastDFS service related data directory (data directory, logs directory) appears in the directory pointed to by path_ FastDFS storage related data records (data directory) also appear in the directory pointed to by path0. Where $store_ By default, several subdirectories are created in path0 / data / directory (256 * 256 directories in total at two levels of directory level), which are used to store specific file data.
The Storage server starts slowly because 256 * 256 directories need to be created when it is started for the first time.
View startup status
service fdfs_storaged status
5, File upload process
1 sequence diagram
2. Process description
1. Client access Tracker
2. Tracker returns the ip and port of the Storage
3. The client directly accesses the Storage and sends the file content and metadata.
4. Storage returns the file storage id. Contains the group name and file name
6, Fastdfs java client
1 add dependency
<dependencies>
<dependency>
<groupId>cn.bestwu</groupId>
<artifactId>fastdfs-client-java</artifactId>
<version>1.27</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.4</version>
</dependency>
</dependencies> `
2 write configuration file
File name: fdfs_client.conf
Change to your own tracker server ip
connect_timeout = 10 network_timeout = 30 charset = UTF-8 http.tracker_http_port = 8080 tracker_server = 192.168.93.10:22122
3 import tool class
On COM utils. Paste configuration tool class under fastdfsclient
package com.jowell.utils;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.lang3.StringUtils;
import org.csource.common.NameValuePair;
import org.csource.fastdfs.ClientGlobal;
import org.csource.fastdfs.StorageClient;
import org.csource.fastdfs.StorageClient1;
import org.csource.fastdfs.StorageServer;
import org.csource.fastdfs.TrackerClient;
import org.csource.fastdfs.TrackerServer;
/**
* FastDFS Distributed file system operation client
*/
public class FastDFSClient {
private static final String CONF_FILENAME = Thread.currentThread().getContextClassLoader().getResource("").getPath() + "fdfs_client.conf";
private static StorageClient storageClient = null;
/**
* Load only once
*/
static {
try {
ClientGlobal.init(CONF_FILENAME);
TrackerClient trackerClient = new TrackerClient(ClientGlobal.g_tracker_group);
TrackerServer trackerServer = trackerClient.getConnection();
StorageServer storageServer = trackerClient.getStoreStorage(trackerServer);
storageClient = new StorageClient(trackerServer, storageServer);
} catch (Exception e) {
e.printStackTrace();
}
}
/**
*
* @param inputStream
* Uploaded file input stream
* @param fileName
* Original name of uploaded file
* @return
*/
public static String[] uploadFile(InputStream inputStream, String fileName) {
try {
// Metadata for files
NameValuePair[] meta_list = new NameValuePair[2];
// The first set of metadata, the original name of the file
meta_list[0] = new NameValuePair("file name", fileName);
// Second set of metadata
meta_list[1] = new NameValuePair("file length", inputStream.available()+"");
// Prepare byte array
byte[] file_buff = null;
if (inputStream != null) {
// View the length of the file
int len = inputStream.available();
// Create a byte array of corresponding length
file_buff = new byte[len];
// Read the byte contents in the input stream into the byte array.
inputStream.read(file_buff);
}
// Upload files. Parameter meaning: the content of the file to be uploaded (passed by byte array), the type (extension) of the uploaded file, and metadata
String[] fileids = storageClient.upload_file(file_buff, getFileExt(fileName), meta_list);
return fileids;
} catch (Exception ex) {
ex.printStackTrace();
return null;
}
}
/**
*
* @param file
* file
* @param fileName
* file name
* @return If Null is returned, it is a failure
*/
public static String[] uploadFile(File file, String fileName) {
FileInputStream fis = null;
try {
NameValuePair[] meta_list = null; // new NameValuePair[0];
fis = new FileInputStream(file);
byte[] file_buff = null;
if (fis != null) {
int len = fis.available();
file_buff = new byte[len];
fis.read(file_buff);
}
String[] fileids = storageClient.upload_file(file_buff, getFileExt(fileName), meta_list);
return fileids;
} catch (Exception ex) {
return null;
}finally{
if (fis != null){
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/**
* Delete a file based on the group name and remote file name
*
* @param groupName
* For example, if "group1" does not specify this value, it defaults to group1
* @param remoteFileName
* For example, "M00/00/00/wKgxgk5HbLvfP86RAAAAChd9X1Y736.jpg"
* @return 0 Is success, non-0 is failure, and the specific error code is
*/
public static int deleteFile(String groupName, String remoteFileName) {
try {
int result = storageClient.delete_file(groupName == null ? "group1" : groupName, remoteFileName);
return result;
} catch (Exception ex) {
return 0;
}
}
/**
* Modify an existing file
*
* @param oldGroupName
* Old group name
* @param oldFileName
* Old file name
* @param file
* new file
* @param fileName
* New file name
* @return If it returns null, it means failure
*/
public static String[] modifyFile(String oldGroupName, String oldFileName, File file, String fileName) {
String[] fileids = null;
try {
// Upload first
fileids = uploadFile(file, fileName);
if (fileids == null) {
return null;
}
// Delete again
int delResult = deleteFile(oldGroupName, oldFileName);
if (delResult != 0) {
return null;
}
} catch (Exception ex) {
return null;
}
return fileids;
}
/**
* File download
*
* @param groupName Volume name
* @param remoteFileName file name
* @return Returns a stream
*/
public static InputStream downloadFile(String groupName, String remoteFileName) {
try {
byte[] bytes = storageClient.download_file(groupName, remoteFileName);
InputStream inputStream = new ByteArrayInputStream(bytes);
return inputStream;
} catch (Exception ex) {
return null;
}
}
public static NameValuePair[] getMetaDate(String groupName, String remoteFileName){
try{
NameValuePair[] nvp = storageClient.get_metadata(groupName, remoteFileName);
return nvp;
}catch(Exception ex){
ex.printStackTrace();
return null;
}
}
/**
* Gets the file suffix (without dots)
*
* @return For example: "jpg" or ""
*/
private static String getFileExt(String fileName) {
if (StringUtils.isBlank(fileName) || !fileName.contains(".")) {
return "";
} else {
return fileName.substring(fileName.lastIndexOf(".") + 1); // Without last point
}
}
}
4 write test code
Feel free to create a new class containing the main method. com.jowell.MyMain
public class MyMain {
public static void main(String[] args) {
try {
File file = new File("D:/b.png");
InputStream is = new FileInputStream(file);
String fileName = UUID.randomUUID().toString()+".png";
String[] result = FastDFSClient.uploadFile(is, fileName);
System.out.println(Arrays.toString(result));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
7, File download
1. Download instructions
1. The client asks the tracker to download the storage of the file. The parameter is the file ID (group name and file name);
2. The tracker returns an available storage;
3. client communicates directly with storage to complete file download.
2. Code implementation
Directly use the tool method to complete the download.
try {
InputStream is = FastDFSClient.downloadFile("group1", "M00/00/00/wKg0gF3zAKCARs6kAAASjQVYlWA098.png");
OutputStream os = new FileOutputStream(new File("D:/jqk.png"));
int index = 0 ;
while((index = is.read())!=-1){
os.write(index);
}
os.flush();
os.close();
is.close();
} catch (IOException e) {
e.printStackTrace();
}
