Overall architecture diagram
1. Distributed memory fast DFS
2. redis cache database
3. MySql database
4.HTTP protocol
5. Server nginx
6. Dynamic request processing fastcgi spawn fcgi
7. Client Qt
According to the function of the client, it is summarized in modules
Directory composition of client
- common public interface, which is some public interfaces used by various modules
- conf: directory of configuration files
- Images: images used in the software
- myselfWidget self drawing control
Server setup function
- The configuration information of the server is written into the configuration file
- The format of the configuration file is JSON, web_ IP and PORT are stored in the server
{ "login": { "pwd": "wqq2b4Ild/w=", "remember": "yes", "user": "Mi/CvL0kLkQ=" }, "type_path": { "path": "/Users/sunguosheng/Code/Yun/Pan/conf/fileType" }, "web_server": { "ip": "10.211.55.11", "port": "80" } }
- When the program runs for the first time, this configuration file is actively generated, and the default information will be written in it
int Login::createWebConf(QString path);
- When the user enters information, the web_ The content of the json object of the server is overwritten, and the other items remain the default
- When organizing json information, QMAP is used
QMap<QString,Qvariant> info; QJsonDocument::fromVariant(info);
Client and server processes at registration
- The information registered by users is verified by regular expressions
// regular expression #define USER_REG "^[a-zA-Z\\d_@#-\\*]{3,16}$" #define PASSWD_REG "^[a-zA-Z\\d_@#-\\*]{6,18}$" #define PHONE_REG "^1(3[0-9]|4[01456879]|5[0-35-9]|6[2567]|7[0-8]|8[0-9]|9[0-35-9])\\d{8}$" #define EMAIL_REG "^\\w+([-+.]\\w+)*@\\w+([-.]\\w+)*\\.\\w+([-.]\\w+)*$" #define IP_REG "((2[0-4]\\d|25[0-5]|[01]?\\d\\d?)\\.){3}(2[0-4]\\d|25[0-5]|[01]?\\d\\d?)" #define PORT_REG "^[1-9]$|(^[1-9][0-9]$)|(^[1-9][0-9][0-9]$)|(^[1-9][0-9][0-9][0-9]$)|(^[1-6][0-5][0-5][0-3][0-5]$)"
//Verification of data QRegExp ex(USER_REG); if(!ex.exactMatch(username)) { QMessageBox::critical(this,"error",QString("The user name you entered does not meet the specification!")); ui->reg_username->clear(); ui->reg_username->setFocus(); return; } ......
- Get this information and send the client organization information to the server
Custom protocols in HTTP
//====================Registered user 127.0.0.1:80/reg post data(json) { userName:xxxx, nickName:xxx, firstPwd:xxx, phone:xxx, email:xxx }
//Set the http request header QNetworkAccessManager * manager = Login::getNetManager();//Ensure that the entire class has only one object QNetworkRequest request; request.setHeader(QNetworkRequest::ContentTypeHeader,"application/json"); request.setHeader(QNetworkRequest::ContentLengthHeader,postData.size()); QString url = QString("http://%1:%2/reg").arg(ip).arg(port); / / the precondition for using ip port directly is the initialization of the configuration file request.setUrl(QUrl(url)); //Send http data QNetworkReply *reply = manager->post(request,postData);
- Server side operation
//json package for parsing user registration information int get_reg_info(char *reg_buf, char *user, char *nick_name, char *pwd, char *tel, char *email); //All outgoing parameters, and the main calling function allocates memory //--------------------------- //If the user is registered, 0 is returned for success, and - 1 is returned for failure. If the user already exists, - 2 is returned int user_register( char *reg_buf ); //The connection information of the database is read from the server-side configuration file, which is also in json format //--------------------------- //sql statement, insert registration information sprintf(sql_cmd, "insert into user (name, nickname, password, phone, createtime, email) values ('%s', '%s', '%s', '%s', '%s', '%s')", user, nick_name, pwd, tel, time_str ,email);
Database design
-- =============================================== User information table -- id: User serial number, auto increment, primary key -- name: User name -- nickname: User nickname -- phone: phone number -- email: mailbox -- createtime: time create table user ( id bigint not null primary key AUTO_INCREMENT, name VARCHAR(128) not null, nickname VARCHAR(128) not null, password VARCHAR(128) not null, phone VARCHAR(15) not null, createtime VARCHAR(128), email VARCHAR(100), constraint uq_nickname unique(nickname), constraint uq_name unique(name) );
Use of MD5 (the password registered by the user needs to be converted into MD5 and stored in the database)
- MD5 is used for encryption. Because it is irreversible, it is used for data verification
- The user's password is encrypted and stored with MD5
- The MD5 conversion of this password is not included in the code (it is convenient to test, but it is not difficult)
- code returned by the server
Status code
Registration: success:{"code":"002"} The user already exists:{"code":"003"} Failed:{"code":"004"}
Success: there is no user with the same name in the database Failed: database connection failed, execution failed sql Statement failed ---------------- Specific errors can be viewed in the server log system LOG(REG_LOG_MODULE, REG_LOG_PROC, "%s Insert failed:%s\n", sql_cmd, mysql_error(conn));
Client and server actions at login
Login information acquisition
- When the user is registered successfully, it will automatically jump to the login interface, and the login user name and password will be filled in automatically
- Remember the password function. The next time you start the software, you will automatically read the configuration file and fill in the user name and password automatically
The client sends login information and the server verifies it
//====================Login user 127.0.0.1:80/login post data(json) { user:xxxx, pwd:xxx }
QByteArray data = getLoginJson(old_username,old_passwd); //Send an HTTP request. This object is obtained by a static function. There is only one object in the whole project QNetworkAccessManager * manager = Login::getNetManager(); //HTTP protocol header QNetworkRequest request; request.setHeader(QNetworkRequest::ContentTypeHeader,"application/json"); request.setHeader(QNetworkRequest::ContentLengthHeader,data.size()); QString url = QString("http://%1:%2/login").arg(ip).arg(port); / / the precondition for using ip port directly is the initialization of the configuration file request.setUrl(QUrl(url)); //Send http data QNetworkReply *reply = manager->post(request,data);
Server operation
//json package login for parsing user login information_ buf //The user name is saved in user and the password is saved in pwd int get_login_info(char *login_buf, char *user, char *pwd); //====================== /* * @brief Judge user login * @param user user name * @param pwd password */ int check_user_pwd( char *user, char *pwd ); //sql statement to find the corresponding password of a user sprintf(sql_cmd, "select password from user where name=\"%s\"", user); //====================== //Generate a random Token of the user for subsequent user authentication to improve user security //The generated tokens will be stored in redis, because subsequent token verification is very frequent, so it is not necessary to check the database every time, which will affect the efficiency int set_token(char *user, char *token); //Generation rules of token //1. Sir, it can be divided into four numbers within 1000 //2. Encrypt the string composed of these four numbers //3. After encryption, convert to BASE64 //4. Convert to 32-bit MD5 string again // redis saves this string. The user name is token. The valid time is 24 hours, //If the user does not drop the line, he will ask to log in again after 24 hours //Each time you log in again, the corresponding token will be regenerated rop_setex_string(redis_conn, user, 86400, token);
token stored in redis
Return to client information
land: success: { "code": "000", "token": "xxx" } Failed: { "code": "001", "token": "xxx" }
- After success, relevant information of the user will be recorded
//Save login information logininfoinstance * instance = logininfoinstance::getInstance(); instance->setLoginInfo(old_username,ip,port,token);
Action corresponding to my file
According to these two menus, you can summarize the functions of "my file" (click on the file and click on the blank)
- File upload
- File download
- Refresh (ascending by downloads, descending by downloads)
- share
- delete
- attribute
File upload
Timing of file upload trigger
- Click upload file item
- Right click the blank space
Preliminary preparation for file upload
- You need to create a task queue to upload files
- The task queue for uploading files is a class in singleton mode. A program has only one upload queue
//Each task is a pointer to the information structure of the uploaded file //Each uploaded file information is described with this structure struct UploadFileInfo { QString md5; //File md5 code QFile *file; //field name pointer QString fileName; //File name qint64 size; //file size QString path; //File path bool isUpload; //Is it already uploading DataProgress *dp; //Upload progress control, which is a custom control used to record the progress of each file transfer };
//In singleton mode, a program can only have one upload list class UploadTask { public: static UploadTask* getInstance(); //Get unique instance //Append uploaded files to the upload list //Parameter: path upload file path //Return value: success is 0 //Failed: // -1: The file length is greater than 30m // -2: Is the uploaded file already in the upload queue // -3: Failed to open file // -4: Failed to get layout int appendUploadList(QString path); //Judge whether the upload queue is empty bool isEmpty(); //Determine whether a task is being uploaded bool isUpload(); //Take out the 0th upload task. If there is no task uploading in the task queue, set the 0th upload task UploadFileInfo * takeTask(); //Delete uploaded tasks void dealUploadTask(); //Clear upload list void clearList(); //Get the tasks in the task list QList<UploadFileInfo*> getUploadTaskList(); private: UploadTask(); ~UploadTask(); UploadTask(const UploadTask&); UploadTask& operator=(const UploadTask&); //Static data member, which needs to be initialized outside the class static UploadTask *instance; class Garbo{ public: ~Garbo() { if(NULL != UploadTask::instance) { UploadTask::instance->clearList(); delete UploadTask::instance; UploadTask::instance = NULL; } } }; static Garbo garbo; QList<UploadFileInfo*> list; }; #endif // UPLOADTASK_H
- In the process of joining the queue, it involves the dynamic addition of the progress bar
//Create a progress bar DataProgress *p = new DataProgress; p->setFileName(info.fileName()); //Set a title for the progress bar tmp->dp = p; //Initialization of progress bar //------------------Get a layout and put the progress bar in it----------- //Get a vertical layout UploadLayout *pUpload = UploadLayout::getInstance(); if(pUpload == NULL) { cout<<__FUNCTION__<<"getInstance error"; return -4; } //Cout < < "come here!"; //Convert to a vertical layout QVBoxLayout *layout = (QVBoxLayout*)pUpload->getUploadLayout(); // Add to the layout, and the last one is the spring, which is inserted on the upper edge of the spring //cout<<__FUNCTION__<<layout->count(); layout->insertWidget(layout->count()-1, p);
- The layout for placing the progress bar is also a class of singleton mode (that is, the code in line 8 above)
#ifndef UPLOADLAYOUT_H #define UPLOADLAYOUT_H #include "common/common.h" #include <QVBoxLayout> //Upload progress layout class, single instance mode class UploadLayout { public: static UploadLayout *getInstance(); //Guarantee a unique instance void setUploadLayout(QWidget *p); //Set layout QLayout *getUploadLayout(); //Get layout private: UploadLayout() { } ~UploadLayout() //Destructor is private { } //Static data member, declared in class, must be defined outside class static UploadLayout *instance; QLayout *m_layout; QWidget *m_wg; //Its only job is to delete the Singleton instance in the destructor class Garbo { public: ~Garbo() { if(NULL != UploadLayout::instance) { delete UploadLayout::instance; UploadLayout::instance = NULL; cout << "instance is detele"; } } }; //Define a static member variable. When the program ends, the system will automatically call its destructor //The destructor of class static is called after main() exits. static Garbo temp; //Static data member, declared in class, defined outside class }; #endif // UPLOADLAYOUT_H
- The timer regularly checks the task queue and processes the task queue
//Regularly check the upload task every 500ms connect(&m_uploadFileTimer,&QTimer::timeout,this,[=](){ //Upload file processing, take out the first task in the upload task list, and then take the next task after uploading uploadFilesAction(); }); //Check the uploaded task list every 500ms //Only one task can be uploaded at a time m_uploadFileTimer.start(500);
Specific processing of second transmission of MD5 files
- First, verify the second file transfer (that is, the file already exists in the server, and the second file transfer is the file count + 1)
url: http://127.0.0.1:80/md5 post data: { user:xxxx, token:xxxx, md5:xxx, fileName: xxx }
- Server side processing action
//Parsing json package of second transmission information int get_md5_info(char *buf, char *user, char *token, char *md5, char *filename); //Verify the login token, return 0 if successful, and - 1 if failed int verify_token(user, token); //util_cgi.h //Check whether the database has MD5 for this file //sql statement to obtain the file counter count of this md5 value file sprintf(sql_cmd, "select count from file_info where md5 = '%s'", md5); //If there is no such file, return {"code":"007"}; //If you have this file //Query whether this file is the file to which the user belongs //Check whether this user already has this file. If it exists, it indicates that this file has been uploaded and does not need to be uploaded again sprintf(sql_cmd, "select * from user_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //Return to {"code":"005"} //If you have this file, but it is not owned by this user, you can transfer the file in seconds //1. Modify file_ Count field in info, + 1 (count file reference count) sprintf(sql_cmd, "update file_info set count = %d where md5 = '%s'", ++count, md5);//Front++ // update file_info set count = 2 where md5 = "bae488ee63cef72efb6a3f1f311b3743"; //2,user_file_list insert a piece of data sprintf(sql_cmd, "insert into user_file_list(user, md5, createtime, filename, shared_status, pv) values ('%s', '%s', '%s', '%s', %d, %d)", user, md5, time_str, filename, 0, 0); //3. Query the number of files to which the user belongs and update the database //If the user's record has not been modified, insert it directly sprintf(sql_cmd, " insert into user_file_count (user, count) values('%s', %d)", user, 1); //If there is already a user's record, update the record directly sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count+1, user); //Return to {"code":"006"}
- Several database tables involved in the above server operations
-- =============================================== Document information table,Store all file information -- md5 file md5 -- file_id file id -- url file url -- size file size, In bytes -- type File type: png, zip, mp4...... -- count File reference count, which is 1 by default. This counter will be for each additional user owning this file+1 create table file_info ( md5 varchar(200) not null primary key, file_id varchar(256) not null, url varchar(512) not null, size bigint, type VARCHAR(20), count int ); -- =============================================== User file list -- user User to which the file belongs -- md5 file md5 -- createtime File creation time -- filename File name -- shared_status Sharing status, 0 Is not shared, 1 is shared -- pv Number of files downloaded. The default value is 0. Add 1 to each download create table user_file_list ( user varchar(128) not null, md5 varchar(200) not null, createtime VARCHAR(128), filename varchar(128), shared_status int, pv int ); -- =============================================== Quantity table of user files -- user User to which the file belongs -- count Number of files owned create table user_file_count ( user varchar(128) not null primary key, count int );
- code returned by the server
token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} File transfer in seconds: File already exists:{"code":"005"} Successful transmission in seconds: {"code":"006"} Second transmission failure: {"code":"007"}
Real upload file
- Information sent by the client
url:http://127.0.0.1:80/upload post data: ------WebKitFormBoundary88asdgewtgewx\r\n Content-Disposition: form-data; user="mike"; filename="xxx.jpg"; md5="xxxx"; size=10240\r\n Content-Type: application/octet-stream\r\n \r\n Real file content\r\n ------WebKitFormBoundary88asdgewtgewx
- The main step on the client side is to construct the sent data
QByteArray data; //Split line data.append(boundary);// QString boundary= m_comm.getBoundary(); data.append("\r\n"); //File information data.append("Content-Disposition: form-data; "); data.append( QString("user=\"%1\" ").arg( login->getUserName() ) ); //Upload user data.append( QString("filename=\"%1\" ").arg(fileName) ); //File name data.append( QString("md5=\"%1\" ").arg(md5) ); //File md5 code data.append( QString("size=%1").arg(size) ); //file size data.append("\r\n"); //Format of data data.append(QString("Content-Type: application/octet-stream")); data.append("\r\n\r\n"); //Real data to send data.append(file->readAll()); data.append("\r\n"); //Add split line data.append(boundary);
- Server side operation
//Analyze the uploaded file to get the file uploader, file name, file MD5 and file size; At the same time, save the file in the temporary directory int recv_save_file(long len, char *user, char *filename, char *md5, long *p_size); //Upload the file to fast DFS to get the ID of the inverted file int upload_to_dstorage(char *filename, char *fileid); //Get the URL and download address of the file int make_file_url(char *fileid, char *fdfs_file_url); //Upload the information about the fast DFS of the uploaded file to mysql store_fileinfo_to_mysql(user, filename, md5, size, fileid, fdfs_file_url); sprintf(sql_cmd, "insert into file_info (md5, file_id, url, size, type, count) values ('%s', '%s', '%s', '%ld', '%s', %d)",md5, fileid, fdfs_file_url, size, suffix, 1); sprintf(sql_cmd, "insert into user_file_list(user, md5, createtime, filename, shared_status, pv) values ('%s', '%s', '%s', '%s', %d, %d)", user, md5, create_time, filename, 0, 0); //Judge whether the user has uploaded records //Query the number of user files sprintf(sql_cmd, "select count from user_file_count where user = '%s'", user); //Record, update value sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count+1, user); //No record, insert new record sprintf(sql_cmd, " insert into user_file_count (user, count) values('%s', %d)", user, 1);
- Server return code
token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} Upload file: success:{"code":"008"} Failed:{"code":"009"}
File download
Preparation before file download
- Queue of tasks to be downloaded
- Download the layout of each progress bar of the task, and download the layout class of the task
- The implementation of the above two parts is the same as uploading
Specific operation of downloading
- Client operation
//Add the file to be downloaded to the download queue int appendDownloadList( FileInfo *info, QString filePathName, bool isShare = false); //Download File QNetworkReply *reply = m_manager->get(QNetworkRequest(url)); //The URL here is the URL uploaded to the database
- Download server side operations
stay fast-DFS of storage On the host of the storage node, the nginx So you can hold it directly URL To request this resource, the server only needs to configure the storage node mginx That's good
- Data after packet capture
- location option set by nguni
After downloading, update the PV field of the corresponding file
- Packet capture data
- Client processing
//You need to handle the download volume of the corresponding file http://%1:%2/dealfile?cmd=pv //Package sent json data /* { "user": "yoyo", "token": "xxxx", "md5": "xxx", "filename": "xxx" }
- Server side operation
//Parsing json data get_json_info(buf, user, token, md5, filename); //Parsing json information //Perform token authentication verify_token(user, token); //util_cgi.h //Authentication changes the PV field value of the file to which the user belongs //View the pv field of the file sprintf(sql_cmd, "select pv from user_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //Update the PV field value, that is, the + 1 operation sprintf(sql_cmd, "update user_file_list set pv = %d where user = '%s' and md5 = '%s' and filename = '%s'", pv+1, user, md5, filename); //Feedback client code
- code returned by the server
token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} Download File pv Field processing success:{"code":"016"} Failed:{"code":"017"}
- The client adjusts the attributes of the file in "my file" according to the returned code
//Download File pv (download volume) field processing void MyFileWg::dealFilePv(QString md5, QString filename); if(code == "016") { //This file pv field + 1 for(int i = 0; i < m_fileList.size(); ++i) { FileInfo *info = m_fileList.at(i); if( info->md5 == md5 && info->filename == filename) { int pv = info->pv; info->pv = pv+1; cout<<filename<<pv; break; //Very important interrupt conditions } } }
Refresh (get user list)
- There is a default sorting method
- In ascending order of Downloads
- In descending order of Downloads
//Enumeration values for three states enum Display{Normal, PvAsc, PvDesc};//The mode used to represent the refresh
Client action 1
- First, the menu style is a self drawn control
//Menu 1: click the menu in the blank space m_menuEmpty = new MyMenu(this); m_pvAscendingAction = new QAction("Ascending by Downloads",this); m_pvDescendingAction = new QAction("In descending order of Downloads",this); m_refreshAction = new QAction("Refresh",this); m_uploadAction = new QAction("upload",this);
- The interfaces for these three modes are the same
void MyFileWg::refreshFiles(MyFileWg::Display cmd);
- Get the number of files for the user first
//Data sent { "user":"xxx", "token":"xxx" } //127.0.0.1:80/myfiles?cmd=count // Get the number of user files
Server action 1
//Get the user name and token through json package void get_count_json_info(buf, user, token); //User token authentication int verify_token(user, token); //util_cgi.h //Get the number of files for the user void get_user_files_count(char *user, int ret); sprintf(sql_cmd, "select count from user_file_count where user=\"%s\"", user); //Feedback to client data
- code returned by the server
token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} Data returned by the server to the front end { "code":"xxx", "num":xx }
Client operation 2
//No files, refresh refreshFileItems(); //When the item is updated, the item of "upload file" will be added //If you have a file, start getting the file //Obtain user file information 127.0 0.1:80/myfiles&cmd=normal //127.0 in ascending order of Downloads 0.1:80/myfiles? cmd=pvasc //127.0 in descending order of Downloads 0.1:80/myfiles? cmd=pvdesc url = QString("http://%1:%2/myfiles?cmd=%3").arg(login->getIp()).arg(login->getPort()).arg(cmdType); post data json The package is as follows: //start is the starting point of the file location and count the number of files. You need to display 0 ~ 9 locations as files { "user": "yoyo" "start": 0 "count": 10 } -- Query page n+1 Line to m+n Row record select * from table1 limit n, m; SELECT * FROM table LIMIT 5,10;Returns the records from lines 6 to 15
Server operation 2
//Obtain user file information 127.0 0.1:80/myfiles&cmd=normal //127.0 in ascending order of Downloads 0.1:80/myfiles? cmd=pvasc //127.0 in descending order of Downloads 0.1:80/myfiles? cmd=pvdesc //Get json package data int get_fileslist_json_info(buf, user, token, &start, &count); //Validate token int verify_token(user, token); //util_cgi.h //Get user list int get_user_filelist(cmd, user, start, count); //Get user file list //Multi table specified row range query if(strcasecmp(cmd, "normal") == 0) //Get user file information { sprintf(sql_cmd, "select user_file_list.*, file_info.url, file_info.size, file_info.type from file_info, user_file_list where user = '%s' and file_info.md5 = user_file_list.md5 limit %d, %d", user, start, count); } else if(strcasecmp(cmd, "pvasc") == 0) //Ascending by Downloads { sprintf(sql_cmd, "select user_file_list.*, file_info.url, file_info.size, file_info.type from file_info, user_file_list where user = '%s' and file_info.md5 = user_file_list.md5 order by pv asc limit %d, %d", user, start, count); } else if(strcasecmp(cmd, "pvdesc") == 0) //In descending order of Downloads { //sql statement sprintf(sql_cmd, "select user_file_list.*, file_info.url, file_info.size, file_info.type from file_info, user_file_list where user = '%s' and file_info.md5 = user_file_list.md5 order by pv desc limit %d, %d", user, start, count); } //Return to client file information
- Information returned by the server
token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} To get a list of user files: Failed:{"code": "015"} Success: file list json Information fed back by the server to the front end,json Inside the object is a large json Array, json Inside the array is one by one json object { "files": [ { "user": "yoyo", "md5": "e8ea6031b779ac26c319ddf949ad9d8d", "time": "2017-02-26 21:35:25", "filename": "test.mp4", "share_status": 0, "pv": 0, "url": "http://192.168.31.109:80/group1/M00/00/00/wKgfbViy2Z2AJ-FTAaM3As-g3Z0782.mp4", "size": 27473666, "type": "mp4" }, { "user": "yoyo", "md5": "e8ea6031b779ac26c319ddf949ad9d8d", "time": "2017-02-26 21:35:25", "filename": "test.mp4", "share_status": 0, "pv": 0, "url": "http://192.168.31.109:80/group1/M00/00/00/wKgfbViy2Z2AJ-FTAaM3As-g3Z0782.mp4", "size": 27473666, "type": "mp4" } ] }
Client operation 3
//Parse each json object and get the corresponding data //Add the data of each file to M_ In the filelist object //Add the corresponding item to the QListWidget of my files
share
- After sharing, the file properties will change and need to be refreshed
- The shared files will be placed in the "sharing list" and can be downloaded by other users
Client operation
//The unified interface accepts commands first void MyFileWg::dealSelectdFile(QString cmd); //Share a file void MyFileWg::shareFile(FileInfo *info)
{ "user": "yoyo", "token": "xxxx", "md5": "xxx", "filename": "xxx" } QString url = QString("http://%1:%2/dealfile?cmd=share").arg(login->getIp()).arg(login->getPort());
Server operation
//a) First judge whether this file has been shared and whether the collection has this file. If so, it means that others have shared this file and interrupt the operation (redis operation) int rop_zset_exit(redis_conn, FILE_PUBLIC_ZSET, fileid);//Judge whether a member exists //-->Essence: reply = rediscommand (Conn, "zlexcount% s [% s], key, member, member); //http://www.redis.cn/commands/zlexcount.html View the meaning of the corresponding zlexcount //Return: someone else has shared this file: {"code", "012"}
b)If the collection does not have this element, it may be because redis There is no record in, and then from mysql Query in, if mysql No, it doesn't(mysql operation) //Check whether this file has been shared by others sprintf(sql_cmd, "select * from share_file_list where md5 = '%s' and filename = '%s'", md5, filename); //If the query has a result, it means someone has shared it //Return: someone else has shared this file: {"code", "012"} //c) //The query has no result, please see d) e) f)
//c) If mysql has a record but redis does not, it means that redis does not save the file. Redis saves the file information and then interrupts the operation (redis operation) int rop_zset_add(redis_conn, FILE_PUBLIC_ZSET, 0, fileid); //--->Essence: reply = rediscommand (Conn, "zadd% s% LD% s", key, score, member);
//d) If this file is not shared, mysql saves a persistent operation (mysql operation) //sql statement to update the shared flag field sprintf(sql_cmd, "update user_file_list set shared_status = 1 where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //Share the information of the file and save it in share_file_list save list sprintf(sql_cmd, "insert into share_file_list (user, md5, createtime, filename, pv) values ('%s', '%s', '%s', '%s', %d)", user, md5, create_time, filename, 0); //xxx_ share_ xxx_ file_ xxx_ list_ xxx_ count_ Update the document quantity table of XXX user //If the user already exists, update it directly sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count+1, "xxx_share_xxx_file_xxx_list_xxx_count_xxx"); //If the user does not exist, insert a new record directly sprintf(sql_cmd, "insert into user_file_count (user, count) values('%s', %d)", "xxx_share_xxx_file_xxx_list_xxx_count_xxx", 1);
- This collection FILE_PUBLIC_ZSET stores files shared by users
- It is put into redis because this data will be used to display the shared list, which belongs to high-frequency access data
- zset's data structure can quickly sort the file download list
e)redis Add an element to the collection(redis operation) int rop_zset_add(redis_conn, FILE_PUBLIC_ZSET, 0, fileid);
This hash stores a mapping relationship (fileid - > filename)
f)redis Corresponding hash Changes are also needed (redis operation) int rop_hash_set(redis_conn, FILE_NAME_HASH, fileid, filename);
delete
- Click item to delete it
Client operation
{ "user": "yoyo", "token": "xxxx", "md5": "xxx", "filename": "xxx" } http://%1:%2/dealfile?cmd=del
Server side operation
//a) First judge whether this file has been shared //b) Judge whether this file exists in the collection. If so, it indicates that others have shared this file (redis operation) int rop_zset_exit(redis_conn, FILE_PUBLIC_ZSET, fileid); flag = 1; //redis has a record indicating that the file has been shared share =1; //If not, check c)
//c) If there is no such element in the collection, it may be because there is no record in redis, and then query from mysql. If there is no mysql, it means there is really no (MySQL operation) //Query the sharing status of the file sprintf(sql_cmd, "select shared_status from user_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //Query results, record share = 1 //The query has no results Failed to return to client:{"code":"014"}
// d) If mysql has records but redis has no records, the shared file processing only needs to process mysql (mysql operation) //Then the above: share = 1 //share==1, delete the corresponding file in the share list sprintf(sql_cmd, "delete from share_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //xxx_ share_ xxx_ file_ xxx_ list_ xxx_ count_ Number of files for XXX users (number of shared files - 1) sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count-1, "xxx_share_xxx_file_xxx_list_xxx_count_xxx");
// e) If redis has records, both mysql and redis need to process and delete the relevant records //flag = 1 indicates that redis also has this record, and the records in redis also need to be deleted //Deletes a specified member from an ordered collection rop_zset_zrem(redis_conn, FILE_PUBLIC_ZSET, fileid); //Remove the corresponding record from the hash rop_hash_del(redis_conn, FILE_NAME_HASH, fileid);
//Number of user files queried //If there is only one record, which happens to be the same file, delete the record sprintf(sql_cmd, "delete from user_file_count where user = '%s'", user); //If not, change the record sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count-1, user);
//Modify the user file list data and delete the corresponding file sprintf(sql_cmd, "delete from user_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename);
//File reference count of file_info minus 1 sprintf(sql_cmd, "select count from file_info where md5 = '%s'", md5); ret2 = process_result_one(conn, sql_cmd, tmp); //Execute sql statement if(ret2 == 0) { count = atoi(tmp); //count field } count--; //Minus one sprintf(sql_cmd, "update file_info set count=%d where md5 = '%s'", count, md5); if(count == 0) //Note: no user references this file. You need to delete this file in storage { //id of query file sprintf(sql_cmd, "select file_id from file_info where md5 = '%s'", md5); ret2 = process_result_one(conn, sql_cmd, tmp); //Execute sql statement if(ret2 != 0) { LOG(DEALFILE_LOG_MODULE, DEALFILE_LOG_PROC, "%s operation failed\n", sql_cmd); ret = -1; goto END; } //Delete the information of the file in the file information table sprintf(sql_cmd, "delete from file_info where md5 = '%s'", md5); if (mysql_query(conn, sql_cmd) != 0) { LOG(DEALFILE_LOG_MODULE, DEALFILE_LOG_PROC, "%s operation failed: %s\n", sql_cmd, mysql_error(conn)); ret = -1; goto END; } //Delete this file from the storage server. The parameter is file id ret2 = remove_file_from_storage(tmp); if(ret2 != 0) { LOG(DEALFILE_LOG_MODULE, DEALFILE_LOG_PROC, "remove_file_from_storage err\n"); ret = -1; goto END; } }
- Server side return code
token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} Delete file: success:{"code":"013"} Failed:{"code":"014"}
- The client gets the corresponding success code
Remove the corresponding item Remove the file from the file list Refresh
attribute
- Property is a self drawn control
- After clicking, the FileInfo *info of the clicked item will be passed in
//Get attribute information void MyFileWg::getFileProperty(FileInfo *info)
Shared list
- The implementation methods of download, attribute and refresh are the same as those of "my file"
- Cancel sharing
- Transfer file
Cancel sharing
Client operation
127.0.0.1:80/dealsharefile?cmd=cancel { "user": "yoyo", "md5": "xxx", "filename": "xxx" }
Server operation
//Parsing json packets int get_json_info(buf, user, md5, filename); //Parsing json information //Modify the file sharing status to not shared sprintf(sql_cmd, "update user_file_list set shared_status = 0 where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //Query the number of shared files //Query the number of shared files xxx_share_xxx_file_xxx_list_xxx_count_xxx sprintf(sql_cmd, "select count from user_file_count where user = '%s'", "xxx_share_xxx_file_xxx_list_xxx_count_xxx"); //If it is a record, delete it sprintf(sql_cmd, "delete from user_file_count where user = '%s'", "xxx_share_xxx_file_xxx_list_xxx_count_xxx"); //If there are multiple records, change the quantity sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count-1, "xxx_share_xxx_file_xxx_list_xxx_count_xxx"); //Delete data from shared list sprintf(sql_cmd, "delete from share_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //redis operation //Deletes a specified member from an ordered collection ret = rop_zset_zrem(redis_conn, FILE_PUBLIC_ZSET, fileid); if(ret != 0) { LOG(DEALSHAREFILE_LOG_MODULE, DEALSHAREFILE_LOG_PROC, "rop_zset_zrem operation failed\n"); goto END; } //Remove the corresponding record from the hash ret = rop_hash_del(redis_conn, FILE_NAME_HASH, fileid); if(ret != 0) { LOG(DEALSHAREFILE_LOG_MODULE, DEALSHAREFILE_LOG_PROC, "rop_hash_del operation failed\n"); goto END; }
- code returned by the server
Cancel sharing: success:{"code":"018"} Failed:{"code":"019"}
- Subsequent operations of the client
Remove the corresponding item
Transfer file
Client operation
127.0.0.1:80/dealsharefile?cmd=save { "user": "yoyo", "md5": "xxx", "filename": "xxx" }
Server operation
//Parsing json packets int get_json_info(buf, user, md5, filename); //Parsing json information //Check whether the user, file name and md5 exist. If so, the file exists sprintf(sql_cmd, "select * from user_file_list where user = '%s' and md5 = '%s' and filename = '%s'", user, md5, filename); //If it exists, directly return that the file already exists: {"code":"021"} //If not, ---> //Reference count of corresponding file + 1 sprintf(sql_cmd, "update file_info set count = %d where md5 = '%s'", count+1, md5); //Insert a new record into the user file table sprintf(sql_cmd, "insert into user_file_list(user, md5, createtime, filename, shared_status, pv) values ('%s', '%s', '%s', '%s', %d, %d)", user, md5, time_str, filename, 0, 0); //Change the number of files for this user sprintf(sql_cmd, "select count from user_file_count where user = '%s'", user); //If there is no record, insert a new one sprintf(sql_cmd, " insert into user_file_count (user, count) values('%s', %d)", user, 1); //Record update sprintf(sql_cmd, "update user_file_count set count = %d where user = '%s'", count+1, user);
code returned by the server
Transfer file: success:{"code":"020"} File already exists:{"code":"021"} Failed:{"code":"022"}
Download ranking
client
http://%1:%2/sharefiles?cmd=count QNetworkReply * reply = m_manager->get( QNetworkRequest( QUrl(url)) );
Server side
//Parse command query_parse_key_value(query, "cmd", cmd, NULL); //get_share_files_count(); // Get the number of shared files //Query the number of shared files sprintf(sql_cmd, "select count from user_file_count where user=\"%s\"", "xxx_share_xxx_file_xxx_list_xxx_count_xxx"); //Number of files returned to the client
Client 2
http:// %1:%2/sharefiles?cmd=pvdesc { "start": 0, "count": 10 }
Server 2
//Parsed json package int get_fileslist_json_info(char *buf, int *p_start, int *p_count); //Get shared file ranking //127.0 in descending order of Downloads 0.1:80/sharefiles? cmd=pvdesc int get_ranking_filelist(int start, int count);
// a) Compare the number of MySQL shared files with the number of redis shared files to determine whether they are equal //===1. Number of mysql shared files sprintf(sql_cmd, "select count from user_file_count where user=\"%s\"", "xxx_share_xxx_file_xxx_list_xxx_count_xxx"); //===2. Number of redis shared files int redis_num = rop_zset_zcard(redis_conn, FILE_PUBLIC_ZSET);
//b) If they are not equal, clear the redis data and import the data from mysql to redis (interaction between mysql and redis) empty redis Ordered data rop_del_key(redis_conn, FILE_PUBLIC_ZSET); rop_del_key(redis_conn, FILE_NAME_HASH); //Import data from mysql to redis strcpy(sql_cmd, "select md5, filename, pv from share_file_list order by pv desc"); //Add ordered collection members rop_zset_add(redis_conn, FILE_PUBLIC_ZSET, atoi(row[2]), fileid); //Add hash record rop_hash_set(redis_conn, FILE_NAME_HASH, fileid, row[1]);
// c) Read the data from redis and feed back the corresponding information to the front end //Gets the elements of an ordered collection in descending order ret = rop_zset_zrevrange(redis_conn, FILE_PUBLIC_ZSET, start, end, value, &n); //--pv file downloads int score = rop_zset_get_score(redis_conn, FILE_PUBLIC_ZSET, value[i]);
Data returned by the server
{ "filename": "test.mp4", "pv": 0 }
client
Parse data Refresh the interface and display the data
Transmission record
- Upload list and download list, view upload and download
- Transfer record is to write the record into the file when uploading and downloading
All code s
Status code fed back by the server to the front end {"code":"000"} land token: land: success: { "code": "000", "token": "xxx" } Failed: { "code": "001", "token": "xxx" } token Validation succeeded:{"code":"110"} token Validation failed:{"code":"111"} Registration: success:{"code":"002"} The user already exists:{"code":"003"} Failed:{"code":"004"} File transfer in seconds: File already exists:{"code":"005"} Successful transmission in seconds: {"code":"006"} Second transmission failure: {"code":"007"} Upload file: success:{"code":"008"} Failed:{"code":"009"} Share files: success:{"code":"010"} Failed:{"code":"011"} Someone else has shared this file:{"code", "012"} Delete file: success:{"code":"013"} Failed:{"code":"014"} To get a list of user files: Success: file list json Failed:{"code": "015"} Download File pv Field processing success:{"code":"016"} Failed:{"code":"017"} Cancel sharing: success:{"code":"018"} Failed:{"code":"019"} Transfer file: success:{"code":"020"} File already exists:{"code":"021"} Failed:{"code":"022"} //====================Login user 127.0.0.1:80/login post data(json) { user:xxxx, pwd:xxx } //====================Registered user 127.0.0.1:80/reg post data(json) { userName:xxxx, nickName:xxx, firstPwd:xxx, phone:xxx, email:xxx } //======================My files Press the middle Icon: download, share, delete, attribute Not by middle Icon: ascending by downloads, descending by downloads, refreshing and uploading //===1. Second transmission function: 127.0.0.1:80/md5 post data(json) { user:xxxx, md5:xxx, fileName: xxx } //===2. Upload file: 127.0.0.1:80/upload post The data are as follows: ------WebKitFormBoundary88asdgewtgewx\r\n Content-Disposition: form-data; user="mike"; filename="xxx.jpg"; md5="xxxx"; size=10240\r\n Content-Type: application/octet-stream\r\n \r\n Real file content\r\n ------WebKitFormBoundary88asdgewtgewx //===3. My document presentation page: 127.0.0.1:80/myfiles?cmd=count //Get the number of user files post data json The package is as follows: { "user": "yoyo" } //Obtain user file information 127.0 0.1:80/myfiles&cmd=normal //127.0 in ascending order of Downloads 0.1:80/myfiles? cmd=pvasc //127.0 in descending order of Downloads 0.1:80/myfiles? cmd=pvdesc post data json The package is as follows: //start is the starting point of the file location and count the number of files. You need to display 0 ~ 9 locations as files { "user": "yoyo" "start": 0 "count": 10 } Information fed back by the server to the front end { "files": [ { "user": "yoyo", "md5": "e8ea6031b779ac26c319ddf949ad9d8d", "time": "2017-02-26 21:35:25", "filename": "test.mp4", "share_status": 0, "pv": 0, "url": "http://192.168.31.109:80/group1/M00/00/00/wKgfbViy2Z2AJ-FTAaM3As-g3Z0782.mp4", "size": 27473666, "type": "mp4" }, { "user": "yoyo", "md5": "e8ea6031b779ac26c319ddf949ad9d8d", "time": "2017-02-26 21:35:25", "filename": "test.mp4", "share_status": 0, "pv": 0, "url": "http://192.168.31.109:80/group1/M00/00/00/wKgfbViy2Z2AJ-FTAaM3As-g3Z0782.mp4", "size": 27473666, "type": "mp4" } ] } /* { "user": "yoyo", "md5": "e8ea6031b779ac26c319ddf949ad9d8d", "time": "2017-02-26 21:35:25", "filename": "test.mp4", "share_status": 0, "pv": 0, "url": "http://192.168.31.109:80/group1/M00/00/00/wKgfbViy2Z2AJ-FTAaM3As-g3Z0782.mp4", "size": 27473666, "type": "mp4" } */ //-- user User to which the file belongs //--md5 file md5 //--createtime file creation time //--filename file name //-- shared_status share status: 0 means no share and 1 means share //--pv file download volume. The default value is 0. Add 1 to download once //--url file url //--Size file size in bytes //--Type file type: png, zip, mp4 //Share files 127.0.0.1:80/dealfile?cmd=share post data json The package is as follows: { "user": "xxx", "token": "xxx", "md5": "xxx", "filename": "xxx" } //Delete file 127.0.0.1:80/dealfile?cmd=del post data json The package is as follows: { "user": "yoyo", "token": "xxx", "md5": "xxx", "filename": "xxx" } //Download File pv field processing 127.0.0.1:80/dealfile?cmd=pv post data json The package is as follows: { "user": "yoyo", "md5": "xxx", "filename": "xxx" } //======================Shared list Press the middle Icon: download, attribute, cancel sharing, transfer file No icon in: refresh 127.0.0.1:80/sharefiles?cmd=count //Get the number of user files get request //Obtain common shared file information 127.0 0.1:80/sharefiles&cmd=normal //127.0 in ascending order of Downloads 0.1:80/sharefiles? cmd=pvasc //127.0 in descending order of Downloads 0.1:80/sharefiles? cmd=pvdesc Descending by downloads 127.0.0.1:80/sharefiles?cmd=pvdesc post data json The package is as follows: //start is the starting point of the file location and count the number of files. You need to display 0 ~ 9 locations as files { "start": 0, "count": 10 } { "filename": "test.mp4", "pv": 0 } //Download File pv field processing //127.0.0.1:80/dealsharefile?cmd=pv //Cancel sharing files //127.0.0.1:80/dealsharefile?cmd=cancel //Transfer file //127.0.0.1:80/dealsharefile?cmd=save token verification: 1,My files 2,Second transmission 3,Share files 4,Delete file
All databases
-- Create a file named dfs Database. -- create database dfs; -- Delete database dfs -- drop database dfs; -- Use database dfs use dfs; -- =============================================== User information table -- id: User serial number, auto increment, primary key -- name: User name -- nickname: User nickname -- phone: phone number -- email: mailbox -- createtime: time create table user ( id bigint not null primary key AUTO_INCREMENT, name VARCHAR(128) not null, nickname VARCHAR(128) not null, password VARCHAR(128) not null, phone VARCHAR(15) not null, createtime VARCHAR(128), email VARCHAR(100), constraint uq_nickname unique(nickname), constraint uq_name unique(name) ); -- insert -- insert into user (name, nickname, password, phone, createtime, email) values ('mike', 'sb', '123456', '110', '2017-01-11 17:47:30', '110@qq.com' ); -- query -- select id from user where name = "mike"; -- =============================================== Document information table -- md5 file md5 -- file_id file id -- url file url -- size file size, In bytes -- type File type: png, zip, mp4...... -- count File reference count, which is 1 by default. This counter will be for each additional user owning this file+1 create table file_info ( md5 varchar(200) not null primary key, file_id varchar(256) not null, url varchar(512) not null, size bigint, type VARCHAR(20), count int ); -- to update -- update file_info set count = 2 where md5 = "bae488ee63cef72efb6a3f1f311b3743"; -- =============================================== User file list -- user User to which the file belongs -- md5 file md5 -- createtime File creation time -- filename File name -- shared_status Sharing status, 0 Is not shared, 1 is shared -- pv Number of files downloaded. The default value is 0. Add 1 to each download create table user_file_list ( user varchar(128) not null, md5 varchar(200) not null, createtime VARCHAR(128), filename varchar(128), shared_status int, pv int ); -- View a list of files for a user -- select md5 from user_file_list where name = "mike"; -- View the properties of a file -- select * from file_info where md5 = "bae488ee63cef72efb6a3f1f311b3743"; -- Set whether a file is shared -- update user_file_list set shared_status = 1 where md5 = "bae488ee63cef72efb6a3f1f311b3743" and user = 'mike'; -- multi-table query select user_file_list.*, file_info.url, file_info.size, file_info.type from file_info , user_file_list where user = "yoyo" and file_info.md5 = user_file_list.md5; select user_file_list.filename, file_info.size, file_info.type, file_info.md5 from file_info , user_file_list where user = "yoyo" and file_info.md5 = user_file_list.md5 limit 2, 3; -- Query page n+1 Line to m+n Row record select * from table1 limit n, m; SELECT * FROM table LIMIT 5,10;Returns the records from lines 6 to 15 -- Delete a row -- DELETE FROM Person WHERE LastName = 'Wilson' -- =============================================== Quantity table of user files -- user User to which the file belongs -- count Number of files owned create table user_file_count ( user varchar(128) not null primary key, count int ); -- to update -- update user_file_count set count = 10 where user = "mike"; -- delete --delete from user_file_count where user = "mike"; --If the user name is: xxx_share_xxx_file_xxx_list_xxx_count_xxx,Represents the number of shared files -- =============================================== Shared file list -- user User to which the file belongs -- md5 file md5 -- createtime File sharing time -- filename File name -- pv File download volume, the default value is 1, plus 1 for each download create table share_file_list ( user varchar(128) not null, md5 varchar(200) not null, createtime VARCHAR(128), filename varchar(128), pv int );
Usage scenarios of redis
Key value setting of redis
An ordered collection of shared user files, using( ZSET)To simulate key: FILE_PUBLIC_LIST value: Of file contents md5+file name ZSET Related statements: ZADD key score member Add member ZREM key member Delete member ZREVRANGE key start stop [withscores] View in descending order ZINCRBY key increment member Weight accumulation increment ZCARD key return key Number of ordered elements ZSCORE key return key Corresponding score ZREMRANGEBYRANK key start stop Delete members of the specified scope ZLEXCOUNT zset member Judge whether a member exists, return 1 if it exists, and return 0 if it does not exist //===================================================================== Correspondence table of document identification and document name key: FILE_NAME_HASH field: file_id(Of file contents md5+file name) value: file_name redis: hset key field value hget key field
User's token authentication
- My files
- Second transmission
- Share files
- Delete file
It mainly realizes the display function of shared file download list. Because this data is hot data (users often download), the information data is saved in redis to improve efficiency
- Interaction between mysql and redis
-
Judge whether the number of mysql shared files is consistent with the number of redis shared files
-
If it is the same, read the content directly from redis
-
If not, clear the data in the redis cache and re import the data in mysql into redis
-
Read data from redis and save json format
-
Give the data to the server, then give the data to the client, and then present it.
-
json data structure
-
{ "files": [ { "filename":"2.jpg", "pv":12 }, { "filename":"1.jpg", "pv":10 } ] }