Last week, I finally finished reading the beauty of design patterns. I always wanted to write something to practice. It happened that I wanted to reconstruct the company's upload library recently, so I had this article. (it's just for learning and practicing. If you have any suggestions, the architecture boss can spray it gently)
0x1. Miscellaneous talk
① Why rebuild the upload library?
Before uploading pictures / videos, our APP needs a series of processing before uploading, such as:
- Picture: judge whether the path exists → judge whether rotation is required through Exif information → judge whether compression is required → obtain MD5 → if second transmission is enabled, query whether there is a second transmission record and directly return if there is one → upload only if there is none → upload completes the corresponding status update;
- Video: judge whether the path exists → judge whether compression is required → compress if compression is required → obtain MD5 → obtain the first frame of video → judge whether compression is required → obtain video MD5 again if compression is required → also second transmission verification → upload the video and transmit the first frame picture
The processing of some business scenarios is more complex. Talk is soap, show you the code. It is normal for such codes to appear (partially):
Thanks to rx chain call, the above code is a simplified version. It can be imagined that it will be more chaotic without rx. It is a burden for the writer and for the viewer. Reconstruction is imperative
② Are there any other low-cost solutions without refactoring?
A: Yes, it is optimized based on the above code. flatMap is extracted as a separate function and called according to the process. Of course, it is not too elegant. The most elegant way is to use the Kotlin coroutine, write several suspension functions, and write asynchronous code synchronously. Of course, there are also problems: first, the learning cost and can not be used in Java.
0x2. Demand disassembly
Original demand
Write a picture upload library, give you a local picture path and an upload interface to complete the picture upload.
Xiaobai Perspective
It's simple. You're welcome for the library. Just write an UploadPicUtils tool class and define an upload method
Speed of light code:
object UploadPicUtils {
fun uploadPic(picPath: String, serverUrl: String) {
val pic = File(picPath)
if(pic.exists()) {
// Perform network upload operations (such as calling OkHttp for direct transmission)
// Use rx or EventBus to notify the upload results and give success and failure feedback
}
}
}
// Call when uploading pictures:
UploadPicUtil.uploadPic("Local picture path", "Upload interface")It looks very simple, but the only constant is change. Demand is often capricious~
- Because the company is not willing to buy pictures and add watermark service, the client should add watermark locally before transmitting pictures;
- BUG: some users take pictures with their mobile phones, but the uploaded pictures rotate. Before uploading, check them and straighten them out;
- BUG: some users have reported that uploading pictures is too slow. Once the picture is checked, it is too large for the server to withstand. Before uploading, compress the picture;
- The size of some pictures is specified (X*Y), and those with wrong size cannot be uploaded;
- Second transfer function, do not transfer the same file as md5, and return the address directly;
- Support uploading multiple pictures at the same time;
- Now not only pictures, but also video, audio, files, folders and other scenes
Then the code becomes the above result. The writer is silent and the receiver tears.
Therefore, when you get the original requirements, you should not start with the code, but disassemble, analyze, assume and think about the requirements.
- Really only upload pictures? Will there be anything else, such as audio and video?
- Do you want to check the validity of the picture? For example, whether the file exists, whether the size is 0, and whether the file format is picture type;
- Does the library need to do anything special to upload pictures? Such as watermark, turning, compression, cutting, size verification, etc;
- Whether to support simultaneous uploading of multiple pictures, and how many pictures can be uploaded at the same time;
- Whether to support second transmission;
- Obtain API compatibility for different mobile phone system versions or device files;
- Whether the upload interface address changes all the time, whether authentication is required, and whether there is a specific upload configuration;
- Whether the upload task is performed asynchronously in the background or blocked synchronously in the foreground, and whether the upload can be cancelled in the middle;
- If the upload task is interrupted (kill the APP), do you need to keep the progress and open the APP again next time;
- Whether to retry the upload failure and the maximum number of retries;
- Whether the upload task has priority, etc;
Of course, don't think about giving a perfect design scheme and complete function realization at once. Limited by the designer's architectural experience and limited scheduling, first give a rough and basic available scheme with an iterative basis, and then slowly optimize to minimize feasible products.
0x3. Architecture design
From a macro perspective, the process of file uploading is very similar to the assembly line in the workshop. Take the production process of bagged potato chips as an example:
Potatoes entering the factory → cleaning and peeling → slicing and drying → frying at 350 ℃ → adding salt → filling nitrogen according to grams → bagged potato chips
From potatoes to bagged potato chips, analogy to our single upload task:
Then it is abstracted and simplified into three parts:
Task construction and task completion, a pipelined way of processing tasks, are very suitable for the responsibility chain mode.
The traditional responsibility chain is realized by one-way backward transmission and layer by layer interception until someone handles it.
Here, refer to the implementation of OkHttp interceptor, two-way responsibility chain, and the general principle:
- The Interceptor implementation class calls intercept(Chain) to pass down the Chain instance (including the requests processed by this Interceptor);
- The last interceptor calls chain The processed() returns the Response instance and passes it up recursively;
Here, you can start with tasks and then disassemble them later, so the combination of individual tasks becomes:
How many interceptors before the request → execute the upload request → how many interceptors after the request
When the upload request is executed, it is left to the user for customization. The method of request construction and sending request is provided. Whether it is successful or not can be notified through callback.
This is the case when a single task is uploaded. Multiple tasks also need: task queue, poller and thread pool.
When an upload task is initiated, the task is added to the queue. The poller continuously takes the task from the queue (until there is no task) and takes a thread from the thread pool to execute the task.
The general principle is very clear, and then the specific code implementation. The code comments are written in detail, so we won't explain them one by one~
0x4. Use of Library
At present, it is still written to play (a pile of pits, vegetables and chickens are changing while stepping ~). If you are interested, you can star, warehouse address: github.com/coder-pig/C...
Add dependency:
allprojects {
repositories {
...
maven { url 'https://jitpack.io' }
}
}
dependencies {
implementation 'com.github.coder-pig:CpLightUpload:v0.0.3'
}① Custom Task
Different scenarios may have different requirements. Customize the attributes as needed:
class CpImageTask : ImageTask() {
var needRotate: Boolean? = null
var needCompress: Boolean? = null
var compressPercent: Int? = 80
}
class CpVideoTask : VideoTask() {
var limitSize: Int? = -1 // Video limit size
var compressVideoPath: String? = null // Compressed video path
var compressVideoMD5: String? = null // Compressed video MD5
var firstFramePath: String? = null // Video first frame path
var firstFrameMD5: String? = null // Video first frame MD5
}② Custom upload configuration
This is the default configuration for uploading. When the corresponding item of the uploaded Task is not configured, the default configuration is filled in:
class ImageUploadConfig : LightUploadConfig() {
var needRotate: Boolean = true // Is rotation correction required
var needCompress: Boolean = true // Need compression
var compressPercent: Int = 80 // Compression scale, default 80
}
class VideoUploadConfig : LightUploadConfig() {
// Customize on demand
}③ Custom front interceptor
Inherit the Interceptor interface and implement the intercept() method:
class PictureRotateInterceptor : Interceptor {
override fun intercept(chain: Interceptor.Chain): Task {
val task = chain.task()
val config = task.config as? ImageUploadConfig
if (task is CpImageTask) {
if(task.needRotate == null) task.needRotate = config?.needRotate
"============ Determine whether to flip the picture ============".logV()
val degree = FileUtils.readPictureDegree(task.filePath!!)
if (degree != 0) {
"Picture rotation correction".logV()
FileUtils.rotateToDegrees(task.filePath!!, degree.toFloat())
"Image rotation processing completed".logV()
} else {
"No rotation correction is required.".logV()
}
}
// Pass down
return chain.proceed(task)
}
}
class VideoFrameInterceptor : Interceptor {
override fun intercept(chain: Interceptor.Chain): Task {
val task = chain.task()
if (task is CpVideoTask) {
"Generate video thumbnails...".logV()
// Get the first frame file name
val tag = task.compressVideoPath!!.substring(task.compressVideoPath!!.lastIndexOf("/"))
val frameFile = File(getExternalVideoPath() + tag + ".jpg")
task.firstFramePath = frameFile.absolutePath
val mmr = MediaMetadataRetriever()
mmr.setDataSource(task.compressVideoPath!!)
val frameBitmap = mmr.frameAtTime
FileUtils.compressImage(frameBitmap, frameFile, 80)
task.firstFrameMD5 = FileUtils.getFileMD5ToString(frameFile)
LightUpload.upload(task = CpImageTask().apply {
filePath = task.firstFramePath
md5 = task.firstFrameMD5
})
frameBitmap?.recycle()
}
return chain.proceed(task)
}
}④ Custom request
Implement the Upload abstract class, override the initRequest() and sendRequest() methods, and call back different request results:
class HucUpload : Upload() {
override fun sendRequest() {
"Start file upload...".logV()
var ins: InputStream? = null
try {
mTask.reqData?.let { req ->
val conn = (URL(req.uploadUrl).openConnection() as HttpURLConnection).apply {
readTimeout = req.timeout!!
connectTimeout = req.timeout!!
doInput = true
doOutput = true
useCaches = false
requestMethod = req.requestMethod
// Request header settings
val boundary = UUID.randomUUID()
req.headers["Content-Type"] = "multipart/form-data;boundary=${boundary}"
for ((k, v) in req.headers) setRequestProperty(k, v)
val dos = DataOutputStream(outputStream)
val sb = StringBuilder().append("--").append(boundary).append("\r\n")
.append("Content-Disposition: form-data; name=\"file\"; filename=\"")
.append(mTask.md5).append(mTask.fileName).append("\"")
.append("\r\n")
.append("Content-Type: application/octet-stream; charset=utf-8")
.append("\r\n").append("\r\n")
dos.write(sb.toString().toByteArray())
ins = FileInputStream(File(mTask.filePath!!))
val bytes = ByteArray(1024)
var len: Int
while (ins!!.read(bytes).also { len = it } != -1) {
dos.write(bytes, 0, len)
}
ins!!.close()
dos.write("\r\n".toByteArray())
val endData: ByteArray = "--$boundary--\r\n".toByteArray()
dos.write(endData)
dos.flush()
}
// Get response
val input = BufferedReader(InputStreamReader(conn.inputStream, "UTF-8"))
val sb1 = StringBuilder()
var ss: Int
while (input.read().also { ss = it } != -1) {
sb1.append(ss.toChar())
}
val result = sb1.toString()
"End of file upload...".logV()
mTask.response = Response(conn.responseCode, result)
mTask.status = TaskStatus.DONE
mCallback?.onSuccess(mTask)
}
} catch (e: IOException) {
e.message?.logE()
mTask.status = TaskStatus.FAILURE
mTask.throwable = e
mCallback?.onFailure(mTask)
LightUpload.postTask(mTask)
} finally {
if (ins != null) {
try {
ins!!.close()
} catch (e: IOException) {
e.message?.logE()
}
}
}
}
}⑤ Custom post interceptor
Process the response data, such as string parsing display
class SimpleParsingInterceptor: Interceptor {
override fun intercept(chain: Interceptor.Chain): Task {
val task = chain.task()
if(task is ImageTask) {
task.response?.let {
var tempContent = it.content
if(tempContent.startsWith("{")) {
val index: Int = tempContent.indexOf("{")
tempContent = tempContent.substring(index)
}
try {
val jsonObject = JSONObject(tempContent)
if (jsonObject.getInt("code") == 200) {
//Parsing the returned content from the server
val mapJson: JSONObject = jsonObject.getJSONObject("data")
var key = ""
var image = ""
val ite = mapJson.keys()
while (ite.hasNext()) {
key = ite.next()
image = mapJson[key] as String
}
task.fileUrl = image
task.fileUrl?.logV()
} else {
jsonObject.toString().logV()
}
} catch (e: Exception) {
e.message?.logD()
}
}
}
return chain.proceed(task)
}
}⑥ Initialization
You can not initialize in the App class, as long as you ensure that init() is completed before upload~
LightUpload.init(LightUploadBuilder()
// Pass in the default configuration and variable parameters, and support the customization of multiple types of tasks
.config(LightUploadTask.IMAGE to ImageUploadConfig().apply {
reqData = ReqData(
uploadUrl = "http://127.0.0.1:5000/upload",
requestMethod = "POST",
headers = hashMapOf(
"Charset" to "utf-8",
"connection" to "keep-alive"
)
)
}, LightUploadTask.VIDEO to VideoUploadConfig()
.apply {
reqData = ReqData(
uploadUrl = "http://127.0.0.1:5000/upload",
requestMethod = "POST",
headers = hashMapOf(
"Charset" to "utf-8",
"connection" to "keep-alive"
)
)
})
// Setting the upload request is also a variable parameter and supports multiple types of customization
.upload(LightUploadTask.IMAGE to HucUpload())
// Add front interceptor
.addBeforeInterceptor(PictureRotateInterceptor())
.addBeforeInterceptor(PictureCompressInterceptor())
.addBeforeInterceptor(VideoCompressInterceptor())
.addBeforeInterceptor(VideoFrameInterceptor())
// Add post interceptor
.addDoneInterceptors(SimpleParsingInterceptor())
)⑦ Call upload
LightUpload.upload(task = CpImageTask().apply {
filePath = path
needCompress = true
compressPercent = (1..100).random()
callback = object : Upload.CallBack {
override fun onSuccess(task: Task) {
// Successful callback
text = " ${task.response!!.content}\n"
}
override fun onFailure(task: Task) {
// Failed callback
task.throwable?.message?.let { it1 -> shortToast(it1) }
}
}
})0x5. Demo test
cd the command line to the upload server project of the project. Before the first run, execute the following commands to install the dependencies related to python scripts:
pip install -r pip install requirements.txt
After installation, type the following command to run the script:
python app.py
Then, the mobile phone and the computer are on the same LAN, configure the agent, and enter ipconfig to view the local IP
Configure the mobile phone and open charles packet capture:
The operation effect is as follows:
Logcat can also see the output information:
