Msgpack / Jason performance data comparison

MsgPack is an efficient binary serialization format. Like Jason, it can exchange data between languages and is used to serialize and store data. The advantage is that when the data scale is large, it is more efficient and occupies less resources than Jason; The disadvantage is that it is not easy to view data in binary format.

Rpc interaction is widely used in the industry: JD, Baidu
Official website address: http://pecl.php.net/package/msgpack

The following is a sample experiment to briefly compare the performance of MsgPack with Json:

Experimental data "select string type for sample data and observe the time-consuming and proportion of comparison":

From this sample library:
In the mixed type of String/Int, MsgPack performs better than Json compression / time-consuming when serializing strings; In Int type, the compression ratio is only 1 percentage point worse, and the time-consuming advantage is still prominent. In the scenario where String type accounts for a large proportion of data objects, MsgPack is more suitable than Json.

Note: in the data of sample 1, the time-consuming proportion of MSG / Jason is close to 3 percentage points; Msg/Json compression accounts for an average of 86%. A short string only needs to be 3-4 bytes larger than its length;

Attached sample data:

//"github.com/vmihailenco/msgpack"
//msgpack vs. json code
var rangeObj []string
var testSliceStr  string
testStr :="{\"from_trans\":1,\"uid_real_tags\":\"\",\"aid\":\"\",\"uid\":\"5537471228\",\"idfa\":\"\",\"pst_basic_profile\":200,\"positions\":\"3|8|13|18|23\",\"service\":\"discover_hotspot_feed\",\"oaid\":\"\",\"user_ip\":\"172.17.0.1\",\"mid_s_reason\":\"\",\"server_addr\":\"10.13.40.145\",\"experiment\":{},\"ts\":1617690796,\"blog_author_id\":\"\",\"loadmore\":\"0\",\"id\":\"16176907970138000027\",\"rc\":200,\"uve\":{\"ui_hc\":200,\"u_hc\":200,\"ui_rc\":200,\"u_rc\":0,\"passport\":\";;200|1.9888877868652\"},\"ua\":\"1\",\"engine_time\":107,\"render\":{\"resources\":[{\"ur_hc\":200,\"position\":3,\"ur_rc\":0,\"like\":\"404|0;404:1\",\"mblog\":\"404|0;200|3;\",\"time\":42},{\"ur_hc\":200,\"position\":8,\"ur_rc\":0,\"like\":\"404|0;404:2\",\"mblog\":\"404|0;200|13;\",\"time\":34}]},\"stocks\":{\"mixrank_ads\":\"18|23\",\"u_positions\":\"3|8|13\",\"std_positions\":\"3|8|13|18|23\",\"last_span\":\"-1\",\"interval\":\"5\"},\"owner_uid\":\"\",\"from\":\"1093193010\",\"version\":5,\"idx_id\":\"16176907970138000027\",\"bc_tags\":{\"l2_tags\":\";\",\"keywords\":\"\",\"l1_tags\":\"\"},\"mixrank\":1,\"mid\":\"\",\"filtered_imps\":\"\",\"head_time\":9,\"webview_ua\":\"0\",\"follows\":77,\"sub_service_id\":\"\",\"products\":[{\"positions\":[3,8,13,18,23],\"time\":104,\"rc\":200,\"product\":\"Sfst\",\"cands\":[{\"price\":[160,149,149,0],\"position\":3,\"status\":2000,\"ustatus\":12106,\"factors\":[\"3003\",\"3068807231\",\"\",\"1\",\"783_006\"],\"ur_hc\":500,\"bid\":\"160|1|1|1|160|0|0.00031228387|agent|4000\",\"ad_mid\":\"4610321166109510\",\"rstatus\":22106,\"feature\":\"88030004\",\"result\":\"4610321166109510\"},{\"status\":2030,\"price\":[160,0,0,0],\"position\":8,\"factors\":[\"3003\",\"3068807231\",\"\",\"1\",\"783_006\"],\"result\":\"4610321166109510\",\"ad_mid\":\"4610321166109510\",\"bid\":\"160|1|1|1|160|0|0.00031228387|agent|4000\",\"feature\":\"88030004\"},{\"price\":[1,1,1,0],\"position\":8,\"status\":2000,\"ustatus\":12106,\"factors\":[\"3001\",\"5931191764\",\"\",\"1\",\"783_006\"],\"ur_hc\":500,\"bid\":\"1|1|1|1|1|0|0.00030010002|agent|4000\",\"ad_mid\":\"4620027461112281\",\"rstatus\":22106,\"feature\":\"88030003\",\"result\":\"4620027461112281\"},{\"status\":2030,\"price\":[160,0,0,0],\"position\":13,\"factors\":[\"3003\",\"3068807231\",\"\",\"1\",\"783_006\"],\"result\":\"4610321166109510\",\"ad_mid\":\"4610321166109510\",\"bid\":\"160|1|1|1|160|0|0.00031228387|agent|4000\",\"feature\":\"88030004\"},{\"status\":2030,\"price\":[1,0,0,0],\"position\":13,\"factors\":[\"3001\",\"5931191764\",\"\",\"1\",\"783_006\"],\"result\":\"4620027461112281\",\"ad_mid\":\"4620027461112281\",\"bid\":\"1|1|1|1|1|0|0.00030010002|agent|4000\",\"feature\":\"88030003\"},{\"status\":2030,\"price\":[160,0,0,0],\"position\":18,\"factors\":[\"3003\",\"3068807231\",\"\",\"1\",\"783_006\"],\"result\":\"4610321166109510\",\"ad_mid\":\"4610321166109510\",\"bid\":\"160|1|1|1|160|0|0.00031228387|agent|4000\",\"feature\":\"88030004\"},{\"status\":2030,\"price\":[1,0,0,0],\"position\":18,\"factors\":[\"3001\",\"5931191764\",\"\",\"1\",\"783_006\"],\"result\":\"4620027461112281\",\"ad_mid\":\"4620027461112281\",\"bid\":\"1|1|1|1|1|0|0.00030010002|agent|4000\",\"feature\":\"88030003\"},{\"status\":2030,\"price\":[160,0,0,0],\"position\":23,\"factors\":[\"3003\",\"3068807231\",\"\",\"1\",\"783_006\"],\"result\":\"4610321166109510\",\"ad_mid\":\"4610321166109510\",\"bid\":\"160|1|1|1|160|0|0.00031228387|agent|4000\",\"feature\":\"88030004\"},{\"status\":2030,\"price\":[1,0,0,0],\"position\":23,\"factors\":[\"3001\",\"5931191764\",\"\",\"1\",\"783_006\"],\"result\":\"4620027461112281\",\"ad_mid\":\"4620027461112281\",\"bid\":\"1|1|1|1|1|0|0.00030010002|agent|4000\",\"feature\":\"88030003\"}]},{\"rc\":200,\"time\":7,\"positions\":[3,8,13,18,23],\"product\":\"WAX\"}],\"mac\":\"\",\"user_info\":[],\"imei\":\"da472ec3b036b57b98e71a1d9f970723\",\"unread_status\":\"15\",\"stock\":{\"783_006\":[3,8,13,18,23],\"30_003\":[3,8,13,18,23]},\"darwin_code\":\"200\",\"idx_time\":125}"
for i := 0;i<10000;i++{
    testSliceStr = testSliceStr + testStr
    if i%1000 == 0{
        rangeObj = append(rangeObj,testSliceStr)
    }
}
for i,v:= range rangeObj {
    testSliceStr = v
    //1. Data size comparison
    timeStart := time.Now()
    msgSizeRes, _ := msgpack.Marshal(testSliceStr)
    timeEnd := time.Now()
    duration1 := timeEnd.Sub(timeStart)
    fmt.Println("msg Serialization time:", duration1)


    timeStart = time.Now()
    jsonSizeRes, _ := json.Marshal(testSliceStr)
    timeEnd = time.Now()
    duration2 := timeEnd.Sub(timeStart)
    fmt.Println("json Serialization time:", duration2)
    fmt.Println("content Size before serialization:", len([]byte(testSliceStr)))
    fmt.Println("msg Size after serialization:", len(msgSizeRes))
    fmt.Println("json Size after serialization:", len(jsonSizeRes))
    fmt.Println("msg:json Time ratio:", float32(duration1.Microseconds())/float32(duration2.Microseconds()))
    fmt.Println("msg:json Compression ratio:", float32(len(msgSizeRes))/float32(len(jsonSizeRes)))
    fmt.Printf("QQQQQQ===test:%+v\n", i)
}

Res is as follows:

msg serialization time: 360.833 μ s json serialization time: 369.75 μ s content size before serialization: 3442 msg
Size after serialization: 3445 json size after serialization: 4002 msg:json time ratio: 0.9756098 msg:json compression ratio:
0.8608196 QQQQQQ===test:0

msg serialization time: 5.112875ms json serialization time: 25.18375ms content size before serialization: 3445442
msg size after serialization: 3445447 json size after serialization: 4004002 msg:json time ratio: 0.20299408
msg:json compression ratio: 0.8605008 QQ =

msg serialization time: 13.055833ms json serialization time: 26.396916ms content size before serialization: 6887442
msg size after serialization: 6887447 json size after serialization: 8004002 msg:json time ratio: 0.4945825
msg:json compression ratio: 0.8605004 QQ = = = Test: 2

msg serialization time: 13.461958ms json serialization time: 39.432458ms content size before serialization:
10329442 msg size after serialization: 10329447 json size after serialization: 1200402 MSG: JSON time ratio:
0.3413725 msg:json compression ratio: 0.8605003 QQ =

msg serialization time: 20.243084ms json serialization time: 40.502458ms content size before serialization:
13771442 msg size after serialization: 13771447 json size after serialization: 1600402 MSG: JSON time ratio:
0.49980247 msg:json compression ratio: 0.8605002 QQ = = = Test: 4

msg serialization time: 21.242417ms json serialization time: 50.946584ms content size before serialization:
17213442 msg size after serialization: 17213447 json size after serialization: 20004002 msg:json time ratio:
0.41695127 msg:json compression ratio: 0.8605002 QQ = = = Test: 5

msg serialization time: 10.010417ms json serialization time: 75.589625ms content size before serialization:
20655442 msg size after serialization: 20655447 json size after serialization: 24004002 msg:json time ratio:
0.13242668 msg:json compression ratio: 0.86050016 QQ =

msg serialization time: 2.457291ms json serialization time: 82.623917ms content size before serialization: 24097442
msg size after serialization: 24097447 json size after serialization: 28004002 msg:json time ratio: 0.029737482
msg:json compression ratio: 0.86050016 QQ =

msg serialization time: 2.797875ms json serialization time: 83.432875ms content size before serialization: 27539442
msg size after serialization: 27539447 json size after serialization: 32004002 msg:json time ratio: 0.03352431
msg:json compression ratio: 0.86050016 QQ =

msg serialization time: 3.324417ms json serialization time: 95.866875ms content size before serialization: 30981442
msg size after serialization: 30981447 json size after serialization: 36004002 msg:json time ratio: 0.034673396
msg:json compression ratio: 0.86050016 QQ =