Lua cjson is a simple and compact open source library that can be loaded by Lua script require. In Lua, the mutual conversion (encoding and decoding) between Lua value and Json value is completed through a series of lua cjson API calls.
A problem is found in the use of lua cjson: after coding with cjson.encode, the order of strings is inconsistent, which leads to different results of two strings when calculating md5.
Based on the examples extracted from the project, this paper deeply analyzes the insertion process of lua table, and finally puts forward an appropriate solution to the inconsistency of table order caused by Lua cjson.
1, Problem example
Take the following table data as an example:
--object type
local lua_object = {
["name"] = "Jiang",
["age"] = 18,
["addr"] = "wuhan",
["email"] = "123456789@163.com",
["tel"] = "1581475xxxx",
}
local lua_object2 = {
["addr"] = "wuhan",
["email"] = "123456789@163.com",
["tel"] = "1581475xxxx",
["name"] = "Jiang",
["age"] = 18,
}
Handle them with json.encode (JSON table to string):
local function Test(tb)
print("1---------------------")
for key, value in pairs(tb) do
print("key = " .. key)
end
print("2---------------------")
local str1 = cjson.encode(tb)
print("3----------------------")
print("str1 = " .. str1)
end
print("[lua_object")
Test(lua_object)
print("lua_object]")
print("\n")
print("[lua_object2")
Test(lua_object2)
print("lua_object2]")
print("\n")
After running, the output result is:
It can be seen that the internal storage order of a table is also different for different table initialization orders. After json.encode, the generated string s must be different.
Let's see what causes it from the source code analysis of lua table.
2, Lua Table principle analysis
The table in lua is a mixture of array and hash table. Only through the table in lua, you can realize the functions of module, meta table, environment, and even object-oriented;
(this section will explain the table code in Lua, because the implementation principle of lua is the same as that of luakit, but the code readability of lua is much better than that of luakit).
Its structure is defined in lobject.h:
typedef struct Table {
CommonHeader;
lu_byte flags; /* 1<<p means tagmethod(p) is not present */
//The length of the hash part. The index is 1 < < lsizenode to get the actual length
lu_byte lsizenode; /* log2 of size of `node' array */
struct Table *metatable;
//The first address of the array part is an array composed of TValue type
TValue *array; /* array part */
//The hash part, an array of nodes, may be more appropriate to call it a bucket
Node *node;
//The idle bucket bit is initialized as the last bucket and moved forward from the back
Node *lastfree; /* any free position is before this position */
GCObject *gclist;
//Length of array part
int sizearray; /* size of `array' array */
} Table;
The array part is relatively simple. We won't pay attention to it for the time being. Let's mainly look at the hash part and look at the following structures first:
/*
** Union of all Lua values
*/
typedef union {
GCObject *gc;
void *p;
lua_Number n;
int b;
} Value;
#define TValuefields Value value; int tt
//The general structure of data storage in lua is actually composed of a union and an int type
typedef struct lua_TValue {
TValuefields;
} TValue;
typedef union TKey {
struct {
TValuefields;
struct Node *next; /* for chaining */
} nk;
TValue tvk;
} TKey;
typedef struct Node {
TValue i_val;
TKey i_key;
} Node;
From the structure, we can see that the TKey of Node is a union structure, which can be either a simple TValue or a linked list Node.
If table uses hash table (bucket) to store data, conflicts will certainly be involved. lua uses the chain address method, that is, the next data of nk in TKey structure. The graph may be more intuitive (red nodes represent non nil values, and white nodes represent nil values (that is, idle nodes):
Among them, the hash values calculated by the keys of the three nodes 0, 3 and 4 are the same. The slot should have been No. 0, but because No. 0 is occupied preferentially, the other two can only find another space, so they can find positions 3 and 4. Then, in order to express their relationship, use next to point to the next node (for searching)
So why are the other two nodes looking for slots 3 and 4 instead of slots 1 and 2?
Let's look at the insert code of table: (in ltable.c)
/*
** inserts a new key into a hash table; first, check whether key's main
** position is free. If not, check whether colliding node is in its main
** position or not: if it is not, move colliding node to an empty place and
** put new key in its main position; otherwise (colliding node is in its main
** position), new key goes to an empty position.
*/
static TValue *newkey (lua_State *L, Table *t, const TValue *key) {
int a = 0;
if (ttype(key) == LUA_TSTRING){
a = lmod(((rawtsvalue(key))->tsv.hash), sizenode(t));
}
Node *mp = mainposition(t, key);
if (!ttisnil(gval(mp)) || mp == dummynode) {
Node *othern;
Node *n = getfreepos(t); /* get a free place */
if (n == NULL) { /* cannot find a free place? */
rehash(L, t, key); /* grow table */
return luaH_set(L, t, key); /* re-insert key into grown table */
}
lua_assert(n != dummynode);
othern = mainposition(t, key2tval(mp));
if (othern != mp) { /* is colliding node out of its main position? */
/* yes; move colliding node into free position */
while (gnext(othern) != mp) othern = gnext(othern); /* find previous */
gnext(othern) = n; /* redo the chain with `n' in place of `mp' */
*n = *mp; /* copy colliding node into free pos. (mp->next also goes) */
gnext(mp) = NULL; /* now `mp' is free */
setnilvalue(gval(mp));
}
else { /* colliding node is in its own main position */
/* new node will go into free position */
gnext(n) = gnext(mp); /* chain new position */
gnext(mp) = n;
mp = n;
}
}
gkey(mp)->value = key->value; gkey(mp)->tt = key->tt;
luaC_barriert(L, t, key);
lua_assert(ttisnil(gval(mp)));
return gval(mp);
}
The comments of the code are written clearly. Briefly describe the whole insertion process:
- (lines 13-14 and 38-41 of the code) find the mainposition of the hash bucket according to the key. If the value of the node is nil in the returned result, directly assign the key and return the TValue pointer of the node;
- (lines 16-20 of the code) if the node corresponding to the mainposition is not empty or dumynode, first obtain a freenode. If the freenode cannot be obtained, re hash the data in the previous hash bucket. The call order is newkey - > rehash - > resize. At this time, the re hash order is in reverse order:
- (lines 31-36 of the code) if freenode is obtained, retrieve mainposition again in reverse according to the node value of the mainposition, and verify whether the mainposition of the mp node taken out before is the current position. If so, store the node to be inserted in the freenode taken out and link it to the mp node
- (lines 23-30 of the code) if the mp node's mainposition is not the current position, find an empty slot for the mp node again and store the current node to be inserted in the original position of the mp.
The function to find free slots is getfreeops, which starts from the last slot and stops when the first value is nil:
static Node *getfreepos (Table *t) {
while (t->lastfree-- > t->node) {
if (ttisnil(gkey(t->lastfree)))
return t->lastfree;
}
return NULL; /* could not find a free place */
}
3, Sample analysis
In order to deepen the understanding of table insertion, we use the above demo to demonstrate:
A. First, the insertion sequence of lua_object initialization is: "name" → "age" → "addr" → "email" → "tel"
(1) When inserting "name", the calculated hash value is 0, occupying bucket 0
(2) When "age" is inserted, the calculated hash value is also 0. At this time, freenode is obtained. If it cannot be obtained, resize. After resizing, the bucket array length is 2 ^ 1 = 2;
Carry out heavy hash(Traverse from back to front and then hash)，hash("name")=0，hash("age")=0
therefore aget obtain freenode，That is, the last bucket (bucket 1), and place bucket 1 behind the linked list of bucket 0
The final location is:
(3) When "addr" is inserted, the calculated hash value is 1. At this time, mainposition obtains a dumynode, so freenode is obtained again. Because the hash bucket length is not enough, resize again. After resizing, the bucket array length is 2 ^ 2 = 4;
Carry out heavy hash(Traverse from back to front and then hash)，hash("age")=2，hash("name")=2，hash("addr")=1
therefore name obtain freenode，That is, the last bucket (bucket No. 3), and place bucket 3 behind the linked list of bucket 2
The final location is:
4) When inserting "email", the calculated hash value is 3. Bucket 3 has been occupied by "name", but this is not the main position of "name", so give the location of bucket 3 to "email";
Find another free bucket for "name" and move to the head of the array. Bucket 2 and bucket 1 are occupied, only bucket 0
The final location is:
(5) When "tel" is inserted, the calculated hash value is 2. At this time, freenode is obtained. If it cannot be obtained, resize. After resizing, the bucket array length is 2 ^ 3 = 8;
Re hash((traverse from back to front and then hash)), hash("email") = 7, hash("age") = 6, hash("addr") = 1, hash("name") = 2, hash("tel") = 6
Bucket 6 is occupied, and its "age" is the main position of bucket 6. Therefore, find an empty bucket for "tel". Bucket 7 and bucket 6 can't work. Traverse to find bucket 5, and link bucket 5 to bucket 6
The final location is:
The order is the same as the table output after "original" in the above demo.
B. When lua_object2 is initialized again, the insertion sequence changes to: "addr" → "name" → "tel" → "age" → "email";
Use the same analysis process, which will not be demonstrated here. Paste the notes during analysis:
1, "addr":0
2, "name":0 ->(Due to insufficient space, it is 1 at this time, resize) "addr" : 1
"name" : 0
3, "tel" :0 ->(Due to insufficient space, it is 2 at this time, resize)
resize Before, bucket 1 contains "yes" addr": "wuhan"
Bucket 0 contains "yes" name": "Jiang"
resize The rear barrel 1 holds "yes" addr": "wuhan"
Barrel 2 holds "yes" name": "Jiang"
Do it again at this time hash Calculation, " tel" hash The value is 2
But in position 2“ name"，its mainposition It's really 2, so " tel"The obtained free bucket is bucket 3, and bucket 3 is linked to the back of bucket 2
Therefore, the position occupancy after this operation is: addr": Barrel 1
"name": Barrel 2
"tel": Barrel 3
4，"age": 2 ->(At this time, the bucket space is enough for 4, and there is another bucket No. 0 empty)
At the same time, due to the change of position 2 key，its mainposition It is indeed 2 (" name"),So here you are. " tel"Find the only free bucket 0, and link bucket 0 to the back of bucket 2, and bucket 3 to the back of bucket 0 (No. 2)->0 No. 3)
Therefore, the position occupancy after this operation is: age": Bucket 0
"addr": Barrel 1
"name" : Barrel 2
"tel" : Barrel 3
5，"email": 3->(Due to insufficient space, it is 4 at this time, resize，Extended to 2^3=8)
Reverse the order of the original barrels before proceeding hash Calculation: (reverse order, first) tel，again age)
"age"(The hash value is 6, causing a conflict) -> "addr"(The hash value is 1) -> "name"(The hash value is 2) -> "tel"(The hash value is 6)
among age The hash value is 6, and the idle bucket 7 needs to be taken
And idle nodes n Hang on " tel"Behind the barrel
Again hash，"email": 7
But because of the above“ age"Hang on“ tel"After that, it occupied barrel 7,
Because“ age"The main barrel is not No. 7, so“ email"Take barrel 7. Here you are“ age"Find a free bucket again. The free bucket in this space-time is No. 5 (No. 7 and No. 6 are occupied, look forward from the back)
Therefore, the final order is:
"addr"(Hash value is 1, bucket 1) -> "name"(Hash value is 2, bucket 2) ->"age"(Hash value is 6, occupying idle bucket 5) -> "tel"(The hash value is 6, occupying bucket 6) ->"email"(Hash value is 7, occupying bucket 7)
The final table position order is:
Since then, we can find that the reason for the inconsistent order in the process of lua table to json string is the hash mechanism at the bottom of the table. Under this mechanism, the final generated table key order is also inconsistent with different insertion order.
4, Solution
1. Scheme 1
Since different insertion order will affect the key order of the table, when calling cjson.encode to convert the table structure into json string, we first sort the keys, so that the generated json string is sequential (natural order from small to large);
Take a look at the core function added (added to lua_cjson.c):
static void sort_lua_stack(lua_State *l, int low, int high)
{
if (low >= high)
return;
int i = low - 1;
int j = high + 1;
lua_pushvalue(l, low);
//Sorting, fast scheduling based on dichotomy, average time complexity nLogn
while (1) {
//Find the first index greater than index=-1 from left to right
while (lua_lessthan(l, -1, ++i));
//Find the first value subscript less than index=-1 from right to left
while (lua_lessthan(l, --j, -1));
if (i >= j)
break;
//Exchange the values of i and j
lua_pushvalue(l, i);
lua_pushvalue(l, j);
lua_replace(l, i);
lua_replace(l, j);
}
lua_pop(l, 1);
//At this time, the values on the left of J are less than j, and the values on the right of J are greater than J
//Therefore, the left and right are sorted separately
sort_lua_stack(l, low, j);
sort_lua_stack(l, j + 1, high);
}
Sort the values in lua stack from small to large (natural order), then traverse the sorted table and parse it into json string;
The call is in the json_append_object function:
static void json_append_object(lua_State *l, json_config_t *cfg,
int current_depth, strbuf_t *json)
{
.....
lua_pushnil(l);
/* table, startkey */
if (current_depth == 1) {
object_start_pos = strbuf_length(json);
}
if (sort) {
while (lua_next(l, tbl_index)) {
lua_pop(l, 1);
lua_pushvalue(l, -1);
}
sort_lua_stack(l, tbl_index + 1, lua_gettop(l));
}
comma = 0;
//while (lua_next(l, -2) != 0)
while (sort ? lua_gettop(l) > tbl_index : lua_next(l, tbl_index)) {
if (comma){....}
if (sort) {
lua_pushvalue(l, -1);
lua_gettable(l, tbl_index);
}
......
//lua_pop(l, 1);
lua_pop(l, sort ? 2 : 1);
/* table, key */
}
strbuf_append_char(json, '}');
}
Then add a new interface of sort switch:
static int lua_cjson_new(lua_State *l)
{
luaL_Reg reg[] = {
{ "encode", json_encode },
......
{ "encode_sort_keys", json_cfg_encode_sort_keys }, //libing add
......
};
....
return 1;
}
/* Configures whether object keys are sorted when encoding */
static int json_cfg_encode_sort_keys(lua_State *l)
{
json_config_t *cfg = json_arg_init(l, 1);
return json_enum_option(l, 1, &cfg->encode_sort_keys, NULL, 1);
}
Modify the demo of the test and add the sort switch before the decode:
...... cjson.encode_sort_keys(true) local a = cjson.encode(tb) ......
Run again and the output is:
The above sorting function stores all the keys of table in lua's stack, but the number of stacks is limited. LUA_MINSTACK=20 by default. Therefore, lua will report an error when the key data of table is greater than 20.
Therefore, you also need to ensure that there are enough stack slots in the current lua virtual machine and modify them in the json_append_object function:
static void json_append_object(lua_State *l, json_config_t *cfg,
int current_depth, strbuf_t *json)
{
.....
if (sort) {
while (lua_next(l, tbl_index)) {
lua_checkstack(l, 1);
lua_pop(l, 1);
lua_pushvalue(l, -1);
}
sort_lua_stack(l, tbl_index + 1, lua_gettop(l));
}
}
Verify that the json string with more than 20 key s is normal.
However, because the stack height in lua is limited to LUAI_MAXCSTACK, the default value is 2048, that is, 2048 key value pairs are supported at most.
2. Scheme 2
In order to solve the limitation of stack 2048 in lua, scheme 2 is proposed: instead of borrowing the stack in lua to sort key values, a user-defined array is used to store the sorted keys and corresponding values. The main ideas are as follows:
- When parsing an object, get the size of the table and dynamically create TValue type arrays pKey and pValue;
- Traverse the table and store the copies of key and value in the table into pKey and pValue arrays;
- Sort the pKey array (here is also the fast sorting modified by the binary idea, with time complexity O(nlogn). Other sorting algorithms are also tried, such as single linked list insertion sorting, which takes basically the same time). At the same time, the array subscript of pValue is also updated
- Traverse the sorted pKey, continue recursive processing, and parse it into a json string
The code implementation is mainly as follows:
First add a json_append_object_ex interface:
//Support sorting object resolution
static void json_append_object_ex(lua_State *l, json_config_t *cfg,
int current_depth, strbuf_t *json)
{
int comma, keytype;
size_t len;
const char *key = "req_id";
int object_start_pos = 0;
int req_id_start_pos = 0;
int sort = cfg->encode_sort_keys;
//Gets the number of elements in the stack
int tbl_index = lua_gettop(l);
/* Object */
strbuf_append_char(json, '{');
//There is a nil at the top of the stack
lua_pushnil(l);
/* table, startkey */
if (current_depth == 1) {
object_start_pos = strbuf_length(json);
}
//Create two new arrays to store copies of key and value respectively
int n = get_table_count(l, tbl_index); //Get the number of table s
TValue *pKey = (void*)malloc(sizeof(TValue)*n);
TValue *pValue = (void*)malloc(sizeof(TValue)*n);
//libing add sort keys
int index = 0;
//Traverse the table once and take out all copies of key and value
while (lua_next(l, tbl_index)) {
//next stack key first, then stack value
/*const char* pp = lua_tolstring(l, -1, &len);
if (pp){
printf("pp = %s\n", pp);
}*/
/*const char* pp = lua_tolstring(l, -2, &len);
if (pp){
printf("pp = %s\n", pp);
}*/
StkId pStkid = ((StkId)(lua_tovoidptr(l, -1)));
pValue[index].value = pStkid->value;
pValue[index].tt = pStkid->tt;
pStkid = ((StkId)(lua_tovoidptr(l, -2)));
pKey[index].value = pStkid->value;
pKey[index].tt = pStkid->tt;
index++;
//Pop up value
lua_pop(l, 1);
}
//Then sort pKey and adjust the position of pValue
sort_table_key(l, pKey, pValue, 0, n-1);
comma = 0;
index = 0;
while (index < n) {
if (comma)
strbuf_append_char(json, ',');
else
comma = 1;
/* table, key, value */
keytype = pKey[index].tt;
if (keytype == LUA_TNUMBER) {
strbuf_append_char(json, '"');
json_append_number(l, cfg, json, -2);
strbuf_append_mem(json, "\":", 2);
} else if (keytype == LUA_TSTRING) {
const char* str = lua_ptrtolstring(l, &pKey[index], &len);;//lua_tolstring(l, -2, &len);
if (str != NULL) {
if ((object_start_pos > 0)
&& (strncmp(str, key, len) == 0)) {
req_id_start_pos = strbuf_length(json);
}
json_append_string(json, str, len);
strbuf_append_char(json, ':');
}
} else {
json_encode_exception(l, cfg, json, -2,
"table key must be a number or string");
/* never returns */
}
/* table, key, value */
//You need to fill value into lua stack
lua_pushpointer(l, &pValue[index]);
json_append_data(l, cfg, current_depth, json);
if ((object_start_pos > 0)
&& (req_id_start_pos > object_start_pos)) {
strbuf_append_char(json, ',');
strbuf_move_buf(json, object_start_pos, req_id_start_pos);
strbuf_reduce_char(json, ',');
object_start_pos = 0;
req_id_start_pos = 0;
}
//Pop up the lua_pushpointer on the stack
lua_pop(l, 1);
index++;
}
if (pKey){
free(pKey);
}
if (pValue) {
free(pValue);
}
strbuf_append_char(json, '}');
}
In order to implement the json_append_object_ex function, many interfaces are added in Lua, such as lua_tovoid PTR, lua_pushpointer, etc
//Turn StkId (that is, TValue) into void * pointer. -- shallow copy, save value
LUA_API void *lua_tovoidptr(lua_State *L, int idx){
StkId o = index2adr(L, idx);
return (void*)o;
}
//Convert the above converted void * to StkId and put it on the stack
LUA_API void lua_pushpointer(lua_State *L, void *p){
lua_lock(L);
StkId t = (StkId)p;
setobj(L, L->top, t);
api_incr_top(L);
lua_unlock(L);
}
sort_table_key implements the sorting of pKey:
//Fast scheduling based on dichotomy, average time complexity nLogn
//In the process of sorting keys, you should also adjust the value of value to avoid subsequent use. You should also find value according to the key
static void sort_table_key(lua_State *l, TValue *pKey, TValue *pValue, int low, int high)
{
if (low >= high)
return;
TValue key = pKey[low];
TValue value = pValue[low];
int i = low;
int j = high;
for (;;){
Use it for the time being lua Sorting method of
Find the first greater than from left to right index=-1 Value subscript of(That is, greater than the top of the stack)
//while (lua_lessthan_ex(l, (void*)(&key), (void*)(&pKey[++i])) && i < j);
Find the first less than from right to left index=-1 Value subscript of(That is, less than the top of the stack)
//while (lua_lessthan_ex(l, (void*)(&pKey[j--]), (void*)(&key)) && i < j);
//Find the first value less than key from right to left
while (lua_lessthan_ex(l, (void*)(&pKey[low]), (void*)(&pKey[j])) && i < j)
{
--j;
}
//Then look from left to right
//Find the first value greater than key from left to right
while (lua_lessthan_ex(l, (void*)(&pKey[i]), (void*)(&pKey[low])) && i < j)
{
++i;
}
if (i >= j)
break;
//Swap the values corresponding to the i and j positions
TValue pTemp = pKey[i];
pKey[i] = pKey[j];
pKey[j] = pTemp;
//Update the value corresponding to i and j
pTemp = pValue[i];
pValue[i] = pValue[j];
pValue[j] = pTemp;
}
pKey[low] = pKey[i];
pKey[i] = key;
pValue[low] = pValue[i];
pValue[j] = value;
//At this time, the values on the left of J are less than j, and the values on the right of J are greater than J
//Therefore, the left and right are sorted separately
sort_table_key(l, pKey, pValue, low, i - 1);
sort_table_key(l, pKey, pValue, i + 1, high);
}
lua_lessthan_ex implements < = comparison of lua structure:
LUA_API int lua_lessthan_ex(lua_State *L, void* ptr1, void* ptr2){
StkId o1, o2;
int i;
o1 = (StkId)ptr1;
o2 = (StkId)ptr2;
//First judge whether it is equal
i = (o1 == NULL || o2 == NULL) ? 0
: equalobj(L, o1, o2);
//If it is not equal, judge whether it is less than
if (i != 1){
i = (o1 == NULL || o2 == NULL) ? 0
: luaV_lessthan(L, o1, o2);
}
//Equality returns 1 directly
return i;
}
Add another encode_sort interface to facilitate calling;
The final effect is the same as that of sorting scheme 1, which will not be shown here.
3. Performance comparison
Since the above modification involves sorting, the performance of cjson.encode before and after modification is verified;
Use a json string with a size of 220kb to perform a certain number of encode operations respectively. The test time is as follows:
From the experimental results:
1. The performance of the two sorting schemes is similar (from the data point of view, scheme 2 is slightly better than scheme 1), and scheme 2 is more applicable and is not limited to the stack height of 2048 in lua;
2. The processing time of sorting scheme 2 is about 2-2.5 times that of non sorting;
3. Sorting scheme 2 uses additional memory. A key:value occupies 16 * 2 = 32 bytes (64 bits) in total. For a case of 2000 key: values, the peak may account for 62.5k more. The overhead is relatively small, which has little impact on the overall memory occupation