Today, I'd like to talk a little bit about JSON data processing. In recent work, because we want to update the database data to elastic search in real time, we encounter some problems of JSON data processing in practice.

real-time data

The real-time data acquisition is realized by the open-source canal component of Alibaba, and transmitted to the handler through the message queue kafka. The JSON data we will receive is similar to the following.

{
    "type": "UPDATE",
    "database": "blog",
    "table": "blog",
    "data": [
        {
            "blogId": "100001",
            "title": "title",
            "content": "this is a blog",
            "uid": "1000012",
            "state": "1"
        }
    ]
}

Simply speaking, in the logic of data, type indicates whether the database event is a new, updated or deleted event, database indicates the corresponding database name, table indicates the corresponding table name, and data is the data in the database.

How to deal with this string of JSON?

json to map

The first thought is to transform JSON into map[string]interface {} through json.Unmarshal.

Example code:

func main () {
    msg := []byte(`{
    "type": "UPDATE",
    "database": "blog",
    "table": "blog",
    "data": [
        {
            "blogId": "100001",
            "title": "title",
            "content": "this is a blog",
            "uid": "1000012",
            "state": "1"
        }
    ]}`)
    var event map[string]interface{}
    if err := json.Unmarshal(msg, &event); err != nil {
        panic(err)
    }

    fmt.Println(event)
}

The printing results are as follows:

map[data:[map[title:title content:this is a blog uid:1000012 state:1 blogId:100001]] type:UPDATE database:blog table:blog]

At this point, the data is successfully parsed. The next step is to use it, but I think map usually has several shortcomings.

• To obtain data through key, there may be nonexistent key. In order to be strict, it is necessary to check whether the key exists.
• Compared with the way of structure, map data extraction is inconvenient and IDE can not be used to complete the check, key is easy to write wrong;

How can we deal with this situation? If only JSON could be transformed into struct.

json to struct

In GO, it is also very convenient to convert json into struct. You only need to define the transformed struct in advance. Let's define the transformed struct first.

type Event struct {
    Type     string              `json:"type"`
    Database string              `json:"database"`
    Table    string              `json:"table"`
    Data     []map[string]string `json:"data"`
}

Explain several points

• In the actual scenario, the data structure of the canal message is determined by the table and cannot be known in advance before the JSON is parsed successfully, so it is defined as map[string]string here;
• The converted struct members must be exportable, so the member variable names are all uppercase, and the mapping with JSON is completed through the tagName of json:"tagName".

The parsing code is very simple, as follows:

e := Event{}
if err := json.Unmarshal(msg, &e); err != nil {
    panic(err)
}

fmt.Println(e)

Print results:

{UPDATE blog blog [map[blogId:100001 title:title content:this is a blog uid:1000012 state:1]]}

Next, the use of data is more convenient. For example, event type can be obtained by event.Type. However, pour cold water on it, because data is still of type [] map[string]string, and there are still problems with map.

Can map be transformed into struct?

map to struct

As far as I know, GO is not built-in when map is converted to struct. If we want to implement it, we need to rely on the reflection mechanism of GO.

Fortunately, someone has already done it. The package name is mapstructure , and it's very easy to use. Type several of them. Example So I learned. README also said that the library mainly meets the scenario that a part of JSON must be read to know the remaining data structure, which is so consistent with my scenario.

The installation command is as follows:

$ go get https://github.com/mitchellh/mapstructure

Before using, define the struct structure that map will transform, i.e. blog structure, as follows:

type Blog struct {
    BlogId  string `mapstructure:"blogId"`
    Title   string `mapstructrue:"title"`
    Content string `mapstructure:"content"`
    Uid     string `mapstructure:"uid"`
    State   string `mapstructure:"state"`
}

Because the next step is to use the mapstructure package, the structure tag identity is no longer json, but mapstructure.

The sample code is as follows:

e := Event{}
if err := json.Unmarshal(msg, &e); err != nil {
    panic(err)
}

if e.Table == "blog" {
    var blogs []Blog

    if err := mapstructure.Decode(e.Data, &blogs); err != nil {
        panic(err)
    }

    fmt.Println(blogs)
}

The resolution of event is the same as the previous one. Use e.Table to determine whether the data is from the blog table. If so, use the blog structure to resolve. Next, the resolution is completed through the Decode of mapstructure.

The printing results are as follows:

[{100001 title this is a blog 1000012 1}]

By now, it seems that all the work has been done. No!

Weak type resolution

I don't know if you find a problem, that is, all members in the blog structure are strings, which should be done by can al. All value types are strings. But actually, the uid and state fields in the blog table are both int.

The ideal definition of structure should be as follows.

type Blog struct {
    BlogId  string `mapstructure:"blogId"`
    Title   string `mapstructrue:"title"`
    Content string `mapstructure:"content"`
    Uid     int32  `mapstructure:"uid"`
    State   int32  `mapstructure:"state"`
}

But when the new Blog type is substituted into the previous code, the following error will occur.

panic: 2 error(s) decoding:

* '[0].state' expected type 'int32', got unconvertible type 'string'
* '[0].uid' expected type 'int32', got unconvertible type 'string'

Prompt type resolution failed. In fact, this form of json also appears in some other soft type languages.

How to solve this problem? Two solutions

• Use to convert, such as data of type int, use to convert with strconv.Atoi.
• Use the soft type map provided by mapstructure to transform struct;

Obviously, the first way is too low, and error checking needs to be done more when converting. What about the second way?

See the example code as follows:

var blogs []Blog
if err := mapstructure.WeakDecode(e.Data, &blogs); err != nil {
    panic(err)
}

fmt.Println(blogs)

In fact, you only need to replace the Decode of mapstructure with the WeakDecode. The word is as it means, weak parsing. So easy.

That's it! The next data processing is much simpler. If you want to learn how to use mapstructure, you should knock out the examples in the source code.