In this article, we introduce how to use Protocol Buffers data format in Golang applications. Including the definition of Protocol Buffers and the advantages compared with traditional xml and json, and practicing through several examples.

Protocol Buffers data format

Protocol Buffers is a data format launched by Google. We know that json/xml can store data in different languages and realize serialization and deserialization. Compared with json/xml format, its storage size is much smaller.

Assuming that there is a person object, we use three data formats for comparative description:

<person>
  <name>Elliot</name>
  <age>24</age>
</person>

Use json to describe the smaller points:

{
  "name": "Elliot",
  "age": 24
}

If the same data is represented by protocol buffer:

[10 6 69 108 108 105 111 116 16 24]

After careful observation, we will find that Elliot is represented from the second position of the array, E=69, l=108, and then the age is 24

The encoding format is much more complex than we have seen, and I am also trying to learn more about it. If necessary, I suggest checking Google Docs. Now, although our example represents the same size as json. Next, let's consider more complex scenarios with more size differences.

Simple object serialization example

First, we need to install the protocol tool. The download address is: https://github.com/protocolbuffers/protobuf/releases ;

The corresponding file of windows 64 is: protocol-3.19.4-win64 zip； After downloading, unzip the protocol Exe to the bin target under the go installation path to ensure that the file can be executed anywhere.

go get google.golang.org/protobuf
go get -u github.com/golang/protobuf/protoc-gen-go

After installation, test whether the protoc ol command is available on the command line.

Next, we define the protobuf data format, and modify the above person object. First, specify the syntax to use. Here, proto3 format is used. Then specify the file location: option go_package ="../pb"; Indicates person The proto file is located in the Pb directory, and then specify the package. The complete contents are as follows:

syntax="proto3";
option go_package ="../pb";

package pb;

message Person {
      string name = 1;
      int32 age = 2;
}

Finally, execute the command in the pb directory to generate the corresponding go file:

protoc --go_out=. person.proto

Normal generation will not cause errors, and generate Person. In the Pb directory pb. Go file. Next, we define the Person object and encode it in protobuf format, and then use FMT Println (data) output protobuf format object:

package main

import (
	"fmt"
	"github.com/dataz.cn/wordcount/pb"
	"github.com/golang/protobuf/proto"
	"log"

)

func main() {
	elliot := &pb.Person{
		Name: "Elliot",
		Age:  24,
	}

	data, err := proto.Marshal(elliot)
	if err != nil {
		log.Fatal("marshaling error: ", err)
	}

	fmt.Println(data)

	newElliot := &pb.Person{}
	err = proto.Unmarshal(data, newElliot)
	if err != nil {
		log.Fatal("unmarshaling error: ", err)
	}

	fmt.Println(newElliot.GetAge())
	fmt.Println(newElliot.GetName())
}

Then, the encoded object is deserialized to generate a new object, and its properties are printed for verification.

Run the above code and the output is as follows:

[10 6 69 108 108 105 111 116 16 24]
24
Elliot

Nested type serialization example

The above example is slightly simpler, and the practical application will be more complex. Let's look at the serialization of nested types. The SocialFollowers object is nested in the Employee object:

syntax="proto3";
option go_package ="../pb";

package pb;

message SocialFollowers {
  int32 weixin = 1;
  int32 weibo  = 2;
}

message Employee {
  string name = 1;
  int32 age = 2;
  SocialFollowers socialFollowers = 3;
}

Generate the corresponding go file with the command in the above example. The following example shows the serialization and deserialization process of nested objects:

package main

import (
	"fmt"
	"github.com/dataz.cn/wordcount/pb"
	"github.com/golang/protobuf/proto"
	"log"
)

func main() {

	elliot := pb.Employee{
		Name: "Elliot",
		Age:  24,
		SocialFollowers: &pb.SocialFollowers{
			Weixin: 2500,
			Weibo: 1400,
		},
	}

	data, err := proto.Marshal(&elliot)
	if err != nil {
		log.Fatal("Serialization error: ", err)
	}

	newElliot := &pb.Employee{}
	err = proto.Unmarshal(data, newElliot)
	if err != nil {
		log.Fatal("Inverse sequence error: ", err)
	}

	fmt.Println(newElliot.GetName())
	fmt.Println(newElliot.GetAge())
	fmt.Println(newElliot.SocialFollowers.GetWeixin())
	fmt.Println(newElliot.SocialFollowers.GetWeibo())
}

Run again and the output is:

Elliot
24
2500
1400

summary

In this paper, we introduce Google's protocol buffer data format, and show the process of serializing and deserializing objects using this format through examples.