proto3 serialization is very useful. When APP was originally developed to store data to the log, the log document can reach 300M in half an hour. When there is a large amount of data, it has a significant impact on performance. Therefore, using proto serialization to store data has improved performance after testing, and the log size is one-third of the original size, so the advantage is still obvious.

However, when proto3 serializes multiple messages to a file, according to the official documentation, it is impossible to distinguish the boundaries of a complete object serialized data when deserializing, that is, no delimiter, so you need to set your own delimiter, which can be resolved by rules when deserializing. The following is an example based on the environment in the previous article

1. Multimedia Continuous Serialization Rules

The length of bytes after serialization of each object is 4 bytes ahead of the serialized content. That is, if a serialized data length is 100 bytes, then the first 4 bytes are added to the data. The content of these 4 bytes is 100, then the total length is 104 bytes. Continue serialization to the file according to this rule

2. Defining proto data structures

syntax = "proto3";
package tutorial;

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
  string phone = 4;
}

3. Serialization testing

/**
     * serialize
     *
     * @param path
     */
    private void serializeProto(String path) {
        System.out.println("serialize file " + path);
        try {
            OutputStream outputStream = new FileOutputStream(new File(path));
            byte[] data;
            for (int i = 0; i < 10; i++) {
                Message.Person person = Message.Person.newBuilder().setId(i)
                        .setEmail(String.valueOf(i)).setName(String.valueOf(i)).setPhone(String.valueOf(i)).build();
                byte[] dataByte = person.toByteArray();
                byte[] lenByte = ProtoHelper.intToByteArray(dataByte.length);

                data = new byte[dataByte.length + lenByte.length];
                System.arraycopy(lenByte, 0, data, 0, lenByte.length);
                System.arraycopy(dataByte, 0, data, lenByte.length, dataByte.length);

                outputStream.write(data);
                System.out.println("serialize len :" + dataByte.length + ",id:" + person.getId() + ",name:" + person.getName() + ",email:" + person.getEmail() + ",phone:" + person.getPhone());
            }
            outputStream.flush();
            outputStream.close();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

IV. Deserialization

Because of the small amount of data mentioned above, no buffer will be set if you read the file at once.

/**
     * Deserialize
     * @param path
     */
    public void deSerializeProto(String path) {
        System.out.println("deSerialize file " + path);
        try {
            InputStream inputStream = new FileInputStream(new File(path));

            byte[] data = new byte[4096];
            int len = 0;
            while ((len = inputStream.read(data)) > 0) {
                int index = 0;
                while (index < len) {
                    byte[] lenByte = new byte[4];
                    System.arraycopy(data, index, lenByte, 0, 4);
                    int itemLen = ProtoHelper.byteArrayToInt(lenByte);
                    byte[] dataByte = new byte[itemLen];
                    System.arraycopy(data, index + lenByte.length, dataByte, 0, itemLen);

                    Message.Person person = Message.Person.parseFrom(dataByte);
                    System.out.println("deSerialize len :" + (dataByte.length + 4) + ",id:" + person.getId() + ",name:" + person.getName() + ",email:" + person.getEmail() + ",phone:" + person.getPhone());
                    index = index + lenByte.length + dataByte.length;
                }

            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

Execution results:

Because proto is cross-platform, after java-side serialization, C-side colleagues can also decode files, so it has some log security.