A pit of Dubbo serialized objects

-- Erwu Zz

Today, I encountered a pit in Dubbo serialization object when dealing with project problems. Record:

When colleagues call the Dubbo rpc interface, they find that the returned value is inconsistent with the expected return value. It is expected to return a collection of objects, but the returned collection is hashMap. After checking their code, they find that their consumer's path definition of the returned object is inconsistent with that of our service provider, The returned data does not match the expectation. This solution is very simple. Just keep the path consistent. (it's best to define objects as public, which can avoid these problems)

But why does this happen? Why does this hashMap appear?

Step by step, it is found that there are problems in the serialization and deserialization of classes at the consumer and server. First, the class for deserialization in Java is JavaSerializer. The construction method of this class calls such a method: getFieldMap. All methods of this class and parent class are placed in fieldMap because it is hashMap. In order to ensure that the method name is not overwritten, One of the operations is:

fieldMap.get(field.getName())!=null;

When the server performs deserialization, it will judge whether the class name under the path exists. If it exists, it will make a circular assignment to the returned collection object. If it does not exist, it will directly return the hashMap

Here, the cause of the problem is found, but think about it:

• Since it is a serialization problem, what problems will be caused if the objects on the consumer side and the server side are in the same path and the UIDs generated by serialization are different?
• What is the problem if the UIDs serialized are the same under different paths?
• If the object has a parent object under the same path and some of its properties are the same, what will be the problem?
• What is the problem if the object path, uid and attribute are the same but serialization is not implemented?

Conclusion:

• It can be seen from the above method called during deserialization that the matching assignment is made according to the method name and path during deserialization. As long as the path and name are the same, no matter what the uid is, the object collection can be returned successfully,

  • Continue to draw deep, why? According to java, it should not be implemented

  • It's been bothering me for a long time. Looking up the data, I see a sentence, "Hessian 2 serialization: Hessian is an efficient binary serialization method across languages. However, this is not the native Hessian 2 serialization, but the hessian lite modified by Ali, which is the serialization method enabled by dubbo RPC by default.". Later, I wondered if it was because of the serialization method, and whether Hessian didn't care about the change of serialVersionUID during serialization. To prove the doubts in the book. Hessian and java serialization methods are used for verification respectively.

  • There are two projects, a sender and a receiver, which are started by SpringBoot. The entity object Person is loaded in both projects. Used to test when the serialVersionUID of the object is the same and different.

  • 1. The first project is the receiving end of the transmission object, which is deserialized here. serializabletest-server. 3 main classes
    1.1.Person.java transfer object.

    @ToString
    @Data
    public class Person implements Serializable {
    private static final long serialVersionUID = -1923645274767028479L;
    private String[] address;
    private String name;
    private int phone;
    }
    

    1.2. HessianController.java hessian deserializes the accept class. Accept the object information submitted from http through a Controller

    @Slf4j
    @RestController
    @RequestMapping("/hessian")
    public class HessianController {
        
        @RequestMapping(value = "/hello")
        public String helloWorld(HttpServletRequest request, HttpServletResponse response) throws IOException {
            log.info("hessian Deserialization start------------------------");
            ServletInputStream sis = request.getInputStream();
            Hessian2Input h2i = new Hessian2Input(sis);
            h2i.startMessage();
            Person person = (Person) h2i.readObject();
            h2i.completeMessage();
            h2i.close();
            sis.close();
            log.info("hessian Deserialization result"+person.toString());
            return person.toString();
            }
        
    }
    

    1.3 JavaController. The Java deserialization class accepts the object information submitted by http through a Controller

    @Slf4j
    @RestController
    @RequestMapping("/java")
    public class JavaController {
    
    @RequestMapping(value = "/seri")
    public String helloWorld(HttpServletRequest request, HttpServletResponse response) throws Exception {
    
        log.info("java Deserialization start------------------------");
        ServletInputStream sis = request.getInputStream();
        ObjectInputStream is = new ObjectInputStream(sis);
        Person person = (Person) is.readObject();
        log.info("java"+person.toString());
        return person.toString();
        
    	}
    }
    

    2. The second project is the transmission object sender, where the object is serialized and the request is sent by post. serializabletest-client. It is also three main classes
    2.1 Person.java

    @ToString
    @Data
    public class Person implements Serializable {
    //By changing the serialversion uid, test the same and different situations as in the serializable test server
    private static final long serialVersionUID = 6457272772L;
    
    private String[] address;
    
    private String name;
    
    private int phone;
    }
    

    2.2 HessianTest.java Hessian serializes the object and sends it

    public class HessianTest {
    public static String urlName = "http://localhost:8080/hessian/hello";
    
    public static void main(String[] args) throws Throwable {
    
        // serialize
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        Hessian2Output h2o = new Hessian2Output(os);
        h2o.startMessage();
        h2o.writeObject(getPerson());
        h2o.writeString("I am client.");
        h2o.completeMessage();
        h2o.close();
    
        byte[] buffer = os.toByteArray();
        os.close();
        ByteArrayEntity byteArrayEntity = new ByteArrayEntity(buffer,
                ContentType.create("x-application/hessian", "UTF-8"));
    
        CloseableHttpClient client = HttpClients.createDefault();
        HttpPost post = new HttpPost(urlName);
        post.setEntity(byteArrayEntity);
        CloseableHttpResponse response = client.execute(post);
    
        System.out.println("response status:\n"
                + response.getStatusLine().getStatusCode());
        HttpEntity body = response.getEntity();
        System.out.println("body:"+body);
    }
    
    public static Person getPerson() {
        Person person = new Person();
        person.setAddress(new String[] { "Beijing", "TaiWan", "GuangZhou" });
        person.setName("Jack");
        person.setPhone(188888888);
        return person;
    }
    

    2.3 JavaTest. Java serialize and send

    public class JavaTest {
    public static String urlName = "http://localhost:8080/java/seri";
    public static void main(String[] args) throws Throwable {   
       ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream os = new ObjectOutputStream(bos);
        os.writeObject(getPerson());
        byte[] buffer = bos.toByteArray();
        os.close();
        ByteArrayEntity byteArrayEntity = new ByteArrayEntity(buffer,
                ContentType.create("x-java-serialized-object", "UTF-8"));
    
        CloseableHttpClient client = HttpClients.createDefault();
        HttpPost post = new HttpPost(urlName);
        post.setEntity(byteArrayEntity);
        CloseableHttpResponse response = client.execute(post);
    
        System.out.println("response status:\n"
                + response.getStatusLine().getStatusCode());
        HttpEntity body = response.getEntity();
        System.out.println("body:"+body);
    }
    
    public static Person getPerson() {
        Person person = new Person();
        person.setAddress(new String[] { "Beijing", "TaiWan", "GuangZhou" });
        person.setName("Jack");
        person.setPhone(188888888);
        return person;
    	}
    }
    

    According to the test results, I come to the conclusion that Hessian can successfully "deserialize" when the serialVersionUID of the transmission object is inconsistent. However, java mode cannot be "deserialized".

• By analogy with the above answer, we can see that different paths cannot obtain the class name, so we can't judge. What is returned is a hashMap

• For this result, we need to introduce the method readObjectInstance of Hessian2Input class during deserialization. It will take all the methods of this class and parent class and put them into an array fieldNames. After that, the readObject of JavaSerializer, the deserialization method, takes values circularly according to the fieldNames array and takes them out one by one in the stream, The set method that has been assigned to this class first has a value. When it reaches the parent class, the value obtained is empty, and the value of this class will be overwritten. The return value will have a null value and the parameters will be lost

• An error will be reported directly because serialization is not implemented.