Find a beautiful machine and bury it. Case Automatic Regression

1, Background and purpose

The total number of buried points on the line is more than 1000, and the main process case is more than 300. In the process of functional iteration, there are often statements about the regression of existing buried points. In the past, it used to consume a lot of time to manually regression, and it is easy to miss the measurement at the two ends.
In order to improve the current situation, the internal research can be made into UI automatic regression, which can improve the efficiency by about 50% after practice, and as long as the scenes are supplemented in time, the missed scenes can be greatly reduced.

2, Assertion method

CASE execution process:

How to obtain the buried point data reported by the client:

1. Android client is realized through adb command:

def save_logcat_to_file(self, file_path, grep_str=``""``, extra_args=``""``, parameter=``"-d"``):
  ``""``"
  ``preservation log To the specified file & Return process
  ``adb logcat -v time > xxx
  ``:param grep_str: Filter character
  ``:param extra_args: Additional parameters
  ``:param file_path: log Storage path
  ``:param parameter: logcat Additional parameters for; default-d(Output cached log, And it won't block;)
  ``:``return``: log process
  ``""``"
  ``logcat_cmd = ``"shell "
  ``to_file_cmd = ``" > "` `+ file_path
  ``if` `extra_args:
    ``logcat_cmd += ``" "` `+ extra_args
  ``if` `grep_str:
    ``if` `config.is_windows:
      ``# logcat_cmd += ``"\"logcat -d -v time | grep \'"` `+ grep_str + ``"\'\""
      ``logcat_cmd += ``"\"logcat "` `+ parameter + ``" -v time | grep \'"` `+ grep_str + ``"\'\""
    ``else``:
      ``# logcat_cmd += ``"logcat -d -v time | grep \'"` `+ grep_str + ``"\'"
      ``logcat_cmd += ``"logcat "` `+ parameter + ``" -v time | grep \'"` `+ grep_str + ``"\'"
  ``else``:
    ``logcat_cmd = ``"logcat -v time"
  ``# Print simple data of log detailed time
  ``return` `self.cmd(logcat_cmd + to_file_cmd, return_proc=True)

2. The IOS client is realized through the idevicesyslog command:

def save_device_log(device_id, device_log_path):
  ``""``"
  ``Start saving device log To the specified file & Return to save log process
  ``:param device_id: define equipment id
  ``:param device_log_path: equipment log file
  ``:``return``: preservation log process / None
  ``""``"
  ``try``:
    ``Logger().setlog(device_id + ``" Ready to collect Case journal"``, LEVEL_INFO)
    ``if` `PLATFORM_IOS in config.operating_system:
      ``# proc = Shell.proc(``"idevicesyslog -u "` `+ device_id + ``" > "` `+ device_log_path)
      ``proc = Shell.proc(``"tidevice -u "` `+ device_id + ``" syslog> "` `+ device_log_path)
    ``else``:
      ``device = ADB(serialno=device_id)
      ``device.clear_logcat()
      ``proc = device.save_logcat_to_file(device_log_path)
    ``Logger().setlog(device_id + ``"Start collection Case journal"``, LEVEL_INFO)
    ``return` `proc
  ``except:
    ``print(``"storage device log abnormal"``)
    ``return` `None

3. The H5 page is realized by calling the client webview capability:

Since H5 is a preliminary attempt, in the process of implementation, in order to ensure the following effects, the research envisages three sets of schemes

• Scheme 1 "discard": assert the H5 log by obtaining the H5 log

  • Reasons for abandonment:
    • The H5 buried point of the current online environment is directly reported to the online environment to avoid the impact of test data on statistics; High implementation cost of H5 log capture;
  • The implementation process is as follows:

    

• Scheme 2 "discard": write a new API for asynchronous call when H5 reports the buried point

  • Reasons for abandonment:
    • The cost of input resources is high, and high-frequency calls will be made online; Unable to guarantee delay and stability, it is easy to affect the business
  • The process is as follows:

    

• Scheme 3 "final":

  • The H5 implementation scheme is in the method of reporting the buried point, while asynchronously calling the ability of the client webview, so as to share the methods implemented by the client and complete the assertion of the buried point.
  • The implementation process is as follows:

    

3, Comparison of test methods before and after implementation

scene	Traditional artificial regression verification	UI automatic test regression	Advantages and disadvantages after implementation
Client operation	Manually hang up the agent and open the egg (client debugging mode) high-frequency repetitive operation APP	Automatic operation	Advantages: it can greatly reduce the disadvantages of manual high-frequency operation: the operation of the client needs to enter the case in advance
Buried point reporting verification	After operating the APP, you need to wait about 3 minutes (there is a delay in platform data query), log in to the relevant SQL query equipment reporting records of Shence platform, and analyze whether the reporting and timing of each field are consistent with expectations	Automatically assert that each buried point is a case according to the log reported by the client, and conduct regression separately without affecting each other's output of relevant SQL. The buried points can be reviewed randomly and quickly manually	Advantages: 1. Greatly reduce the missed measurement rate and improve the quality and efficiency by about 50%. 2. Make assertions in real time after operation without waiting. Disadvantages: manual random review is required in the early stage
Regression frequency and range	Buried point of return to mainstream process	As long as you enter, you can overwrite the regression	Advantages: full coverage, more regression times, more obvious quality and efficiency prompt. Disadvantages: the released manpower is invested in the case entry and maintenance of buried points in part of the time

4, Summary

1. Sort out business processes and clarify the scope of automation

2. Automatic frame modeling

3. What are the pain points in the buried point test? Find solutions after comparison

4. Automatic data verification and comparison

5. Improve the utilization rate of automation, trigger the automation of project release automation, and carry out detection regularly every day