Author: Liu Tianyu (Qian Feng)
Engineering corruption is a very difficult problem in the process of app iteration, involving extensive and detailed details, which has a relatively "hidden" and indirect impact on R & D Efficiency & experience, Engineering & product quality, stability, package size and performance. Generally, it will not cause unbearable obstacles, but it often jumps out and leads to "labor pains". It is a bit like decayed teeth or wisdom teeth. It is impossible not to pull them out to a certain extent. However, the difference is that the corruption of the project is difficult to be cured by one-time "removal". After any "removal", effective sustainable treatment schemes are needed to form a normalized anti-corrosion system.
From the perspective of Engineering corruption disassembly, it is the corruption of the engineering structure itself and various "elements" (manifest, code, resource, so, configuration) in the code engineering that makes up the app. In recent years, Youku architecture team has continued to think, practice and manage, and precipitated some technologies, tools and schemes. Now it is classified and summarized one by one, supplemented by the explanation of relevant field knowledge, and sorted into a series of technical articles on shooting at engineering corruption, which will be shared with you. I hope more students will join in this protracted war against project corruption.
First article in a series< Crack down on project corruption | proguard control >. This article, the second in a series of articles, will focus on the sub domain of manifest. Direct fire on the corruption of the project!
background
manifest refers to androidmanifest in apk XML file, as the overall information list of apk, contains a lot of important information, which has a vital impact on app construction processing, runtime behavior, App Store filtering, etc.
List content & impact
When androidmanifest When the content in XML file changes unexpectedly, it will bring unexpected consequences. For example: minSdkVersion becomes smaller, and after going online, low version os users upgrade to the latest apk, resulting in serious user experience problems; targetSdkVersion increases, the os's specific handling of app runtime changes, and the unadapted code crash / function is abnormal; The new permission was introduced, the privacy agreement was not declared, and was found by regulators. The above problems are only caused by a "small" change in the configuration value in the list file. The corruption of the list leads to such unexpected changes, which is more and more likely to occur. Manifest governance revolves around Android manifest The content arrangement and prevention and control of XML are carried out step by step.
Basic knowledge
This chapter first briefly introduces some basic knowledge to facilitate you to have a clear understanding of the "framework" of manifest. First, take a look at Android manifest The generation (merging) process of XML files.
Consolidation process
app projects, aar type subproject projects, and externally dependent aar modules all contain androidmanifest XML file. During apk construction, these androidmanifest After the XML file is merged (+ some additional processing), a unique AndroidManifest.xml file is generated XML file, which is compiled and finally placed in the apk root directory.
Merging is from low priority to high priority. Horizontal is the priority of different sources; The priority between modules is from high to low, which is the declaration order in the app project; The priority among build variant, build type and product flavor decreases gradually; If product flavor contains multiple dimensions, the priority from high to low is the order specified in flavor dimensions.
AndroidManifest.xml priority & Merge order
In the process of merging, the attributes of the same xml element (generally android:name attribute value, or element label) will have merging conflicts. The basic principle is: if both high priority and low priority attribute values exist and are inconsistent, they will be regarded as conflicts. Due to the diversity of elements / attributes in the manifest file, the actual rules are much more complex. For details, please refer to the official google Documents. Merge conflict resolution, in addition to modifying the corresponding androidmanifest In addition to the xml file, you can also use Android manifest. xml in the app project xml, add the "merge rule tag" implementation. In addition, even if there is no conflict, when it is necessary to control the contents of the list, it can also be implemented in the same way, which will be introduced next.
Merger control
The "merge rule tag" mentioned above realizes the control of merge results by specifying merge rules for xml nodes and attributes with different granularity. First, add the tools namespace at the root node of the manifest:
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.myapplication"
xmlns:tools="http://schemas.android.com/tools">
Then, add the corresponding tools: attribute in the node according to the specific situation.
Merge rule tag description
Please refer to the official documents for the specific rules of merger control, which will not be detailed here.
manifest placeholder
In addition to the above merge control, you can also control the attribute values of the nodes in the list through the manifest placeholder.
# build. Variables and values are defined in the gradle file
android {
defaultConfig {
manifestPlaceholders = [customKey:"customValue", ...]
}
...
}
# AndroidManifest. Use placeholders in XML files
<intent-filter ... >
<data android:scheme="https" android:host="${customKey}" ... />
...
</intent-filter>
<meta-data android:name="sampleMeta" android:value="${customKey}"/>
In addition, there is a default placeholder ${applicationId}, which is bound to the applicationId configuration value in android DSL. During the build process, all placeholders are replaced with corresponding values.
Merge decision log
Finally, Android manifest Each node and attribute of XML, which list file it comes from and which strategy it is generated by, are recorded in the consolidated decision log to provide important auxiliary information for problem analysis and troubleshooting. The file is located in the app project build / outputs / logs / manifest merge [- productflavor] - [BuildType > - report Txt, the example is as follows:
activity#com.example.myapplication.MainActivity
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:18:9-24:20
android:name
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:18:19-47
intent-filter#action:name:android.intent.action.MAIN+category:name:android.intent.category.LAUNCHER
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:19:13-23:29
action#android.intent.action.MAIN
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:20:17-69
android:name
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:20:25-66
category#android.intent.category.LAUNCHER
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:22:17-77
android:name
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:22:27-74
...
uses-permission#android.permission.READ_EXTERNAL_STORAGE
IMPLIED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:2:1-28:12 reason: com.example.libraryaar1 requested WRITE_EXTERNAL_STORAGE
MERGED from [com.youku.arch:Hound:2.8.15] /Users/flyeek/.gradle/caches/transforms-2/files-2.1/d42ba59a47f7160082879236533c4582/AndroidManifest.xml:11:5-80
MERGED from [com.youku.arch:Hound:2.8.15] /Users/flyeek/.gradle/caches/transforms-2/files-2.1/d42ba59a47f7160082879236533c4582/AndroidManifest.xml:11:5-80
uses-permission#android.permission.WRITE_CALL_LOG
IMPLIED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:2:1-28:12 reason: com.example.libraryaar1 has targetSdkVersion < 16 and requested WRITE_CONTACTS
After reading the basic knowledge of this article, you should be able to understand the content, so I won't repeat it.
Several interesting configurations
So far, we have a "framework" overall understanding of the manifest file. Finally, let's look at some interesting configurations.
package vs applicationId
These two concepts are easily confused. From the perspective of the final apk file, the only apk is androidmanifest The package attribute value of the manifest node in XML, which is often referred to as "appId" and "app package name". Directly above:
package vs applicationId
The package value in the app project only affects the construction process. The applicationId value in android DSL will finally replace androidmanifest The package attribute value in XML becomes the unique identifier of the final apk.
Implicit system permissions
Under some conditions, the merging process of the list file will add additional system permission statements automatically. If it is not processed and not stated in the app privacy agreement, it will lead to compliance risk. The automatic addition of permission statements is shown in the following table (directly extracted from official documents):
List of permission statements added by the merge process
For example, the targetSdkVersion of app is 28. In the form of external dependency, a module is introduced, which contains androidmanifest targetSdkVersion in XML (low priority list) is 14 and read is declared_ With the permissions of contacts, the final apk manifest file will contain READ_CALL_LOG permission statement.
Component export control
Component export means that the android:exported attribute is true (explicit / implicit), and the component can be called by other apps. If the android:exported value is explicitly set in the list, this shall prevail; If not set, the implicit rule is: if intent filter is set, the exported value is true, otherwise it is false. Many apps will use the intra app routing mechanism of components (especially activity), so some intent filters will be set, which will lead to unexpected export of components and bring security risks. This requires special attention and will be discussed later.
It is worth noting that when targetSdkVersion is set to 31 (Android12) or above, if the component is set to intent filter, the android:exported value must be explicitly set at the same time. If the exported value is not explicitly set, the build of IDE will fail for high version Android Studio, and it can succeed for low version Android Studio, but it will fail when installed on Android 12 and above devices.
Governance practice
The basic knowledge and engineering application of manifest have been explained earlier. I believe you have formed a preliminary overall understanding. With the increase of engineering modules / codes, the controllability of the list file gradually decreases: whether it is the unexpected change of key configuration values, the introduction of unexpected permissions, or even the accumulation of useless / redundant / risk nodes and attributes. In the struggle against the "corruption" of manifest, Youku starts from the actual needs of the upper level (such as privacy compliance, security vulnerabilities and online problems), establishes effective detection capabilities through relevant tools, and forms a daily R & D bayonet mechanism based on this. On the premise of ensuring zero new problems, gradually digest the existing stock problems.
Global configuration
Some global configurations in manifest have an important impact on apk installation and runtime behavior. The most typical ones are minSdkVersion and targetSdkVersion. Once unexpected changes are brought online, the consequences will be unimaginable.
The global configuration detection tool provides global configuration detection capability based on white list, including the following situations:
- It is configured in the white list and does not exist in the list;
- The configuration in the white list exists in the list, but the configuration values are inconsistent.
At the same time, options are provided to terminate the construction process when the global configuration is inconsistent with the white list. The example detection results are as follows:
[absent] [uses-feature] android.hardware.camera # The uses feature in the whitelist does not exist in the list [conflict] [uses-sdk] # The attribute value of the uses SDK node in the whitelist is inconsistent with that in the list |-- com.youku.arch:testlib:0.1-SNAPSHOT # Module containing uses SDK node |-- project:library-aar-1:1.0 # Module containing uses SDK node |-- com.youku.arch:testlib2:0.1-SNAPSHOT # Module containing uses SDK node |-- [attr] targetSdkVersion # The targetSdkVersion attribute value is inconsistent | |-- [whitelist] 29 | |-- [current] 28 |-- [attr] minSdkVersion # The value of minSdkVersion attribute is inconsistent | |-- [whitelist] 14 | |-- [current] 21
Youku global configuration white list and new prevention and control are as follows:
Global configuration and Governance
Through this detection capability and bayonet mechanism, the protection of key global configurations is realized, so as to effectively avoid unexpected changes.
jurisdiction
According to the authority statement, under the current situation of privacy compliance supervision, it needs to be strictly controlled. The "strictness" here is reflected in neither more nor less. It must be consistent with the app privacy agreement. In the previous basic knowledge section, we know that androidmanifest XML is merged. At the same time, system permissions are brought in implicitly, which makes it more difficult to "strictly" control permissions.
In this regard, two detection capabilities are developed: the module includes permission list and permission detection.
The module contains a permission list, which lists the uses permission and permission definitions contained in each module, so as to locate the permission source. Example results:
com.youku.android:YPx:1.20.10.19 |-- [uses-permission] android.permission.ACCESS_NETWORK_STATE |-- [uses-permission] android.permission.BLUETOOTH |-- [uses-permission] android.permission.VIBRATE com.taobao.android:ls:4.10.6.6 |-- [uses-permission] android.permission.READ_PHONE_STATE |-- [uses-permission] android.permission.ACCESS_WIFI_STATE
Permission detection, providing two-way detection capability based on white list:
- Permission in the white list does not exist in the list;
- Permission in the list, not in the white list.
[excess] [uses-permission] android.permission.CALL_PHONE # Call in list_ Phone permission statement, not in the white list |-- project:app:1.0 # Permission statement from app project [absent] [uses-permission] android.permission.ACCESS_NETWORK_STATE # Access in white list_ NETWORK_ State, does not exist in the list |-- com.youku.arch:testlib:1.0 # com.youku.arch:testlib module, which contains this permission statement |-- com.youku.arch:testlib2:1.0 # com.youku.arch:testlib2 module, which contains this permission statement
Further, provide the option to terminate the construction process when the test result fails. Through this detection capability and bayonet mechanism, the continuous consistency between the permission statement and the app privacy protocol is guaranteed. The governance & prevention and control of Youku are as follows:
Authority Governance
Four components
The four components need to be declared in the manifest file before they can be recognized by the system after apk installation and operation, so as to function normally. At the same time, some key behaviors of the four components also need to be configured in the list. In the process of Youku's practice, two types of problems are mainly found: the lack of corresponding classes of components and the export of unnecessary components.
The missing corresponding class of component refers to the four components declared in the list. The value of android:name attribute corresponds to java class, which does not exist in apk. The negative effects of missing component classes are as follows:
- A useless keep rule of proguard will be generated, which will increase the construction time (although a keep is small, it is also considerable);
- Once the component is called (started) at runtime, java exceptions (crash / function unavailable) or security vulnerabilities will be generated. Even for useless components, it is also necessary to take into account that there are some black tissues that will automatically scan the components and start them (sharp spikes will appear in the crash rate curve).
The export of unnecessary components (see the definition above) will increase the risk of security vulnerabilities at runtime. Youku has received relevant security vulnerabilities for many times. The processing principle of exported components is as follows:
- It is unnecessary to export and is self-developed code. Close export;
- It is unnecessary to export, but it is a second-party or third-party code. In the manifest file of the app project, modify the android:exported attribute value to false through "merge rule tag";
- It needs to be exported and is self-developed code for debugging during development. Close the export and converge to the unified R & D debugging toolbox;
- It needs to be exported and is a self-developed code, which is used for online actual business (external calling end, etc.). Close the export and converge to the unified routing center;
- It needs to be exported, but it is a second-party or third-party code, which is used for online actual business (external calling end, etc.). Add whitelist.
In this regard, three detection capabilities have been developed:
- List of component belonging modules, listing all four components and modules containing this component declaration:
# Components that do not exist after the merge of manifest will be preceded by [delete] # Components contained by more than two modules will be preceded by [duplicate] [duplicate] [activity] com.example.myapplication.MainActivity |-- project:app:1.0 |-- project:library-aar-1:1.0 [deleted] [service] com.example.myapplication.FirstService |-- project:app:1.0 [receiver] com.example.myapplication.FirstReceiver |-- project:library-aar-1:1.0 [provider] com.example.myapplication.FirstProvider |-- com.youku.arch:testlib:1.0
- Missing component reference detection to identify the name of the missing referenced component and which modules declare this component. At the same time, options and white lists are provided. When the test result fails, the construction process is terminated. The sample test results are as follows:
[activity] org.cocos2dx.javascript.AActivity |-- com.youku.android:interactive-engine:0.2.9 [activity] com.ali.lv.HLActivity |-- com.ali.phone.wt:n-build:10.2.3.592 [activity] com.youku.pc.debug.DActivity |-- com.youku.android:YKPChannel:2.14.1.28 [service] com.youku.feed.utils.FAService |-- com.youku.android:FBase:1.5.20.8
- Export component detection to identify the export component and which modules declare this component. At the same time, options and white lists are provided. When the test result fails, the construction process is terminated. For the behavior change of Target31 safer export component, the configuration item "prohibit implicit export" is specially provided. The white list will be ignored and an identifiable mark will be added to the analysis results. The sample test results are as follows:
# For components in the white list, the [ignored] mark will be added before the name; If the "prohibit implicit export" configuration item is enabled, the [implicit] identification will be added before the name of the implicit export component [activity] com.youku.app.NPageActivity |-- com.youku.android:YoukuHPage:1.9.43.8 [ignored][activity] com.ali.MIPreviewActivity |-- com.ali:m-image-selector:10.1.6.190 [implicit] [activity] com.youku.fbiz.RPageActivity |-- com.youku.android:fbizSDK:1.0.2.48
In the practice of Youku governance, considering the impact on business R & D students, the stock problems are added to the white list, and then choose the opportunity to launch a unified clean-up action. With the iteration of the version, in addition to the effective interception of new problems, there are also some "natural repairs" to the stock problems. The overall situation is as follows:
Governance of four components
In addition, Youku's current targetSdkVersion is 30, and the adaptation of target31 will be carried out next year. There are 161 implicit export components in stock, accounting for about 50% of all export components. At that time, all these need to be solved. Under the current tool and bayonet system, it is believed that the rectification of this problem will become easy and controllable.
Governance panorama
So far, a more comprehensive and effective anti-corrosion capacity-building and treatment have been carried out for the manifest list. Finally, a panorama is given:
Panorama of manifest governance
What else can I do
The content of manifest is limited. Therefore, the above governance should cover most of the problems, but there are still some low probability edge case s that can be identified & solved in advance through the same idea. For ex amp le, the scheme definitions of multiple activities are repeated, resulting in a selection pop-up when an activity is started implicitly.
It is still difficult to fight against the corruption of the project. There is a heavy task and a long way to go. We should encourage you.
[reference documents]
- [google]App Manifest Overview:https://developer.android.com/guide/topics/manifest/manifest-intro
- [google]Manage manifest files:https://developer.android.com/studio/build/manage-manifests
Focus on Alibaba mobile technology WeChat official account, 3 mobile technology practices dry cargo per week to give you thought!