Optimizing Android Sync with Jetifier at Bilibili

Earlier this year, we noticed a long waiting time after the Sync phase in Android Studio finished. The following image shows our sync duration in the second half of 2021, which was approximately 10 minutes.

We ultimately pinpointed the issue to the Jetifier operation during the sync process. This operation primarily replaces Support dependencies with AndroidX dependencies. Regardless of whether the project includes support dependencies, as long as enableJetifier = true is enabled, the conversion will occur. In fact, the Android team has recently released a warning about enabling Jetifier in the latest Chipmunk patch 1 version (https://developer.android.com/studio/releases/gradle-plugin?buildsystem=ndk-build#jetifier-build-analyze). Even earlier, tools like gradle-doctor (https://runningcode.github.io/gradle-doctor/) also issued warnings about enabling Jetifier.

Jetifier was enabled which means your builds are slower by 4-20%.

Ultimately, we replaced all Support dependencies within the project with AndroidX dependencies, which reduced our sync time by 2 minutes and 30 seconds.

Incremental build time was reduced by 28%

Starting from the Gradle lifecycle analysis, Gradle is divided into three phases: initialization, configuration, and execution.

• The initialization phase allocates the Gradle instance, executes the init.gradle script, and generates the Setting object to determine which modules will participate in the build process.

• The configuration phase accesses the build.gradle files of all participating modules to obtain the DAG (Directed Acyclic Graph) for this task execution.

• The execution phase executes the tasks on the TaskGraph sequentially based on the DAG determined in the configuration phase.

When analyzing the sync process, we found that there were actually no tasks being executed by breaking at gradle.startParameter.taskNames. The Sync process is more like executing an empty task, but with some strange parameters. A rough simulation of this command line is as follows; note that this command is only a rough simulation since it does not return a model to the IDE for project indexing, and the IDE does not provide external command calls.

 ./gradlew -P android.injected.build.model.only.advanced=true -P android.injected.build.model.only=true -P android.injected.build.model.disable.src.download=true -P org.gradle.kotlin.dsl.provider.cid=465291211918416 -P kotlin.mpp.enableIntransitiveMetadataConfiguration=true -P android.injected.invoked.from.ide=true -P android.injected.build.model.only.versioned=3 -P android.injected.studio.version=2022.2.1 -P idea.gradle.do.not.build.tasks=false -D idea.active=true -D idea.version=2022.2 -D java.awt.headless=true -D idea.sync.active=true -D org.gradle.internal.GradleProjectBuilderOptions=omit_all_tasks --init-script /private/var/folders/0c/zkqf7nrj6pl_d40jp4m9h3xm0000gn/T/sync.studio.tooling11.gradle  --init-script /private/var/folders/0c/zkqf7nrj6pl_d40jp4m9h3xm0000gn/T/ijmapper.gradle --init-script /private/var/folders/0c/zkqf7nrj6pl_d40jp4m9h3xm0000gn/T/ijinit.gradle

Having determined that no tasks were executed during the sync phase, we focused on analyzing the configuration phase process. After repeatedly observing the log output, we found that there would be post-output after the sync ended, where the term JetifyTransform would frequently appear.

> Transform jetified-openDefault-10.10.0.aar (project :weibo) with JetifyTransformWARNING: [XmlResourcesTransformer] No mapping for: android/support/FILE_PROVIDER_PATHSWARNING: [XmlResourcesTransformer] No mapping for: android/support/FILE_PROVIDER_PATHS

It was inferred that the Jetifier operation caused the product transformation during the sync phase. Setting android.enableJetifier=false made the JetifyTransform post-output disappear upon re-syncing. Thus, we identified the root cause as the Jetifier operation extending the overall sync time.

Jetifier is implemented through Gradle’s Artifact transform (https://docs.gradle.org/current/userguide/artifact_transforms.html#implementing_incremental_artifact_transforms). The main function of Artifact transform is to convert dependency artifacts. Unlike AGP’s localized Transform of intermediate products, Gradle’s global artifact transformation ensures that the transformed artifacts can be used post-sync, allowing for simultaneous processing of class, layout, and manifest support dependencies in a single transform process, which cannot be achieved in AGP’s transform.

@CacheableTransformabstract class JetifyTransform : TransformAction<JetifyTransform.Parameters> {}

The input for the Jetifiy Transform process is aar/jar, scanning each file in the aar and performing the corresponding file type transformation before writing it back. The transformations for Class and Xml files can be divided into:

• Class files: Bytecode transformation based on mapping using ASM operations.

• Xml files: Parsed using regex matching to change strings based on mapping.

After the transformation ends, the extracted files need to be recompressed as jeitfier-xxx.aar/jar to be consumed by downstream transforms.

We know that the problem is caused by Jetifier, so we just need to modify the value of android.enableJetifier in memory during the sync phase, theoretically allowing us to hook AGP to read the results. My colleague and I started trying to modify the Gradle configuration at runtime.

First, we tried adding android.enableJetifier=false and android.useAndroidX=false parameters to gradle.startParameter.projectProperties or gradle.startParameter.systemPropertiesArgs. These two configurations are Gradle global configuration parameters. However, we found that the modifications did not take effect, and JetifierTransform logs were still output.

Next, we attempted to change the value of android.enableJetifier=false in settings.ext to true, but the modification still did not take effect.

By reviewing the source code, we found that this value is read by AGP through BasePlugin#ProjectServices. BasePlugin is the common base class for com.android.library and com.android.application plugins. Therefore, as long as we successfully hook the value read here, we can disable Jetifier transform.

// com.android.build.gradle.internal.DependencyConfigurator#configureDependencyChecksfun configureDependencyChecks(): DependencyConfigurator {        val useAndroidX = projectServices.projectOptions.get(BooleanOption.USE_ANDROID_X)        val enableJetifier = projectServices.projectOptions.get(BooleanOption.ENABLE_JETIFIER)}

class TurnOffJetifierPlugin : Plugin<Project> {        override fun apply(project: Project) {               project.plugins.withType(BasePlugin::class.java) {                   val service = it.getProjectService() ?: return@withType                   val service = it.getProjectService() ?: return@withType                   val projectOptions = service.projectOptions                   val projectOptionsReflect = Reflect.on(projectOptions)                   val optionValueReflect = Reflect.onClass(                       "com.android.build.gradle.options.ProjectOptions\$OptionValue",                        projectOptions.javaClass.classLoader                   )                    val defaultProvider = DefaultProvider() { false }                   val optionValueObj = optionValueReflect.create(projectOptions, BooleanOption.ENABLE_JETIFIER).get<Any>()                   Reflect.on(optionValueObj)                             .set("valueForUseAtConfiguration", defaultProvider)                             .set("valueForUseAtExecution", defaultProvider)                   val map = getNewMap(projectOptionsReflect, optionValueObj)                   projectOptionsReflect.set("booleanOptionValues", map)             }        }                private fun BasePlugin<*, *, *>?.getProjectService() =                Reflect.on(this)                       .field("projectServices")                       .get<ProjectServices?>()}

In demo tests, we found this approach feasible, and then we prepared to try to apply this plugin across all modules:

// Apply plugin to root projectallProjects {        apply plugin: TurnOffJetifierPlugin.class}// Include built project Apply pluginsettings.gradle.startParameter.initScripts = [initFile]# initFileallprojects { Project project ->      apply plugin: TurnOffJetifierPlugin.class}

We found that applying the plugin in the root project successfully modified the value, but for the included built repositories, the modification was unsuccessful. In the previous article 《Bilibili Android Compilation Optimization》, it was mentioned that our repositories are organized together through Composite builds, and triggering the initialization of included build sub-repositories is achieved through Init Script. The apply plugin in the Init Script is also executed in afterEvaluate, which means the modification occurs later than the reading timing, so the modification still failed.

We considered that even if we found the right timing to modify the value, such black magic relies on the corresponding implementation of AGP, increasing maintenance costs. Moreover, disabling Jetifier operations during the sync phase does not resolve the transform during the compilation and packaging phase, which is treating the symptoms rather than the root cause. Our real demand is to remove all support packages, ultimately closing all Jetifier operations at all phases. Therefore, we decided to migrate the remaining support dependencies.

Thanks to the MonoRepo architecture, the source code within the project has already replaced Support dependencies with AndroidX dependencies. By removing the support dependencies in the Configuration phase and enabling A8 checks, we located the remaining support dependencies within the repository. For more information on A8 checks, please refer to the subsequent anti-deterioration section. The remaining code includes a few internal libraries floating outside the main repository and some external third-party libraries. For these repositories, we adopted the following three migration measures:

Internal Libraries

For internal repositories, we notified the codebase owners to migrate, performing the operation as AS → Refactor → Migrate to AndroidX, and then manually publishing to the remote Maven, with the main repository updating the version number of the internal library.

Third-party Libraries with Source Access

After cloning the source code of third-party libraries, if they directly support AndroidX and the Change Log indicates no breaking changes, we directly publish an upgraded version of the third-party library. In other cases, we choose to perform the automated migration operation in Android Studio under the same version number of the third-party library and publish it to the company Nexus repository. Finally, we update the main repository version number.

Third-party Libraries without Source Access (Only AAR and POM)

We packaged some commonly used CLI tools as babeltools, which internally integrated Jetifier CLI (https://developer.android.com/studio/command-line/jetifier). By executing the command below, we can convert AAR/JAR containing support to AndroidX AAR/JAR:

bJetifier -i [input.aar] -o [output.aar]

Additionally, if there is a POM file, it needs to be manually modified to change the corresponding support dependencies to AndroidX. The corresponding dependency conversions can refer to CSV (https://developer.android.com/topic/libraries/support-library/downloads/androidx-class-mapping.csv) table.

Some Pitfalls Encountered During Migration

Package Size Changes

After migrating the support packages, we compared the package sizes using Android Studio. The 32-bit version reduced by 6KB, and the 64-bit version reduced by 600KB. Theoretically, the difference at the same commit point should not exceed 1KB.

Using diffuse (https://github.com/JakeWharton/diffuse) tool to compare the differences between APKs, we found that the main impact on package size came from two aspects:

) The tool compared the differences between APKs and found that the main impact on package size came from two aspects:

• R.class

• Size differences in so files

The reduction in R.class was mainly due to an internal library, which after upgrading to AndroidX led to R file transmission. The solution was to enable non-transitive R for that module.

The size differences in so files stemmed from the inconsistency between the NDK version we used for releasing Fresco and the version used during the official release. We covered the so files in the previous AAR with those generated during packaging to reduce the size difference.

After resolving the above issues, we compared the same commit point again, and the size difference narrowed to within 1KB.

Nexus Upload

For some third-party libraries that only have JAR and AAR, like com.tencent.tauth:qqopensdk and com.huawei.android.hms:security-base, they need to be manually uploaded to Nexus. The UI for manual uploads looks like this:

When uploading an AAR with a suffix like -bili4 to Nexus, it will automatically add the suffix as a classifier, which causes com.facebook.fresco:imagepipeline:1.13.0-bili4 to fail to match correctly. Its corresponding dependency is com.facebook.fresco:imagepipeline:1.13.0:bili4.

Additionally, for com.github.bmelnychuk:atv:1.2.9 which includes a POM dependency, it must be uploaded along with the POM file, as the POM describes the module’s dependency information. Missing the POM file will lead to compilation errors due to missing dependencies.

AndroidX Log Checks

After we disabled Jetifier operations, the Jetify Transform output logs disappeared, but there were still a large number of AndroidX check logs in the post-sync phase, as shown below, with only one data point displayed.

WARNING: Your project has set `android.useAndroidX=true`, but jetified-openDefault-10.10.0.aar still contains legacy support libraries, which may cause runtime issues. This behavior will not be allowed in Android Gradle plugin 8.0. Please use only AndroidX dependencies or set `android.enableJetifier=true` in the `gradle.properties` file to migrate your project to AndroidX (see https://developer.android.com/jetpack/androidx/migrate for more info).

However, by unpacking with jadx, we found that the AAR no longer contained support dependencies. This check is not accurate.

Reviewing the source code, we identified that AGP, when enableJetifier is disabled and useAndroidX is enabled, will trigger AndroidX checks for developers, with the check time being proportional to the project size.

    class AndroidXEnabledJetifierDisabled(            private val project: Project,            private val configurationName: String,            private val issueReporter: IssueReporter    ) : Action<ResolvableDependencies> {        private val issueReported =                "${AndroidXEnabledJetifierDisabled::class.java.name}_issue_reported"        override fun execute(resolvableDependencies: ResolvableDependencies) {            // Report only once            if (project.extensions.extraProperties.has(issueReported)) {                return            }        } }

The source code includes a switch to prevent repeated checks. We can disable the AndroidX checks by adding the following parameters in the gradle.properties file:

com.android.build.gradle.internal.dependency.AndroidXDependencyCheck$AndroidXEnabledJetifierDisabled_issue_reported=true

To ensure that future business parties introducing third-party SDKs do not bring in support dependencies again through transitive dependencies, we will exclude all support dependencies in the configuration phase, thus ensuring that the APK generated will no longer contain support dependencies. However, if the APK uses these dependencies at runtime, the app will crash. Our solution is to utilize the A8 checks mentioned earlier in 《Bilibili Android Compilation Optimization》 to check for missing dependencies during the pipeline merge to prevent new support dependencies from being introduced. The code to exclude support dependencies is as follows:

allprojects {    configurations.all { Configuration c ->        if (c.state == Configuration.State.UNRESOLVED) {                exclude group: 'com.android.support'                exclude group: 'android.arch.core'                exclude group: 'android.arch.lifecycle'                exclude group: 'android.arch.persistence.room'                exclude group: 'android.arch.persistence'                exclude group: 'com.squareup.leakcanary', module: "leakcanary-object-watcher-android-support-fragments"        }    }}

Originating from the Hello World, AndroidX (https://android-developers.googleblog.com/2018/05/hello-world-androidx.html) in May 2018, AndroidX began to enter the Android development landscape. With the 28.0 version of the support package (https://developer.android.com/topic/libraries/support-library/revisions?hl=zh-cn#28-0-0), the support package ceased maintenance, and all updates are now iterated solely on AndroidX. In a recent blog post Migrate to AndroidX (https://developer.android.com/jetpack/androidx/migrate) warned:

Caution: As of late 2021, most of the library ecosystem already supports AndroidX natively. This means that your project is most likely already using AndroidX libraries directly and there is no need to follow the steps in this migration guide. Additionally, the enableJetifier(https://developer.android.com/jetpack/androidx/migrate#migrate_an_existing_project_using_android_studio) flag mentioned in this guide can lead to slower build times and should not be used unless it’s necessary.

If your project already has the enableJetifier flag and it’s turned on, you can run Build Analyzer’s Jetifier check to confirm if it’s actually needed. The Build Analyzer check is available starting in Android Studio Chipmunk (https://developer.android.com/studio/preview/features#jetifier-build-analyzer).

If there are still a small number of transitive dependencies from Support in your project, I believe migrating them is necessary. Maintaining an updated toolchain overall is a high return on investment for improving compilation performance.