Build cache

much faster builds

Lóránt Pintér (@lptr), Gradle

:navigation:™

Agenda

  • What is the build cache?

  • Support available today

  • Next steps

Why go faster?

  • faster build → quicker feedback → better productivity

  • faster build → fewer CI machines

Incremental build

Designed for:

  • local development

  • one change at a time

Moving forward in history

How it works

  1. check task’s inputs

  2. check task’s outputs

  3. profit!

:compileJava UP-TO-DATE

Metadata

class CustomTask extends DefaultTask {
  @InputDirectory
  File sourceDirectory

  @OutputFile
  File outputFile

  // ...
}

Limitations

  • works locally

  • optimized for incremental changes

  • "best-effort"

Build cache

is-a semi-permanent storage

enabled by --build-cache

Task output caching

is-a feature using the build cache where:

  • the address is the task’s inputs,

  • the content is the task’s outputs.

(with a few tricks…​)

Local build cache

Useful when:

  • working on branches

  • git bisect

Moving around in history

Shared build caches

Implementations:

  • HTTP build cache

  • Gradle Enterprise build cache

GE build cache

Shared cache on CI

Reuse outputs:

  • between changesets

  • between CI agents

  • between CI jobs

CI jobs

Shared cache for developers

  • no need to rebuild other people’s changes

Sharing strategies

  • CI pushes only

  • CI and developers push

How much faster?

33.9%

Measuring is hard

  • code architecture

  • build structure

  • nature of the change

  • what tasks are cached

Gradle developers

  • slow connections

  • geographically diverse

  • one big :core module with 30%+ of the code

7.96% faster (167 builds)

Gradle’s CI #1

Code quality checks: -25.7%

Cache gains -- stage 1

3 hours saved (42 builds)

Gradle’s CI #2

Linux & Windows integration tests: -33.9%

Cache gains -- stage 3

62 hours saved (32 builds)

Statistics

GE cache statistics

Challenges

  • caching uses same metadata* as incremental build

  • more permanent, no clean to fix problems

* — that can be faulty

Fix it for good

For third-party tasks

  • more warnings

  • enforce good practices

  • disable caching when unsafe

  • later: isolated execution

Fixes in Gradle

  • better stale file cleanup

  • track Java version

  • remove Java-Groovy compilation overlap

Improved documentation

  • user guide chapter for inputs and outputs

  • build cache guide

New problems with sharing outputs

  • non-homogenous environments

    • OS, locale, env. vars

    • tool versions installed

  • new concept: relocatability

    • where’s your $HOME?

Relocatability

class CustomTask extends DefaultTask {

  @PathSensitive(PathSensitivity.RELATIVE)
  @InputDirectory
  File sourceDirectory

  // ...
}
@CacheableTask
class ConcatenateTask extends DefaultTask {
  @PathSensitive(PathSensitivity.NONE)
  @InputFiles FileCollection sourceFiles
  @OutputFile File outputFile

  @TaskAction
  void concatenate() {
    outputFile.createNewFile()
    sourceFiles.each {
      outputFile << it.text + '\n'
    }
  }
}
@CacheableTask
class ConcatenateTask extends DefaultTask {
  @PathSensitive(PathSensitivity.NONE)
  @InputFiles FileCollection sourceFiles
  @OutputFile File outputFile

  @TaskAction
  void concatenate() {
    outputFile.createNewFile()
    sourceFiles.sort().each {
      outputFile << it.text + '\n'
    }
  }
}

Roadmap

  • opt-in: tasks are marked with @CacheableTask

    • custom tasks support later

  • Java projects supported in Gradle 4.0

  • caching support coming in Android plugin 3.0

  • full Scala, Groovy and native support coming

More resources

Learn more at gradle.org

Summit talks

  • Slack: #build-cache

  • Moving existing builds towards full cacheability

    • Tomorrow 1pm in MEDITERRANEAN III

    • Stefan Wolf and Sterling Greene

  • Maximizing incrementality

    • Tomorrow 4.40pm in MEDITERRANEAN III

    • Cédric Champeau

Q & A