Controlling JAR Content Merging

Shadow allows for customizing the process by which the output JAR is generated through the ResourceTransformer interface. This is a concept that has been carried over from the original Maven Shade implementation. A ResourceTransformer is invoked for each entry in the JAR before being written to the final output JAR. This allows a ResourceTransformer to determine if it should process a particular entry and apply any modifications before writing the stream to the output.

Important: ResourceTransformer follows a guaranteed processing order:

1. Project files first: All files in projects are processed before any dependency files.
2. Dependency files second: Files from configurations (runtime dependencies) or added via ShadowJar.from are processed after project files.

This ordering is crucial when merging configuration files where you want to preserve project-specific values while merging in additional data from dependencies.

Handling Duplicates Strategy

ShadowJar is a subclass of org.gradle.api.tasks.AbstractCopyTask, which means it honors the duplicatesStrategy property as its parent classes do. There are several strategies to handle:

• EXCLUDE: Do not allow duplicates by ignoring subsequent items to be created at the same path.
• FAIL: Throw a DuplicateFileCopyingException when subsequent items are to be created at the same path.
• INCLUDE: Do not attempt to prevent duplicates.
• INHERIT: Use the same strategy as the parent copy specification.
• WARN: Do not attempt to prevent duplicates, but log a warning message when multiple items are to be created at the same path.

see more details about them in DuplicatesStrategy.

ShadowJar recognizes EXCLUDE as the default, if you want to change the strategy, you can override it like:

KotlinGroovy

tasks.shadowJar {
  duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or something else.
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or something else.
}

Different strategies will lead to different results for foo/bar files in the JARs to be merged:

• EXCLUDE: The first foo/bar file will be included in the final JAR.
• FAIL: Fail the build with a DuplicateFileCopyingException if there are duplicate foo/bar files.
• INCLUDE: Duplicate foo/bar entries will be included in the final JAR.
• INHERIT: Fail the build with an exception like Entry .* is a duplicate but no duplicate handling strategy has been set.
• WARN: Warn about duplicates in the build log, this behaves exactly as INHERIT otherwise.

NOTE: The duplicatesStrategy takes precedence over transforming and relocating. If you mix the usages of duplicatesStrategy and ResourceTransformer like below:

KotlinGroovy

tasks.shadowJar {
  duplicatesStrategy = DuplicatesStrategy.EXCLUDE // The default strategy.
  mergeServiceFiles()
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  duplicatesStrategy = DuplicatesStrategy.EXCLUDE // The default strategy.
  mergeServiceFiles()
}

The ResourceTransformers like ServiceFileTransformer will not work as expected as the duplicate resource files fed for them are excluded beforehand. However, this behavior might be what you expected for duplicate foo/bar files, preventing them from being included.

Want ResourceTransformers and duplicatesStrategy to work together? There are several common steps to take:

1. Set the default strategy to INCLUDE or WARN.
2. Apply your ResourceTransformers.
3. Remove duplicate entries by
   • overriding the default strategy for specific files to EXCLUDE or FAIL using filesMatching, filesNotMatching, or eachFile functions
   • or applying PreserveFirstFoundResourceTransformer for specific files
   • or write your own ResourceTransformer to handle duplicates
   • or mechanism similar.

Alternatively, you can follow these steps:

1. Set the default strategy to EXCLUDE or FAIL.
2. Apply your ResourceTransformers.
3. Bypass the duplicate entries which should be handled by the ResourceTransformers using filesMatching, filesNotMatching, or eachFile functions to set their duplicatesStrategy to INCLUDE or WARN.

Optional steps:

• Enable ShadowJar.failOnDuplicateEntries to check duplicate entries in the final JAR. This can also ensure the regressions are caught in the future.
• Use Diffuse to diff the JARs.

Here are some examples:

KotlinGroovy

tasks.shadowJar {
  // Step 1.
  duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
  // Step 2.
  mergeServiceFiles()
  // Step 3. Using `filesNotMatching`:
  filesNotMatching("META-INF/services/**") {
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
  }
  // Step 3. Using `PreserveFirstFoundResourceTransformer`:
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.PreserveFirstFoundResourceTransformer>() {
    resources.add("META-INF/foo/**") // Or something else where the first occurrence should be preserved.
  }
}

tasks.shadowJar {
  // Step 1.
  duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
  // Step 2.
  mergeServiceFiles()
  // Step 3. Using `filesMatching`:
  filesMatching("META-INF/services/**") {
    duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
  }
  // Step 3. Using `eachFile`:
  eachFile {
    if (path.startsWith("META-INF/services/")) {
      duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
    }
  }
}

tasks.shadowJar {
  // Optional step.
  failOnDuplicateEntries = true
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Step 1.
  duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
  // Step 2.
  mergeServiceFiles()
  // Step 3. Using `filesNotMatching`:
  filesNotMatching('META-INF/services/**') {
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
  }
  // Step 3. Using `PreserveFirstFoundResourceTransformer`:
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.PreserveFirstFoundResourceTransformer) {
    resources.add('META-INF/foo/**') // Or something else where the first occurrence should be preserved.
  }
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Step 1.
  duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
  // Step 2.
  mergeServiceFiles()
  // Step 3. Using `filesMatching`:
  filesMatching('META-INF/services/**') {
    duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
  }
  // Step 3. Using `eachFile`:
  eachFile {
    if (it.path.startsWith('META-INF/services/')) {
      it.duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
    }
  }
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Optional step.
  failOnDuplicateEntries = true
}

Basic ResourceTransformer Usage

For simpler use cases, you can create a basic transformer:

KotlinGroovy

import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement

class MyTransformer : ResourceTransformer {
  override fun canTransformResource(element: FileTreeElement): Boolean = true
  override fun transform(context: TransformerContext) {}
  override fun hasTransformedResource(): Boolean = true
  override fun modifyOutputStream(os: ZipOutputStream, preserveFileTimestamps: Boolean) {}
}

tasks.shadowJar {
  transform<MyTransformer>()
}

import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement

class MyTransformer implements ResourceTransformer {
  @Override boolean canTransformResource(FileTreeElement element) { return true }
  @Override void transform(TransformerContext context) {}
  @Override boolean hasTransformedResource() { return true }
  @Override void modifyOutputStream(ZipOutputStream os, boolean preserveFileTimestamps) {}
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  transform(MyTransformer)
}

Additionally, a ResourceTransformer can accept a closure to configure the provided ResourceTransformer. An instantiated instance of a ResourceTransformer can also be provided.

KotlinGroovy

import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement

class MyTransformer(@get:Input var enabled: Boolean = false) : ResourceTransformer {
  override fun canTransformResource(element: FileTreeElement): Boolean = enabled
  override fun transform(context: TransformerContext) {}
  override fun hasTransformedResource(): Boolean = enabled
  override fun modifyOutputStream(os: ZipOutputStream, preserveFileTimestamps: Boolean) {}
}

tasks.shadowJar {
  // Initialize with default constructor and configure with closure.
  transform<MyTransformer>() {
    enabled = true
  }

  // Or use the instantiated instance with closure.
  transform(MyTransformer(false)) {
    enabled = true
  }
}

import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement

class MyTransformer implements ResourceTransformer {
  @Input boolean enabled
  MyTransformer(boolean enabled = false) { this.enabled = enabled }
  @Override boolean canTransformResource(FileTreeElement element) { return enabled }
  @Override void transform(TransformerContext context) {}
  @Override boolean hasTransformedResource() { return enabled }
  @Override void modifyOutputStream(ZipOutputStream os, boolean preserveFileTimestamps) {}
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Initialize with default constructor and configure with closure.
  transform(MyTransformer) {
    enabled = true
  }

  // Or use the instantiated instance with closure.
  transform(new MyTransformer(false)) {
    enabled = true
  }
}

Merging Service Descriptor Files

Java libraries often contain service descriptors files in the META-INF/services directory of the JAR. A service descriptor typically contains a line delimited list of classes that are supported for a particular service. At runtime, this file is read and used to configure library or application behavior.

Multiple dependencies may use the same service descriptor file name. In this case, it is generally desired to merge the content of each instance of the file into a single output file. The ServiceFileTransformer class is used to perform this merging. By default, it will merge each copy of a file under META-INF/services into a single file in the output JAR. You can use either the short syntax method mergeServiceFiles() or the full syntax method transform to add the ServiceFileTransformer:

KotlinGroovy

tasks.shadowJar {
  // Short syntax.
  mergeServiceFiles()

  // Full syntax.
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer>()
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Short syntax.
  mergeServiceFiles()

  // Full syntax.
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer)
}

Groovy Extension Module descriptor files (located at META-INF/services/org.codehaus.groovy.runtime.ExtensionModule) are ignored by the ServiceFileTransformer. This is due to these files having a different syntax than standard service descriptor files. Use the mergeGroovyExtensionModules() method to merge these files if your dependencies contain them.

Configuring the Location of Service Descriptor Files

By default, the ServiceFileTransformer is configured to merge files in META-INF/services. This directory can be overridden to merge descriptor files in a different location.

KotlinGroovy

tasks.shadowJar {
  // Short syntax.
  mergeServiceFiles {
    path = "META-INF/custom"
  }

  // Full syntax.
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer>() {
    path = "META-INF/custom"
  }
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Short syntax.      
  mergeServiceFiles {
    path = 'META-INF/custom'
  }

  // Full syntax.
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer) {
    path = 'META-INF/custom'
  }
}

Excluding/Including Specific Service Descriptor Files From Merging

The ServiceFileTransformer class supports specifying specific files to include or exclude from merging.

KotlinGroovy

tasks.shadowJar {
  // Short syntax.
  mergeServiceFiles {
    exclude("META-INF/services/com.acme.*")
  }

  // Full syntax.
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer>() {
    exclude("META-INF/services/com.acme.*")
  }
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Short syntax.
  mergeServiceFiles {
    exclude 'META-INF/services/com.acme.*'
  }

  // Full syntax.
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer) {
    exclude 'META-INF/services/com.acme.*'
  }
}

Merging Groovy Extension Modules

Shadow provides a specific transformer for dealing with Groovy extension module files. This is due to their special syntax and how they need to be merged together. The GroovyExtensionModuleTransformer will handle these files. The ShadowJar task also provides a short syntax method to add this transformer.

KotlinGroovy

tasks.shadowJar {
  // Short syntax.
  mergeGroovyExtensionModules()

  // Full syntax.
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.GroovyExtensionModuleTransformer>()
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // Short syntax.
  mergeGroovyExtensionModules()

  // Full syntax.
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.GroovyExtensionModuleTransformer)
}

Merging Log4j2 Plugin Cache Files (Log4j2Plugins.dat)

Log4j2PluginsCacheFileTransformer is a ResourceTransformer that merges META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat plugin caches from all the jars containing Log4j 2.x Core components. It’s a Gradle equivalent of Log4j Plugin Descriptor Transformer.

KotlinGroovy

tasks.shadowJar {
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer>()
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer)
}

Appending Text Files

Generic text files can be appended together using the AppendingTransformer. Each file is appended using separators (defaults to \n) to separate content. The ShadowJar task provides a short syntax method of append(String) to configure this transformer.

KotlinGroovyKotlinGroovy

tasks.shadowJar {
  append("test.properties")
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  append 'test.properties'
}

tasks.shadowJar {
  // short syntax
  append("resources/application.yml", "\n---\n")
  // full syntax
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.AppendingTransformer>() {
    resource = "resources/custom-config/application.yml"
    separator = "\n---\n"
  }
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  // short syntax
  append('resources/application.yml', '\n---\n')
  // full syntax
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.AppendingTransformer) {
    resource = 'resources/custom-config/application.yml'
    separator = '\n---\n'
  }
}

Appending XML Files

XML files require a special transformer for merging. The XmlAppendingTransformer reads each XML document and merges each root element into a single document. There is no short syntax method for the XmlAppendingTransformer. It must be added using the transform methods.

KotlinGroovy

tasks.shadowJar {
  transform<com.github.jengelman.gradle.plugins.shadow.transformers.XmlAppendingTransformer>() {
    resource = "properties.xml"
  }
}

tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
  transform(com.github.jengelman.gradle.plugins.shadow.transformers.XmlAppendingTransformer) {
    resource = 'properties.xml'
  }
}