Controlling JAR Content Merging¶
Shadow allows for customizing the process by which the output JAR is generated through the
ResourceTransformer
interface. This is a concept that has been carried over from the original
Maven Shade implementation. A ResourceTransformer
is invoked for each entry in the JAR before
being written to the final output JAR. This allows a ResourceTransformer
to determine if it
should process a particular entry and apply any modifications before writing the stream to the output.
Important: ResourceTransformer
follows a guaranteed processing order:
- Project files first: All files in projects are processed before any dependency files.
- Dependency files second: Files from configurations (runtime dependencies) or added via
ShadowJar.from
are processed after project files.
This ordering is crucial when merging configuration files where you want to preserve project-specific values while merging in additional data from dependencies.
Handling Duplicates Strategy¶
ShadowJar
is a subclass of org.gradle.api.tasks.AbstractCopyTask
, which means it honors the
duplicatesStrategy
property as its parent classes do. There are several strategies to handle:
EXCLUDE
: Do not allow duplicates by ignoring subsequent items to be created at the same path.FAIL
: Throw aDuplicateFileCopyingException
when subsequent items are to be created at the same path.INCLUDE
: Do not attempt to prevent duplicates.INHERIT
: Use the same strategy as the parent copy specification.WARN
: Do not attempt to prevent duplicates, but log a warning message when multiple items are to be created at the same path.
see more details about them in DuplicatesStrategy
.
ShadowJar
recognizes EXCLUDE
as the default, if you want to change the strategy, you can override it like:
tasks.shadowJar {
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or something else.
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or something else.
}
Different strategies will lead to different results for foo/bar
files in the JARs to be merged:
EXCLUDE
: The firstfoo/bar
file will be included in the final JAR.FAIL
: Fail the build with aDuplicateFileCopyingException
if there are duplicatefoo/bar
files.INCLUDE
: Duplicatefoo/bar
entries will be included in the final JAR.INHERIT
: Fail the build with an exception likeEntry .* is a duplicate but no duplicate handling strategy has been set
.WARN
: Warn about duplicates in the build log, this behaves exactly asINHERIT
otherwise.
NOTE: The duplicatesStrategy
takes precedence over transforming and relocating. If you mix the usages of
duplicatesStrategy
and ResourceTransformer
like below:
tasks.shadowJar {
duplicatesStrategy = DuplicatesStrategy.EXCLUDE // The default strategy.
mergeServiceFiles()
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
duplicatesStrategy = DuplicatesStrategy.EXCLUDE // The default strategy.
mergeServiceFiles()
}
The ResourceTransformer
s like ServiceFileTransformer
will not work
as expected as the duplicate resource files fed for them are excluded beforehand. However, this behavior might be what
you expected for duplicate foo/bar
files, preventing them from being included.
Want ResourceTransformer
s and duplicatesStrategy
to work together? There are several common
steps to take:
- Set the default strategy to
INCLUDE
orWARN
. - Apply your
ResourceTransformer
s. - Remove duplicate entries by
- overriding the default strategy for specific files to
EXCLUDE
orFAIL
usingfilesMatching
,filesNotMatching
, oreachFile
functions - or applying
PreserveFirstFoundResourceTransformer
for specific files - or write your own
ResourceTransformer
to handle duplicates - or mechanism similar.
- overriding the default strategy for specific files to
Alternatively, you can follow these steps:
- Set the default strategy to
EXCLUDE
orFAIL
. - Apply your
ResourceTransformer
s. - Bypass the duplicate entries which should be handled by the
ResourceTransformer
s usingfilesMatching
,filesNotMatching
, oreachFile
functions to set theirduplicatesStrategy
toINCLUDE
orWARN
.
Optional steps:
- Enable
ShadowJar.failOnDuplicateEntries
to check duplicate entries in the final JAR. - Use Diffuse to diff the JARs.
Here are some examples:
tasks.shadowJar {
// Step 1.
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
// Step 2.
mergeServiceFiles()
// Step 3. Using `filesNotMatching`:
filesNotMatching("META-INF/services/**") {
duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
}
// Step 3. Using `PreserveFirstFoundResourceTransformer`:
transform<com.github.jengelman.gradle.plugins.shadow.transformers.PreserveFirstFoundResourceTransformer>() {
resources.add("META-INF/foo/**") // Or something else where the first occurrence should be preserved.
}
}
tasks.shadowJar {
// Step 1.
duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
// Step 2.
mergeServiceFiles()
// Step 3. Using `filesMatching`:
filesMatching("META-INF/services/**") {
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
}
// Step 3. Using `eachFile`:
eachFile {
if (path.startsWith("META-INF/services/")) {
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
}
}
}
tasks.shadowJar {
// Optional step.
failOnDuplicateEntries = true
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Step 1.
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
// Step 2.
mergeServiceFiles()
// Step 3. Using `filesNotMatching`:
filesNotMatching('META-INF/services/**') {
duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
}
// Step 3. Using `PreserveFirstFoundResourceTransformer`:
transform(com.github.jengelman.gradle.plugins.shadow.transformers.PreserveFirstFoundResourceTransformer) {
resources.add('META-INF/foo/**') // Or something else where the first occurrence should be preserved.
}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Step 1.
duplicatesStrategy = DuplicatesStrategy.EXCLUDE // Or FAIL.
// Step 2.
mergeServiceFiles()
// Step 3. Using `filesMatching`:
filesMatching('META-INF/services/**') {
duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
}
// Step 3. Using `eachFile`:
eachFile {
if (it.path.startsWith('META-INF/services/')) {
it.duplicatesStrategy = DuplicatesStrategy.INCLUDE // Or WARN.
}
}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Optional step.
failOnDuplicateEntries = true
}
Basic ResourceTransformer Usage¶
For simpler use cases, you can create a basic transformer:
import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement
class MyTransformer : ResourceTransformer {
override fun canTransformResource(element: FileTreeElement): Boolean = true
override fun transform(context: TransformerContext) {}
override fun hasTransformedResource(): Boolean = true
override fun modifyOutputStream(os: ZipOutputStream, preserveFileTimestamps: Boolean) {}
}
tasks.shadowJar {
transform<MyTransformer>()
}
import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement
class MyTransformer implements ResourceTransformer {
@Override boolean canTransformResource(FileTreeElement element) { return true }
@Override void transform(TransformerContext context) {}
@Override boolean hasTransformedResource() { return true }
@Override void modifyOutputStream(ZipOutputStream os, boolean preserveFileTimestamps) {}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
transform(MyTransformer)
}
Additionally, a ResourceTransformer
can accept a closure to configure the provided
ResourceTransformer
. An instantiated instance of a ResourceTransformer
can also be provided.
import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement
class MyTransformer(@get:Input var enabled: Boolean = false) : ResourceTransformer {
override fun canTransformResource(element: FileTreeElement): Boolean = enabled
override fun transform(context: TransformerContext) {}
override fun hasTransformedResource(): Boolean = enabled
override fun modifyOutputStream(os: ZipOutputStream, preserveFileTimestamps: Boolean) {}
}
tasks.shadowJar {
// Initialize with default constructor and configure with closure.
transform<MyTransformer>() {
enabled = true
}
// Or use the instantiated instance with closure.
transform(MyTransformer(false)) {
enabled = true
}
}
import com.github.jengelman.gradle.plugins.shadow.transformers.ResourceTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import org.apache.tools.zip.ZipOutputStream
import org.gradle.api.file.FileTreeElement
class MyTransformer implements ResourceTransformer {
@Input boolean enabled
MyTransformer(boolean enabled = false) { this.enabled = enabled }
@Override boolean canTransformResource(FileTreeElement element) { return enabled }
@Override void transform(TransformerContext context) {}
@Override boolean hasTransformedResource() { return enabled }
@Override void modifyOutputStream(ZipOutputStream os, boolean preserveFileTimestamps) {}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Initialize with default constructor and configure with closure.
transform(MyTransformer) {
enabled = true
}
// Or use the instantiated instance with closure.
transform(new MyTransformer(false)) {
enabled = true
}
}
Merging Service Descriptor Files¶
Java libraries often contain service descriptors files in the META-INF/services
directory of the JAR.
A service descriptor typically contains a line delimited list of classes that are supported for a particular service.
At runtime, this file is read and used to configure library or application behavior.
Multiple dependencies may use the same service descriptor file name.
In this case, it is generally desired to merge the content of each instance of the file into a single output file.
The ServiceFileTransformer
class is used to perform this merging.
By default, it will merge each copy of a file under META-INF/services
into a single file in the output JAR.
You can use either the short syntax method mergeServiceFiles()
or the full syntax
method transform
to add the ServiceFileTransformer
:
tasks.shadowJar {
// Short syntax.
mergeServiceFiles()
// Full syntax.
transform<com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer>()
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Short syntax.
mergeServiceFiles()
// Full syntax.
transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer)
}
Groovy Extension Module descriptor files (located at
META-INF/services/org.codehaus.groovy.runtime.ExtensionModule
) are ignored by theServiceFileTransformer
. This is due to these files having a different syntax than standard service descriptor files. Use themergeGroovyExtensionModules()
method to merge these files if your dependencies contain them.
Configuring the Location of Service Descriptor Files¶
By default, the ServiceFileTransformer
is configured to merge files in META-INF/services
.
This directory can be overridden to merge descriptor files in a different location.
tasks.shadowJar {
// Short syntax.
mergeServiceFiles {
path = "META-INF/custom"
}
// Full syntax.
transform<com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer>() {
path = "META-INF/custom"
}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Short syntax.
mergeServiceFiles {
path = 'META-INF/custom'
}
// Full syntax.
transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer) {
path = 'META-INF/custom'
}
}
Excluding/Including Specific Service Descriptor Files From Merging¶
The ServiceFileTransformer
class supports specifying specific files to include or exclude
from merging.
tasks.shadowJar {
// Short syntax.
mergeServiceFiles {
exclude("META-INF/services/com.acme.*")
}
// Full syntax.
transform<com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer>() {
exclude("META-INF/services/com.acme.*")
}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Short syntax.
mergeServiceFiles {
exclude 'META-INF/services/com.acme.*'
}
// Full syntax.
transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer) {
exclude 'META-INF/services/com.acme.*'
}
}
Merging Groovy Extension Modules¶
Shadow provides a specific transformer for dealing with Groovy extension module files.
This is due to their special syntax and how they need to be merged together.
The GroovyExtensionModuleTransformer
will handle these files.
The ShadowJar
task also provides a short syntax method to add this transformer.
tasks.shadowJar {
// Short syntax.
mergeGroovyExtensionModules()
// Full syntax.
transform<com.github.jengelman.gradle.plugins.shadow.transformers.GroovyExtensionModuleTransformer>()
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// Short syntax.
mergeGroovyExtensionModules()
// Full syntax.
transform(com.github.jengelman.gradle.plugins.shadow.transformers.GroovyExtensionModuleTransformer)
}
Merging Log4j2 Plugin Cache Files (Log4j2Plugins.dat
)¶
Log4j2PluginsCacheFileTransformer
is a
ResourceTransformer
that merges
META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat
plugin caches from all the jars
containing Log4j 2.x Core components. It’s a Gradle equivalent of
Log4j Plugin Descriptor Transformer.
tasks.shadowJar {
transform<com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer>()
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
transform(com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer)
}
Appending Text Files¶
Generic text files can be appended together using the AppendingTransformer
.
Each file is appended using separators (defaults to \n
) to separate content.
The ShadowJar
task provides a short syntax method of append(String)
to configure
this transformer.
tasks.shadowJar {
append("test.properties")
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
append 'test.properties'
}
tasks.shadowJar {
// short syntax
append("resources/application.yml", "\n---\n")
// full syntax
transform<com.github.jengelman.gradle.plugins.shadow.transformers.AppendingTransformer>() {
resource = "resources/custom-config/application.yml"
separator = "\n---\n"
}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
// short syntax
append('resources/application.yml', '\n---\n')
// full syntax
transform(com.github.jengelman.gradle.plugins.shadow.transformers.AppendingTransformer) {
resource = 'resources/custom-config/application.yml'
separator = '\n---\n'
}
}
Appending XML Files¶
XML files require a special transformer for merging. The XmlAppendingTransformer
reads each XML document and merges each root element into a single document.
There is no short syntax method for the XmlAppendingTransformer
.
It must be added using the transform
methods.
tasks.shadowJar {
transform<com.github.jengelman.gradle.plugins.shadow.transformers.XmlAppendingTransformer>() {
resource = "properties.xml"
}
}
tasks.named('shadowJar', com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
transform(com.github.jengelman.gradle.plugins.shadow.transformers.XmlAppendingTransformer) {
resource = 'properties.xml'
}
}