Containerizing Legacy Java Apps: Why We Switched to Jib and Cloud Build

August 20, 2021

In part 1 we used the strangler fig pattern with Cloud Load Balancing to peel traffic off the monolith. The unanswered question was where the new traffic was going. The answer is containers — but containerizing a decade-old Java app with a stock Dockerfile turned out to be the slowest part of the pipeline. We threw the Dockerfiles out and moved to Jib + Cloud Build. Here's the reasoning.

The layering problem

A typical legacy Java service has two very different kinds of content:

  1. Application code — maybe 5MB, changes constantly.
  2. Dependencies (Spring Boot, Hibernate, the usual suspects) — easily 200MB+, changes rarely.

The naive Dockerfile treats both the same way:

FROM openjdk:11-jre-slim
COPY target/my-app-1.0.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]

Every code change invalidates the COPY layer, which means rebuilding and re-pushing 200MB of dependencies on every commit. We were sitting at ~10 minutes per build, which is enough time for developers to context-switch out and not come back. Productivity is hard to recover after that.

Jib's contribution: layers by volatility

Jib is a Maven/Gradle plugin from Google that builds OCI images directly, without a Docker daemon. The thing that mattered for us is how it lays out the image:

  • Layer 1: dependencies
  • Layer 2: resources
  • Layer 3: classes

Change a single Java file and Jib pushes the classes layer (a few KB). The 200MB dependency layer is already in the registry. Build cycles dropped from ~10 minutes to ~45 seconds for incremental changes.

Skipping the Docker daemon is the secondary win. Running Docker-in-Docker inside a build pipeline is a known footgun for both security and performance, and Jib sidesteps it entirely.

Wiring it into Cloud Build

We paired Jib with Cloud Build because we didn't want to maintain a Jenkins server. Pay for the build minutes, don't patch a build OS.

   ┌───────────┐  git push   ┌────────────────────┐  trigger
    Developer  ──────────►  Cloud Source Repos  ─────────┐
   └───────────┘                / GitHub                   
                             └────────────────────┘          
                                                             
                                              ┌─────────────────────────┐
                                                 Cloud Build Pipeline  
                                                ┌───────────────────┐  
                                                 Maven + Jib         
                                                 Plugin              
                                                └─────────┬─────────┘  
                                              └────────────┼────────────┘
                                                            build & push
                                                            layers
                                                           
                                              ┌─────────────────────────┐
                                                 Artifact Registry     
                                              └────────────┬────────────┘
                                                            deploy
                                                           
                                              ┌─────────────────────────┐
                                                    GKE Cluster        
                                              └─────────────────────────┘

The pom.xml change:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>3.1.4</version>
    <configuration>
        <to>
            <image>gcr.io/my-project/my-service</image>
        </to>
        <container>
            <mainClass>com.example.MyApplication</mainClass>
        </container>
    </configuration>
</plugin>

cloudbuild.yaml collapses to:

steps:
  - name: 'gcr.io/cloud-builders/mvn'
    args: ['compile', 'jib:build']

That's the whole pipeline. No docker build, no docker push.

What we got out of the switch

  • Build speed. Incremental builds dropped roughly an order of magnitude.
  • Smaller attack surface. Jib defaults to Distroless base images. There's no shell, no package manager, almost nothing for an attacker to use if they get RCE. It's not a silver bullet but it's a meaningful default.
  • Reproducibility. Same inputs produce the same image digest. We stopped getting "works on my machine" reports caused by timestamp drift in image layers.

What tripped us up

Exploded WAR assumptions. The legacy app read files relative to catalina.base — paths that only exist when a WAR is unpacked under Tomcat. Jib lays the app out exploded by default, which mostly worked, but we still had to remove a few hardcoded servlet-context path assumptions in the legacy code.

Private Maven repositories. Cloud Build can't reach an internal Nexus on its own. We encrypted settings.xml with Cloud KMS and decrypted it inside the build step so Maven could authenticate.

Local dev workflow. Developers still want to iterate locally. jib:dockerBuild builds the image into the local Docker daemon, which keeps the inner-loop story intact for anyone who isn't ready to live entirely in CI.

Wrap up

The point of moving to Jib wasn't really build speed — it was to stop spending team time on Dockerfile syntax so we could spend it on splitting the monolith. Fast, small, secure containers landing in Artifact Registry is now table stakes; nobody on the team thinks about it.

Now that services are running in containers, the next question is networking. How do you keep the cart team's services from accidentally talking to the inventory team's services without dumping everyone into a shared default VPC? Part 3 covers our Shared VPC design — host projects, service projects, secondary IP ranges, and the IP-overlap mistakes that almost burned us.