Sameer
Sameer Senior Software Engineer. Travel photographer.

Why we migrated from Nexus and Jenkins to Github Packages and Actions

This article provides an overview and our learnings about why we migrated from own hosted CI/CD to SaaS CI/CD. The Tignum Engineering Team.

Let’s first understand the history

Software Engineers from Generation X, Millennials generation must have at least once in their life worked with Jenkins (Back in time was Hudson). Java Stack CI/CD evolved around Jenkins. As all of us who started with Tignum in 2017, were using Jenkins for most of the time, that’s why we picked Jenkins for our CI/CD. And to compliment Jenkins we also introduced initially Artifactory later Nexus to host our released artifacts.

How our Self-hosted Jenkins and Nexus setup looked like.

Jenkins was mostly used for CI/CD, but as a Batch processor (We were using our CI for other purpose breaking the rule of: Use the best tool for the right job!). To satisfy this requirement we had two self hosted Jenkins instances running over AWS then self hosted pods on Kubernetes(K8S), mainly in MASTER -> SLAVE (agents) configuration.

Nexus instance was also self hosted over AWS initially then we moved it to K8S.

During this journey from 2017 until 2021 we faced many problems and came up with some solutions to fix some of them.

Problem 1. Security

  • Solution 1: Our 1st approach to make it secure was to use, Open VPN. Of course that makes it secure on the cost of performance plus overhead maintaining Open VPN instance as well.
  • Solution 2: To over come Open VPN issues, we introduced Client Side Certificates. This setup involved writing a small code that was creating a certificate and pushing the information to the certificate repo. But still maintaining the certificate is kind of painful task.

Problem 2. Random Jenkins restarts, auto updates

  • Solution: Disabling auto update feature.

Problem 3. Outdated Plugins

  • To make Jenkins work with our needs, we had to use several outdated plugins which are or ain’t work well together, also updating Jenkins may kill the plugin in completely.

Problem 4. Pipeline builds

  • Again maintenance and writing Jenkinsfile without much of documentation is kind of hard.

Problem 5. Nexus maintenance is nightmare

  • Our DevOps team was 24x7, was patching and fixing the Nexus instance as we always had the No Storage Left Error. So they were manually adding the space to the instance plus we had a CRON job that was deleting the old artifacts periodically.

Last but not the least, Jenkins will fail when its needed the most. As most of the teams were relying on Jenkins for CI/CD these problems grew exponentially and the maintenance was causing DevOps precious time.

To make everybody life easier we came to conclusion (We did a RFC for it! Usually RFC at Tignum is that we create a document with the whole idea of design/implementation, and ask for opinion from Engineering Team.) to try out GitHub Actions and GitHub Packages.

How current CI/CD looks like.

The advantage of migrating to GitHub Action was to use YAML instead of different way of defining the Job with Jenkins. Those YAML files are maintained under .github folder under the respective project that gives the ownership of the flows to the Project/Module owner. We have 3 different environments STG - STAGING aka DEVELOP, PRE - PRE RELEASE and LIVE. Below are the 8 workflows we defined per project.

GHA per Project

  1. Build Develop Manual - For manually building develop or main.
  2. Build PR - When ever a PR is raised for a merge on develop or main then this workflow is executed. This is runs whenever there is commit on the branch.
  3. Publish BootJar - After successful build of develop or main, built jar is pushed to GitHub Packages.
  4. Release Manual - Manually releasing & tagging a fixed version.
  5. Auto Deploy API to STG - After successful built of develop or main, changes are deployed to STAGING aka DEVELOP environment.
  6. Deploy API to LIVE - Manual Deployment of fixed version of API to LIVE environment.
  7. Deploy API to PRE - Manual Deployment of fixed version of API to PRE RELEASE environment.
  8. Deploy API to STG Manual - Manual Deployment of API to STAGING environment.

Detailed information about some of these workflows.

PR Build workflow

CI/CD workflow

Our Learnings after we migrated to GitHub Actions and Packages.

  • Stable CI /CD is not a Myth!
  • Now the project CI/CD is maintained by respective team, instead of Infra team.
  • Saved a lot of developer time, troubleshoot the issues with Jenkins and Nexus.
  • Saved resources as now we use GitHub instance instead of self hosted.
  • Cost effective, as most basic subscription of GitHub provides access to Actions and Packages.
  • Stable and efficient as compared to Jenkins and Nexus.
  • Large Marketplace.
  • Large Developer community, more and more free actions (e.g - Slack Notification etc.) available.
  • Very short migration time.
  • We still have one instance of Jenkins running for Batch Processing, may be now its the time to migrate to something else.

Conclusion

We migrated to GitHub Actions and Packages out of frustration we had with our old setup. This gave us a chance to try out latest tools rather than being with old and outdated technologies. The process that we have setup may not be ideal for everyone, but it solves most of the issues we were facing in past. Next step is to make these workflow more refined and efficient.

As I said earlier, Stable CI/CD is not a Myth!

References