Staging the future; our path to secure and efficient deployments

In today’s fast-paced tech world, deploying projects securely and efficiently is paramount. At Yummygum, we recognized the need for a robust staging system that not only ensured the security of our deployments but also streamlined the process. In this blog post, we’ll take you through the journey of our staging system, from its inception to its current state.

Background & motivation

The original version of our staging system can be traced back to a client marketing website project where we had a need for ensuring that the preview environment for the client was behind a secure login to ensure the project would not leak before the release date. We threw together a quick prototype that we continuously used during the project.

When we extended its use to another client project, the feedback was overwhelmingly positive. The prototype succeeded in addressing security concerns and the gaps in our deployment process became evident. It wasn’t just a tool; it was a solution that filled a significant gap in our workflow.

Recognizing its potential, we made the decision to evolve this prototype into a full-fledged system. Our aim? To make it a cornerstone of our deployment process, ensuring security, efficiency and consistency in every project we undertook.

The evolution of our staging server

Our staging server has gone through several iterations, ensuring that we optimize it whenever we find issues that limit our productivity.

Version 1: The prototype

What started as a side project quickly became an integral part of our deployment process. Built to cater to our needs during the Martin Garrix project, this version was designed to add an extra layer of security, ensuring the project wouldn’t leak before its official release. At its core, it was a simple express server equipped with a login panel and enhanced security with Duo for multi-factor authentication (MFA). The immediate success of this prototype was evident when we continued to use it for another project. However as with any initial version, it had its set of challenges. The manual labor required to update the client projects and the limited authentication mechanisms were some of the areas we identified for improvement.

Version 2: The upgrade to Arcade

Addressing the challenges we faced in version one, we decided to actually spend time working on the system to turn it into a full-fledged application, given the internal code name Arcade.

Tech stack and integrations

In our pursuit to refine the staging system, version 2 was architected using a combination of Next JS for the Admin panel, Express for the middleware, Keystone JS with Prisma and PostgreSQL for the API. To further enhance our workflow, we seamlessly integrated a Slack app and a GitHub Apps bot, ensuring that we informed our team when deployments of projects happened, in addition to providing extra logging when users or projects were modified for an additional layer of security. The GitHub bot allowed us to automatically deploy and build projects when changes were pushed.

Automated processes

One of the main objectives of this upgrade was to address the manual labor that was involved in version one. By introducing automatic building and deployment, we drastically cut down both time and effort for each project deployment.

Admin panel and access control

The newly designed admin panel in version two empowered us to manage projects and users with ease. Moreover we implemented a robust access control system, ensuring that users and clients had tailored access, aligning with their specific roles and needs.

Security revisions

Feedback is a powerful tool. While the security that Duo provided was great and definitely important, we listened to the feedback and struggles faced by fellow employees and clients. We decided to phase it out in version two, aiming for a more streamlined and user-friendly login experience.

Centralized hosting

Perhaps one of the most significant shifts was moving from a project-specific deployment model in version one to a centralized hosting in version two. This meant all projects were hosted on a single server, simplifying the deployment process and ensuring uniformity across projects.

Challenges & solutions

The current version of the staging server is not perfect, there are definitely some challenges that we are currently working on.

Challenge 1: Inconsistent build process

One of the most perplexing challenges we faced was the inconsistencies in our build process. There were times when the build would either fail to run or not succeed at all. Sometimes the build would run with an outdated version of the repo, even though the output of the git pull would say it was successful. Our team spent countless hours debugging, trying to pinpoint the root cause.

After careful consideration, we concluded that the best course of action would be to transition the build process from our servers to GitHub Actions. This decision came with a bunch of benefits:

• Security: By leveraging GitHub actions, we could ensure a more secure build process, minimizing the potential for vulnerabilities.

• Performance: GitHub Actions provided a more streamlined and efficient build process, reducing the time taken for each build.

• Cost reduction: Moving to GitHub actions allowed us to cut down on server costs, making it a cost-effective solution.

• Consistency: With GitHub Actions, we achieved a more consistent build process, eliminating the inconsistencies we previously faced.

This transition to GitHub actions is the cornerstone of what will become version three of our staging server, a project we are actively working on and are very excited to roll out soon.

Challenge 2: storage scalability issues

As the amount of projects grew, so did our storage needs. While having so many projects running at once is great of course, it brought along a new challenge. Running a large number of projects began to consume a significant amount of storage on our server. While the storage wasn’t inherently an issue, the way in which we had to scale it,was. Each time we needed to increase storage, it required us to take the server offline, and resize it which significantly disrupted our workflow. Moreover the cost of scaling the server this way was exponential rather than linear, making it a financial concern as well.

In our upcoming version three, we have decided to address this head-on. Instead of relying on the primary disk of the server, we are transitioning to using volumes introduced by DigitalOcean. These are highly available units of block storage that can be seamlessly attached to our server. Not only does this solution offer great flexibility and scalability, but it also ensures that we can scale without significant downtime or disproportionate cost implications.

What is next?

As we continue to innovate and refine our staging system, we are excited to share some of the major enhancements we are working on for version three.

• API code rewrite: The heart of the system the API has received a massive overhaul in version three, optimizing the code for the new build system but also ensuring future scalability and adaptability.

• Transition of the build process: We are moving the entire build process over to GitHub utilizing the GitHub Actions to build the processes for us. This shift promises to streamline our deployment workflow even further.

• Storage evolution: Projects will no longer be stored on the primary disk. Instead, we are transitioning to using volumes, ensuring that our storage scales not just efficiently but also cost-effectively.

• Docker containers for development: Our development process is getting a major upgrade. We are moving to run our applications inside Docker containers. Given the multifaceted nature of the staging environment, this allows us to run all parts of the staging server simultaneously, ensuring a smoother development experience.

• Cloudflare tunnels for local development: Local development often comes with its set of challenges, especially with this staging server being able to have domains and subdomains running locally that are accessible to the internet makes development a lot easier. Because of this we are integrating cloudflare tunnels into our local development environment, ensuring that we have proper domains and HTTPS.

Finally lets have a look at what is next in version four. As we gear up for version three, our eyes are already set on the horizon making plans for version four. We’re looking forward to implementing a beautifully redesigned admin panel, which was a collaborative effort between our UX and Visual design squads. These designs came to be during our annual Yummygum week, where we can focus on internal Yummygum projects instead of focussing on client projects. With version four the clients and employees can expect more intuitive, visually appealing and user-friendly admin experience.

Conclusion

From its humble beginnings as a prototype for the Martin Garrix project to its evolution into a core component of our workflow, our staging system has come a long way. Each version brought with it new innovations, addressing challenges and enhancing our deployment process. With the introduction of version three we are addressing some hard to tackle bugs, along with optimizing our API, storage and development processes.

Our journey with the staging system is a testament to our commitment to continuous improvement, innovation and excellence. We are very excited to see where we can get the staging environment over the next few years.

Interested to see what we can do for your project? Learn more about how we go about development at Yummygum, or set up a call!