The development and deployment of modern apps is very different from how it used to be. Networked applications are now very common. Development at scale is now common: hundreds of developers working on a system, which may be split into many microservices. Deployment at scale is now also common: apps serving millions of users. Various practices emerged, or became popular, to deal with these new challenges. I've learned many practices over the past decade, but resources that describe these practices appear to be scattered. This post is an attempt to inventorize the various practices that are used for the development, deployment and management of modern apps.

In this part 1 will discuss practices in the following categories.

  • Quality assurance
  • Delivery
  • Integration

In future parts, I will discuss other categories such as:

  • Monitoring & alerting
  • Observability & troubleshooting
  • Reliability
  • Performance
  • Scalability
  • Service discovery
  • Security
  • Protocols
  • Business transactions
  • API management

Quality assurance

  • Automated testing

    Manual testing takes a lot of time and is error-prone. By writing automated tests, you gain more confidence during coding. You do not have to worry about introducing bugs and regressions all the time and double-checking every change rigorously. This speeds up development and reduces the amount of new bugs and regressions.

    Since automated tests can be run quickly and at scale, this practice reduces the need for QA personnel and allows you to deliver apps faster.

    Of course, this practice is only as effective as the quality of the tests themselves, so developers must constantly strive to keep the test suite high-quality.

  • Continuous testing

    You don't want to run tests too late. A bug is best fixed as closely as possible to the commit that caused it. In an organizational setting, different developers have different habits and different levels of discipline. Some developers may commit code while forgetting to run the automated test suite. Continuous testing is the practice of running tests on a server. The server enforces that tests are run, and gives the team feedback on when exactly a breakage occurred and by which change. Continuous testing thus introduces a level of standardization.

    Most continuous testing practices operate on the push level, e.g. every time a developer pushes a set of commits, the test suite is run against the latest commit in that set. But there are also practices that operate on a time level. The latter is useful for tracking a quantifiable quality metric. A good example is Mozilla's Are We Fast Yet? which tracks the performance of the Firefox codebase on a daily basis.

Delivery

  • Continuous delivery

    Continuous delivery is the practice of ensuring that the code is always in a deployable state, even if you choose not to deploy yet for business reasons. It goes one step further than continuous testing.

  • Deployment server

    In the old days, deploying an app often means uploading a bunch of files from one's laptop to the production server and running a bunch of commands there. This practice is no longer encouraged in organizations at scale. Instead, it is common practice to have a server perform the deployment in an automated manner. Benefits:

    • The people who are allowed to initiate a deployment, don't have access to production credentials, reducing security risks. If a person leaves the organization or their laptop gets stolen, you don't have to change production credentials.
    • The people who are allowed to initiate a deployment, don't have to set up anything on their laptop. Especially for complicated deployments which involves many parts, standardizing and automating the deployment via a server makes things less error-prone.
    • Makes deployment faster since there are no manual steps involved.
    • Developers don't have to care that much about production concerns and can focus on development. Indeed, they often don't want to care.

  • Continuous deployment

    Continuous deployment is a practice that combines (and goes further than) the practices of having a deployment server, and that of continuous delivery. If all tests pass, then the app can and will be deployed, without a human initiator. Organizations that use continuous deployment can push hundreds or even thousands of releases per day. This practice forces the organization to keep the deployment process as fast and lean as possible, as well as forces the test suite to be high-quality. It also becomes a good idea to roll out deployments slowly (using the rolling deployment or canary deployment pattern) just in case something goes wrong.

  • Delivery format

    Apps can be delivered to the production servers in a variety of formats. There are roughly 4 categories:

    • Source: you deliver the app's source to the production server and compile and install it there, possibly including dependencies that aren't expected to be already installed. Example: Ruby apps with Capistrano and Bundler (Ruby is expected to be already installed).
    • Binary: you compile the app (if possible) and bundle it together with any dependencies that aren't expected to be already installed. You then deliver this package to the production server. Example: Java JARs (the JRE is expected to be already installed), or apps packaged as Debian packages.
    • Container: you compile the app (if possible) and bundle all dependencies, including OS libraries but excluding the kernel. It is to be run on an OS that run multiple containers on the same kernel. Example: Docker.
    • VM: you compile the app (if possible) and bundle all dependencies, including OS libraries and the kernel, into a VM image. You then deploy a VM based on that image.

Integration

  • Continuous integration

    Continuous integration is an ambiguous term and seems to be used for many different purposes. People use it to refer to either the practice continuous testing, or the practice of having a deployment server, or both. The server that runs tests and the server that performs deployment can both be called "the continuous integration server". Thus, continuous deployment is typically also implement with a continuous integration server.

References & further reading

What's next?

What more practices do you think I should cover? Let me know via Twitter of email.