Environments
The traditional approach to software testing involves the use of multiple pre-production, or staging, environments. A snapshot of the application is deployed to the first environment in the chain and then promoted to the next environment as soon as all the tests pass, until this version of the application finally reaches production.
The common perception here is that testing the application multiple times provides a greater guarantee of quality. While this may be true for monolithic applications, the opposite is true for distributed, serverless applications. Simulating the variables and emergent complexity of your serverless application’s production environment is impossible. You should instead run tests in as few environments as you are comforta‐ ble with; ideally no more than one or two. The further away a code change is from production, the less confidence (and therefore value) a test can provide.
One of the conundrums of serverless is the proliferation of non-production environ‐ ments. The ability to provision and replicate applications across AWS accounts with speed and integrity is definitely a major benefit of serverless. However, this has been misused in an attempt to provide temporary environments to run automated tests for every single pull request. This is ultimately the antithesis of continuous integration. Many serverless teams that have adopted this practice have seen their pull request integration times exponentially increase as their codebases grow.
These ephemeral pull request environments can still be useful, but they should be used sparingly for specific scenarios. In general, you should use pull request environments with caution and only run tests against deployed resources when you absolutely need to. Instead, you should leverage instant environments if you need to run integration tests on deployed resources. Instant environments are provisioned just for tests and are isolated to the system under test, which is usually two or three integrated components.