What do you think of Continuous Deployment, where you push code to main and, if the tests pass, that code is automatically deployed to production?

For high stakes systems an alternative I've used that gives much of the benefits while avoiding any risk is Continous Deployment to a staging environment - that way every change is instantly available for manual testing but there's still a deliberate "go live" moment with an extra human in the loop

@simon a big old it depends, internal libraries, CD, applications to staging CD, external libraries, people like to have understandable change sets, so I often batch changes, CD to a beta tag though

@simon See, my experience is that a (dedicated) staging environment *feels* like it reduces risk, but actually increases it: you’re more likely to treat production as a pet, and batch up changes from fear, where spinning up dedicated test environments before merging changes (or, better still, blue-green deploying) actually tests for issues explicitly. But, I’ll grant, requires more trust in the process, as well as more process being built.

@simon we go commit, deploy to stage, automated smoke tests, deploy to production for most of our sites.

We have one that has a really long deploy time that might break things when the site is under heavy load. For this we make use of the circle ci manual approval feature, but that's mostly to prevent accidental deploys rather than for QA

@simon This is what we do where I work, and it seems like the most sensible choice. The convenience of deploying automatically is going to be completely offset by that one bug that gets deployed, which would have been caught if it had sat in the staging environment for a few days first.

@simon agreed. I like to use proper semver tagging for projects that want that go-live step, with automatic deployment of the new tag. So even the go-live is completely hands-off once you’ve done the tag.

@simon I have become a feature flag convert! Ship code behind a feature flag, such that the system doesn’t change upon deployment. Then, ramp feature flag to a small percentage of requests and compare metrics against the “control” group.

@simon $dayjob has better test coverage than many projects, but our QA pros still find bugs. Other issues include risk of deadlock deploying schema changes on a large and active database, and customers who dislike being surprised by unannounced changes, even if we do think they're improvements. But automatic deploy to staging (well, scheduled twice a day, plus more on demand when requested) has been a huge win.

@simon this is what I do.

@simon this is the way.

@simon Yeah for my work I'd wish we had CD at least into a QA. And one button push each for pilot/live. With very small increments and a fix forward approach by default (but rollback on button push available in rarer cases - though that might just be a git revert usually)

@stevemarshall yeah, I totally buy that. I think the staging environment option is good for when the test suite for an older system just isn't thorough enough to be trustworthy

@simon @stevemarshall the one time I worked on a system that had explicitly designed for a staging system for testing purposes had then (before my time there) also spun out testing, qa, qa2, qa3, etc, all of which showing subtle differences, all of which gave little confidence when a prod deploy happened such that manual QA had to happen AGAIN anyway

@simon Even then, I’d push for not having “a staging environment” and instead spinning up a new environment for each pull request. So instead of one staging environment, one per change. Does require the ability to spin up and tear down environments, though, and confidence that they won’t cost the earth.

@norm @simon @stevemarshall we do CD to staging with a promote to prod, but the process is that each individual change still gets released on its own - no batching

Works pretty well for us