DevOps from the Infrastructure Engineer's Point-Of-View
07/04/2017 Industry insights
I read a lot of blog articles which tell me what DevOps is from a Vendor trying to sell a product, or from an evangelist who works for a particular company. A lot of people say it’s about culture and attitude, and while this is certainly true it’s by no means the whole story. You would not employ a plumber with the right attitude. Here is a list of what I consider the ingredients needed to really be doing the DevOps. You don’t have to do all of them, but if you should miss any out then you should have a damned good reason why, and not just miss them out because they are difficult - I’m looking at you automated acceptance tests.
Really it amounts to being able to deliver value to the customer quickly. This does not mean 3 months or more. To achieve that you need to automate “all the things”. You also need to keep up with a tonne of internet memes, so that you can laugh derisively at your less clued up colleagues.
Or in plain English, every time someone checks in code, a build should be kicked off. This build should stand up a fully working and testable version of the system.
Useful tools: Jenkins, Bamboo, Codeshop, Travis CI, TeamCity…
A large proportion of the tests should be automated, but you should still have human testers perform the more difficult (and interesting) exploratory testing.
Useful tools: Selenium.
Once you are happy that the build is good you should be able to select a version and target environment and deploy to it. You should ONLY have to select those two things and the rest should just happen.
Useful tools: Jenkins, Go CD, Bamboo, …
If you are using immutable infrastructure this might not be a concern anymore, or you may find that you only run the CM tool once at image build time. With immutable, you regularly recycle your instances. CM tools will use an “idempotent” and platform-agnostic method, to enforce the configuration onto your servers. This includes users, files, packages, services, etc.
Useful tools: Puppet, Chef, Ansible, Salt, CFEngine.
Provision your infrastructure with a scripted service. This is a real no-brainer, and it doesn’t even have to be a cloud infrastructure, although that’s where it makes the most sense. You can achieve this with Red Hat Satellite for instance or by using VMWare vCloud Orchestrator.
Useful tools: Terraform, Vagrant, Docker.
Monitoring of Apps
You need to know when things are going wrong with the app, and this should be automatically built into your monitoring system as soon as you stand up infrastructure. There are too many tools to mention here, and you’ll probably have your own in place already.
Monitoring of Infrastructure
Same as for apps, but you need to know if the servers or containers themselves are running.
Limit Work in Progress
Don’t take on new tasks when you are already overloaded. Keep a backlog. Groom tasks. Plan your work in detail as late as possible as things will inevitably change if you plan too far in advance.
Useful methods: Agile, Scrum, Kanban. Useful tool: Jira and others.
Everything must be checked into source control. You should probably work on a branch, or have a branching strategy. (Short lived!) For most people, this goes without saying, but I still meet Sysadmins who don’t really get this. You need to understand branching, merging and tagging. The book “Git for Teams” is really useful. Git flow as a process might be too heavy if you are running a smallish team, but for anything larger than a handful of developers it’s something to consider.
Useful tool: git. (there are others apparently, but mostly git).
Work in small batches
Most people split up work into a two week or one week sprint and try to have a working demo to show at the end of it. You might not be able to achieve a huge amount in one week, but you should be able to show it working. You should spend a little time at the start planning your sprint at the start, and every day, have a very quick “stand up” to discuss what you are working on, and if anything is holding you up.
Security built into the delivery
As part of your build process, you should perform security scanning, static code analysis, etc. This is a fairly new and evolving idea and some are calling it DevSecOps (because we need more buzzwords).
This comes from Extreme Programming (XP) and you should tidy up any code which “smells bad” as you are working until you have the best code you can achieve. All of the above (build and test) helps with this - as if you break something, you’ll know about it quickly.
If something is going to fail, you want it to fail as quickly as possible. Ideally, as your developer is typing (in the IDE) or as soon as they check in code. Git pre-commit hooks can help with this on the client side and post-receive hooks on the master repo. As well as avoiding breaking things, you should enforce code standards and test coverage (unit tests).
This really depends on the organisation you are working for, and the size of the project as well as the individual needs. This is a recipe for DevOps, but you can leave out some of the flavours if they leave a bad taste. Automate the change approval process, reduce technical debt, get feedback from perf and acceptance tests. Also, if you want to try out an idea create a Minimal Viable Product (MVP) which is the quickest and smallest way of testing out that idea. Produce metrics from logs which are useful to the business, and give them easy ways to visualise them. Look at immutable architecture. Containers. Microservices. Container orchestration. Functions as a Service, such as AWS Lambda look really interesting.
As you can see, it’s a rapidly changing landscape, and part of the job is to keep up with the latest trends as some great ideas are being tested by companies large and small. It’s sometimes difficult to separate the wheat from the chaff, as there are a lot of experiments being carried out some of which are going to turn out to be dead ends.
If you are still building servers by hand, and installing the software following a process on a 27 page MS Word document then you are part of a dying breed. DevOps is becoming the new norm, and companies which continue to use the slow and unreliable methods of the past will struggle to compete and will need to adapt or end up going out of business.
About Steve Button
Steve is a highly experienced DevOps Engineer/ Infrastructure Developer, specialising in Linux, Ruby, Puppet, Terraform and CI / CD.