4 Years of DevOps: Insights and Key Lessons Learned

If you seeking to transition to a DevOps role then these are my top 10 lessons for you.

These are hands-on skills, tried and battle-tested!

#1: Knowing DevOps is just a start

Knowing the fundamentals is not enough. You need to know the problems that DevOps tries to solve and how to sell that as a value proposition to stakeholders.

In most cases, stakeholders will be primarily concerned with how the DevOps practice impacts the company's bottom line.

For example, if you are advocating for the adoption of Terraform, you should be adept at explaining its direct contributions to the company's overarching goal: profitability. You should be able to articulate how Terraform accelerates the provisioning of environments and faster time-to-market for the business's products.

Similarly, costs will play a pivotal role in your decision-making as a DevOps practitioner. Deploying a CronJob that incurs thousands in runtime charges, even in the name of automation, can be counterproductive. In such cases, you will need to strike a balance between functionality and cost-effectiveness.

As a discipline, DevOps can be thought of as part culture shift and part tooling shift. The cultural shift can be quite challenging, especially in environments where speed is prioritized over quality. Most business cultures may be categorized as hurry cultures and fostering the values of a shared commitment to quality will not be easy. You have to prepare for a future where coordinating who must do what, why and how, is an exercise in effort and determination.

#2: Harness the Power of Custom Docker Images

You should make custom Docker images an integral part of your workflow. While repositories like Dockerhub offer functional images, often, technical requirements will often demand that you create customized images. This will entail anything from installing packages, multi-stage builds or overriding the runtime user of the image.

#3: Take care of your pipelines

In the world of DevOps, Continuous Integration and Continuous Delivery (CI/CD) pipelines serve as the backbone of success. They demand meticulous attention and expertise.

Your CI/CD pipelines will be subject to upgrade, security, storage and high-availability constraints. Performance is also an important constraint and according to this CircleCI report pipelines should take an average of ten minutes to complete.

#4: Measure as much as possible

If you are not measuring what you are doing you have no way of knowing what is working and what is not working. This means that you must be proficient at designing KPIs across every layer of your DevOps ecosystem.

Consider the following strategies as a good start :

DORA Metrics: Evaluate your DevOps workflows using DORA metrics.
Trace Statistics: Track performance across process boundaries.
Logging: Use varying debug levels.
SLOs, SLIs and SLAs: Adopt RED and USE metrics.
Load tests: Forecast what peak performance looks like.

#5: Documentation is King

Documentation is important because it serves as a beacon that transforms complex ideas into digestible knowledge. And when it comes to work-related documentation it is always a good practice to maintain living documents.

But the significance of documentation extends far beyond work-related files— it's a cornerstone practice for excellence. Great engineers maintain journals to capture their ideas and daily learnings. They use their journals to challenge assumptions and unlearn concepts.

Consider the following strategies as a good start:

Component URLs and Ports: Keep track of vital URLs and port information for your components.
Configuration File Paths: Document the file paths crucial for configuring your components.
Architectural Insights: Outline architectural and subsystem boundaries for clarity in your documents.
Visual Aids: Enhance comprehension with self-explanatory icons and images.
Journal: Document key takeaways for the day's work, if any.

#6: Embrace Init and Sidecar Containers

Containers by design run on exactly one process and this means that you will need ways to inject extra work into a pod. This is where init and sidecar containers will come in handy for you:

Init Containers: They do any "administrative" work that the main container process requires and then exit before the main container starts.

Sidecar Containers: In contrast, sidecar containers run alongside the main container.

#7: Do Kubernetes the Hard Way

When embarking on your journey to master Kubernetes, a hands-on approach will get you far.

My recommendation? Start by building on docker-compose. After that, you move to the K8s world. Build a K8s cluster from the ground up, complete with at least one worker node and master node. You should aim to deploy a range of workloads, including cronjobs, stateful, and stateless applications. This way you gain invaluable insights into the K8s core as well as the Kubernetes orchestration system as a whole.

#8: Version your Terraform plans

Cloud architectures can become very complex and sometimes bloated. However, there's a strategic approach to maintain control and clarity: version your Terraform plans.

Each commit in your Terraform project gains a new layer of insight when accompanied by a Terraform plan file. This file serves as a valuable reference point, offering a deeper perspective than a conventional commit message. Its significance becomes apparent when things go wrong and you are looking at several hundred commits in your project.

As a best practice, consider establishing a dedicated directory for Terraform plans at the root of your Terraform projects. You can also use the plans as build artifacts in your CI/CD pipeline.

Here's an example command to integrate a Terraform plan into your project:

terraform plan -no-color > ./plans/ADD_ALB_SECURITY_GROUP.txt

#9: Learn Bash Scripting

In DevOps, almost everything runs on Linux. This means that you will encounter (and write) Bash scripts a lot. To do all this a fundamental understanding of Bash scripting is a must-have.

The most common use cases you should know include mounting Bash scripts in containers and automating processes with Bash.

#10: Master the art of debugging!

You are as good a DevOps Engineer as your debugging prowess. The depth of your debugging skills will often determine your success. On the other hand, not all debugging tools are created equal, and knowing which one to use is a skill in itself.

Here's a list of debugging tools to elevate your debugging game:

Prometheus query issues - Promtool
Kubernetes DNS issues - Pragma
Helm chart issues - Yaml Lint
Bash script issues - Shellcheck

Summary

DevOps is a vast and ever-evolving discipline. The ideas presented here are not exhaustive and as a DevOps practitioner, it is important to adopt a continuous learning mindset.

What are some of the lessons you have discovered in your DevOps journey?

If you liked this article follow me on Twitter for more DevOps, Cloud & SRE.

Ed Tshuma's Blog