Menu
Schedule a Call

DevOps Engineer: The Hottest Tech Job of 2020

Leroy Ware
Nov 3, 2020 3:52:05 AM

According to The Linux Foundation's and edX's most recent Open Source Jobs Report, the hottest job of 2020 is DevOps Engineer. Despite the COVID pandemic and the resulting economic slowdown, 65% of hiring managers are looking to hire more DevOps talent, up from 59% in 2018, and 70% are looking to hire engineers with Cloud expertise. This is important news for engineers who are interested in picking up new skills that will make them more attractive to hiring managers and increase their earning potential.

So what is DevOps, exactly? As the word implies, DevOps is a blend of  historically separate areas of expertise. It emerged in an attempt to “bridge the gap” or “remove silos between” software development (“Dev”) and IT operations (“Ops”), with the goal of releasing software faster and with greater reliability.

Most DevOps engineers started their careers on one side of the spectrum or the other: (a) they were software developers who gradually expanded their knowledge of IT operations, or (b) they were IT operations professionals who gradually expanded their knowledge of software development. Being a DevOps Engineer is akin to being ambidextrous, capable of shooting baskets with either hand, or being a boxer who can spar in both the orthodox and southpaw stance. It's a very broad skillset that usually requires years to acquire, but the reward is tremendous and those who do it well know that their career is future proof -- and maybe even apocalypse proof!

In the rest of this article, we'll take a look at some of the skills you should consider learning if you're interested in becoming a DevOps Engineer.

Fullstack Engineering

Whatever side of the DevOps skill spectrum you're starting out on, one of the first things you should do is broaden your coding skills. If you're a frontend developer, expand to backend. If you're a backend developer, expand to frontend. If you come from the IT Operations/Infrastructure side, take online courses to start beefing up your knowledge of programming concepts and web development. Here are a few good online resources for ramping up quickly.

Instead of diving deeper into your existing area of expertise, go broader. Generalists with a broad knowledge of software development will find it easier to evolve into DevOps Engineers and will also find themselves in higher demand now and in the future than specialists with a narrow area of expertise.

Virtualization

DevOps engineers should have a good historical and functional understanding of virtualization, which is one of the biggest things to happen in computing in the last couple decades.

Hypervisors & VMs

In the old days, every server was a physical box that ran an operating system (e.g., Windows Server). You got the box, installed the OS, and you were off and running. As server hardware became more powerful, it didn't take long for the "hypervisor" to make its debut,  which allows a single physical server could be logically split into multiple "virtual machines" (VMs), each of which can run its own operating system and operate independently and in isolation from other VMs running on the same physical machine. The first release of an open-source hypervisor was Xen in 2003. 

Managing VMs via a hypervisor was a step up from managing physical servers, but it still left a lot to be desired. It wasn't easy to fully automate the creation, scaling, and destruction of VMs and associated resources. These challenges eventually led to a new approach to infrastructure management called "software defined infrastructure" (SDI) or "infrastructure as code" (IaC). Some of the earliest efforts at SDI include Chef, Puppet, Ansible, and similar tools. The true disruption didn't happen, however, until a new paradigm called "containerization" emerged with the release of Docker in 2013.

Containers

Containerization is like VMs inside of VMs. It allows you to chop a VM into even smaller units called "containers", each of which can run its own operating system and can even engage in networking across their "host" VM. What is more noteworthy is that you can completely define a "container" using a single file called a Dockerfile, including the operating system, packages to be installed, ports that will be open, and any other configuration required by a "server" to operate. With Docker, a complete description of the server environment required by an application could reside in the same source repository as the application itself, which gives software developers more transparency into how their application will be deployed.

Containers were just as much of an improvement over VMs as VMs were over "bare metal", but configuring and running containers -- especially in production -- can get hairy. Even though the entire environment can be defined in a single file, the amount of server knowledge required to create Dockerfiles is still a high barrier to entry for most, and as soon as you introduce containers into a production environment, all sorts of other complex tasks emerge, including container orchestration and cluster scaling.

Serverless

The next evolutionary step of virtualization was serverless or "Functions as a Service" (FaaS). In 2014, Amazon introduced AWS Lambda, making it the first public cloud infrastructure vendor with an abstract serverless computing offering. AWS Lambda was quickly followed by Google Cloud Functions, IBM Cloud Functions, and Azure Functions, and ultimately there were on-premise FaaS platforms such as Kubeless that made it possible to create serverless applications that run on-premise rather than in the public cloud.

Serverless completely abstracts away all details about the server environment, allowing engineers to adopt an "event-oriented" approach to software development. The only thing that matters in serverless development is the events that trigger functions, and how those events will be processed. This paradigm more accurately reflects the way we think about software and even does a better job of imitating the way things happen in the universe physically and philosophically. Serverless computing is still in its early infancy, and it is common to see containerization and serverless computing mixed together in modern production deployments -- especially when applications leverage open-source solutions.

Infrastructure as Code (IaC)

Modern application deployments -- whether they use VMs, containers, or the serverless approach -- are generally done via Infrastructure as Code (IaC). It is rare to see successful companies these days who still have system administrators or IT Operations folks manually managing infrastructure using the command line or a GUI.  There are a lot of ways to do IaC, and which approach you use will depend on many factors. 

A few things worth familiarizing yourself with in this area are:

  • Kubernetes - Introduced by Google in 2014. This is a container orchestration system that makes it possible to deploy and manage clusters of containers using config files.
  • Docker Compose - This is the native method for managing Docker both locally and in a cluster configuration using Docker Swarm
  • AWS CDK - This is Amazon's newest approach to describing AWS-deployed infrastructure and is extremely powerful. Developers can use languages they are already familiar with, such as TypeScript or Python, to completely describe the AWS resources required by their application. 
  • Terraform - This is a cross-cloud IaC tool that makes it possible to automate infrastructure deployments across almost all public cloud platforms, including AWS, Google Cloud Platform, IBM, and Microsoft Azure.
  • The Serverless Framework - This is my favorite of all IaC tools. Like Terraform, it is cross-cloud. It is also cross-language, allowing developers to deploy serverless applications in Node.js, Python, and other languages across almost any public cloud platform. It also supports on-premise deployments using Kubeless.

As an fullstack Javascript engineer who has been in a long-term love affair with AWS, I prefer the TypeScript AWS CDK and Serverless Node.js. When a project requires containerization, I use Docker on ECS and the AWS CDK. Otherwise, I use the Serverless Framework. Sometimes I use them both. 

CI/CD

CI/CD has also evolved over the years and can now be fully defined via software. Most modern CI/CD tools allow pipelines to be described using a simple YAML syntax, so YAML is an important (and simple) skill to pick up if you're intent on becoming a DevOps Engineer. A few of the best CI/CD tools in 2020 are:

  • Github Actions - Github launched its CI/CD service in 2019, and it is pretty amazing. It reminds me of NPM in that developers can create "actions" that perform specific, common deployment tasks and publish them to a marketplace, which can then be used by other DevOps Engineers. 
  • AWS CodePipeline - AWS CodePipeline is a nice solution offered by Amazon that also uses YAML to describe AWS deployments. It integrates well with Github or Amazon's own source-code solution, CodeCommit, and can automate highly complex deployment tasks on AWS. 
  • Travis - Travis has been around forever and is one of the most widely used CI/CD platforms in the world, right up there with Jenkins. Travis supports a simple YAML syntax and can be used to automate everything from Docker image builds and web app deployments to mobile builds. 
  • Seed - Seed is a new kid on the block, but it is pretty special. It was one of the first CI/CD platforms specifically designed for Serverless Framework deployments, and it works wonderfully. It requires no special configuration beyond the serverless.yml file required by the Serverless Framework itself, and in that sense it is one of the easiest ways to deploy applications built using the Serverless Framework. 
  • Serverless CI/CD - The Serverless team also offers its own CI/CD platform, which is also nice.

There are many other popular CI/CD services, including CircleCI, Drone.io, and Shippable. Of course, Microsoft has Azure Pipelines, Google has Cloud Build, and on and on. Too many to cover in a single article. The ones above are some of my favorites, and arguably some of the best ones to learn in 2020. The most important thing is to learn YAML and experiment around with a couple CI/CD platforms that use it. You'll soon discover that most of them share similarities in YAML syntax, so once you've learned a couple of them, you can pick others up in a matter of days.

Databases

In the olden days, there were DBAs on almost every corner. These days, however, it's quite common to see teams without a DBA at all. Most modern applications use open-source databases like MySQL, PostgreSQL, or MongoDB, which are much easier to manage than the DBMSs of yore, like MS SQL Server and Oracle, and so the task of managing databases often falls to software developers or, more commonly, DevOps Engineers these days. Thus, it's critical that you learn more about databases.

SQL Databases

Databases come in two major flavors: SQL and NoSQL. SQL databases go back to the 1970s when storage was expensive and most of our thinking about data storage was tabular. SQL databases store data in "tables", where each row is a record and each column is a field. Fields in a table can have relationships to fields in other tables. When you query a SQL database, you can "join" these tables together on the fly to create the results you need for your application. The query language you use to fetch or modify data is called "Structured Query Language" (SQL). The most popular SQL databases used in modern application development are MySQL and PostgreSQL. 

NoSQL Databases

NoSQL is a very different and much newer approach. Data is stored in the form of objects, each of which is a collection of key/value pairs. These data objects more closely reflect the way data is consumed by modern web and mobile applications. Most modern applications involve a "frontend user interface" making requests to a "backend API", which returns data in JSON format. If a SQL database is used, it is necessary to transform the table-like structure of the data into JSON format before sending it back to the frontend. NoSQL databases eliminate this transformation step by storing data in a format that more closely resembles the way it will be sent back to the frontend. Some of the most popular NoSQL databases in use today are MongoDB, ElasticSearch, and DynamoDB. Unlike with SQL (or "relational") databases, there is no standard query language for NoSQL.

ORMs

A common approach when using SQL databases in modern applications is to abstract away their table-like nature by using an object relationship mapper (ORM) such as Bookshelf, Sequelize, or Hibernate. This allows software developers to think about data stored in a SQL database in much the same way they would think about it if it were stored in a NoSQL database: as a collection of objects rather than a collection rows and columns.

As with all things, there are pros and cons to using SQL versus NoSQL, and there are reasons to use one over the other in particular situations. It's important that you gain some exposure to both if you aspire to be a DevOps Engineer. Below are a few resources for learning more about SQL, NoSQL, and ORMs.

The Cloud

No discussion about DevOps would be complete without some discussion about "the cloud". When most people think of "the cloud", they think of Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, or IBM Cloud. These are the 800-pound gorilla public cloud vendors that have risen to dominance over the last decade as companies try to reduce the cost and complexity of managing infrastructure by moving it offsite. However, "cloud" technology started way back with VMs and hypervisors, which allowed companies to launch "private clouds" on-premise. VMware, Red Hat, and many other vendors are still big players in the private cloud space with large companies that prefer to keep their infrastructure on-premise. So which should you learn, public or private cloud? And if you focus on the public cloud, which vendor will present the most job opportunities and have the greatest potential for increasing your income?

According to a recent report, "Amazon Web Services (AWS) was the leading cloud service provider in Q2 2020, accounting for 31% share of total spend. Microsoft Azure reached 20% market share for the first time after a strong end to its fiscal year. Google Cloud remained the third largest cloud service provider with 6%, after exceeding the US $2 billion mark, followed by Alibaba Cloud with 5%." Thus, learning AWS will give you a strong advantage as a DevOps Engineer. If you come from a Microsoft background, it makes sense to learn Azure as well. There are many online resources available for learning AWS, and Amazon offers many of its own certifications that are widely recognized throughout the world. If you combine AWS certifications with a strong knowledge of the AWS CDK, you'll really be a force to reckon with, and if you go one step further and add some Serverless Framework experience to the mix, I daresay you'll be unstoppable -- by COVID or any other dangers this universe throws your way.

Below are a few resources for learning more about devops and cloud engineering:

Conclusion

DevOps and Cloud Engineering are two of the most sought after skills in 2020. As you think about making your next job move, challenge yourself by seeking a role that will broaden your skills. If there is an opportunity within your current company to move to other roles that will help you learn some of the skills describe above, jump! The age of specialization in software engineering is rapidly coming to an end. The highest paid engineers and the ones who have the most freedom and control over their lives five years from now will be those who work hard to broaden their skills today.

You May Also Like

These Stories on Engineering

Subscribe To Our Magazine

Subscribe