8 min read

What is DevOps?

Amazon:

"DevOps is the combination of cultural philosophies, practices, and tools that increases an organization's ability to deliver applications and services at high velocity: evolving and improving products at a faster pace that organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market."

Microsoft:

"DevOps is the union of people, process, and products to enable continuous delivery of value to our end users. Th contraction of "Dev" and "Ops" refers to replacing siloed Development and Operations to create multidisciplinary teams that now work together with shared and efficient practices and tools. Essential DevOps practices include agile planning, continuous integration, continuous delivery, and monitoring of applications."

Red Hat:

"DevOps describes approaches to speeding up the processes by which an idea (like a new software feature, a request for enhancement, or a bug fix) goes from development to deployment in a production environment where it can provide value to the user. These approaches require that development teams and operations teams communicate frequently and approach their work with empathy for their teammates. Scalability and flexible provisioning are also necessary. With DevOps, those that need power the most, get it - through self service and automation. Developers, usually coding in a standard development environment, work closely with IT operations to speed software builds, tests, and releases - without sacrificing reliability."

Google:

"The organizational and cultural movement that aims to increase software delivery and velocity, improve service reliability, and build shared ownership among software stakeholders"

What are the benefits of DevOps? What can it help us to achieve?

  • Collaboration
  • Improved delivery
  • Security
  • Speed
  • Scale
  • Reliability

What are the anti-patterns of DevOps?

  • One person is in charge of different tasks. For example, there is only one person who is allowed to merge the code of everyone else into repository.
  • Treating production differently from development environment. For example, not implementing security in development environment.
  • Not allowing someone to push to production on Friday ;)

What is continuous integration?

A development practice where developers integrate code into a shared repository frequently. It can range from a couple of changes every day or a week to a couple of changes in one hour in larger scales.

Each piece of code (change/patch) is verified, to make the change is safe to merge. Today, it's a common practice to test the change using an automated build that makes sure the code can integrated. It can be one build which runs several tests n different levels (unit, functional, etc.) or several separate builds that all or some has to pass in order for the change to be merged into the repository.

Can you describe an example of a CI/CD process starting the moment a developer submitted a change/PR to a repository?

There are many answers for such question as CI processes vary depending on the technologies used and the type of the project to where the change was submitted. Such processes can include one or more of the following stages:

  • Compile
  • Build
  • Install
  • Configure
  • Update
  • Test

For example:

A developer submitted a pull request to a project. The PR (pull request) triggered two jobs (or one combined job). One job for running lint test on the change and the second job for building a package which includes the submitted change, and running multiple api/scenario tests usuing that package. Once all tests passed and the change was approved by a maintainer/core, it's merged/pushed to the repository. If some of the tests failed, the change will not be allowed to merged/pushed to the repository.

A complete different CI process can describe how a developer pushes code to a repository, a workflow then triggered to build a container image and push it to the registry. Once in the registry, the k8s cluster is applied with the new changes.

What is Continuous Deployment?

A development strategy used by developers to release software automatically into production where any code commit must pass through an automated testing phase. Only when this is successful is the release considered production worthy. This eliminates any human interaction and should be implemented only after production-ready pipelines have been set with real-time monitoring and reporting of deployed assets. If any issues are detected in production it should be easy to rollback to previous working state.

What is Continuous Delivery?

A development strategy used to frequently deliver code to QA and Ops for testing. This entails having a staging area that has production like features where changes can only be accepted for production after a manual review. Because of this human entaglement there is usually a time lag between release and review making it slower and error prone as compared to continuous deployment.

Would you prefer a "configuration->deployment" model or "deployment-configuration"? Why?

Both have advantages and disadvantages. With "configuration->deployment" model for example, where you build one image to be used by multiple deployments, there is less chance of deployments being different from one another, so it has a clear advantage of a consistent environment.

What CI/CD best practices are you familiar with? Or what do you consider as CI/CD best practice?

  • Automated process of building, testing and deploying software
  • Commit and test often
  • Testing/Staging environment should be a clone of production environment

Where do you store CI/CD pipelines? Why?

There are multiple approaches as to where to store the CI/CD pipeline definitions:

  1. App Repository - store them in the same repository of the application they are building or testing
  2. Central Repository - store all organization's/project's CI/CD pipelines in one separate repository
  3. CI repo for every app repo - you separate CI related code from app code but you don't put everything in one place

What systems and/or tools are you using for the following areas/tasks? Why?

  • CI/CD
  • Privisioning infrastructure
  • Configuration management
  • Monitoring & alerting
  • Logging
  • Code review
  • Code coverage
  • Issue Tracking
  • Containers and Containers Orchestration
  • Test

What are you taking into consideration when choosing a tool/technology?

In your answer you can mention one or more of the following:

  • mature/stable vs. cutting edge
  • community size
  • architecture aspects - agent vs. agentless, master vs. masterless, etc.
  • learning curve

Explain mutable vs. immutable implementation

In mutable infrastructure paradigm, changes are applied on top of the existing infrastructure and over time the infrastructure builds up a history of changes. Ansible, Puppet and Chef are examples of tools which follow mutable infrastructure paradigm.

In immutable infrastructure paradigm, every change is actually a new infrastructure. So a change to a server will result in a new server instead of updating it. Terraform is an example of technology which follows the immutable infrastructure paradigm.

Explain "Software Distribution"

From popular article: "Thus, software ditribution is about the mechanism and the community that takes the burden and decisions to build an assemblage of coherent software that can be shipped."

Why are there multiple software distributions? What differences can they have?

Different distributions can focus on different things like: focus on different envrionments (server vs. mobile vs. desktop), support specific hardware, specialize in different domains (security, multimedia, ...), etc. Basically, different aspects of the software and what it supports, get different priority in each distribution.

What is Software Repository?

(Wikipedia) "A software repository, or "repo" for short, is a storage location for software packages. Often a table of contents is stored, as well as metadata."

What ways are there to distribute software? What are the advantages and disadvantages of each method?

  • Source - Maintain build script within version control system so that user can build your app after cloning repository. Advantage: User can quickly checkout different versions of application. Disadvantage: requires build tools installed on users machine.
  • Archive - collect all your app files into one archive (e.g. tar) and diliver it to the user. Advantage: User gets everying he needs in one file. Disadvantage: Requires repeating the same procedure when updating, not good if there are a lot of dependencies.
  • Package - depends on the OS, you can use yoru OS package format (e.g. in RHEL/Fefodra it's RPM) to deliver your software with a way to install, uninstall and update it using the standard pacakager commands. Advantages: Package manager takes care of support for installation, uninstallation, updating and dependency management. Disadvantage: Requires managing package repository.
  • Images - Either VM or container images where your package is included with everything it needs in order to run successfully. Advantage: everything is preinstalled, it has high degree of environment isolation. Disadvantage: Requires knowledge of building and optimizing images.

Are you familiar with "The Cathedral and the Bazaar models"? Explain each of the models

  • Cathedral - source code released when software is released
  • Bazaar - source code is always available publicly (e.g. Linux Kernel)

What is caching? How does it work? Why is it important?

Caching is fast access to frequently used resources which are computationally expensive or IO intensive and do not change often. There can be several layers of cache that can start from CPU caches to distributed cache systems. Common ones are in memory caching and distributed caching. Caches are typically data structures that contains some data, such as a hashtable or dictionary. However, any data structure can provide caching capabilities, like set, sorted set, sorted dictionary etc. While, caching is used in many applications, they can create subtle bugs if not implemented correctly or used correctly. For example,cache invalidation, expiration or updating is usually quite challenging and hard.

Explain stateless vs. stateful

Stateless applications don't store any data in the host which makes it ideal for horizontal scaling and microservices. Stateful applications depend ot the storage to save state and data, typically databases are stateful applications.

What is Reliability? How does it fit DevOps?

Reliability, when used in DevOps context, is the ability of a system to recover from infrastructure failure or disruption. Part of it is also being able to scale based on your organization or team demands.

What types of tests are yuo familiar with?

Styling, unit, functional, API, integration, smoke, scenario, ...

You need to install periodically a package (unless it's already exists) on different operating systems (Ubuntu, RHEL, ...). How would you do it?

There are multiple ways to answer this questions (there is no right and wrong here):

  • Simple cron job
  • Pipeline with configuration management technology (such as Puppet, Ansible, Chef, etc.) ...

What are MTTF (mean time failure) and MTTR (mean time to repair)? What these metrics help us to evaluate?

  • MTTF (mean time to failure) other known as uptime, can be defined as how long the system run before if fails.
  • MTTR (mean time to recover) on the other hand, is the amount of time it takes to repair a system.
  • MTBF (mean time between failues) is the amount of time between failures of the system. These errors can be intermittent or fatal.

What is "infrastructure as code"? What implementation of IAC are you familiar with?

IAC (infrastructure as code) is a declarative approach of defining infrastructure or architecture of a system. Some implementations are ARM templates for Azure and Terraform that can work across multiple cloud providers.

How do you manage build artifacts?

Build artifacts are usually stored in a repository. They can be used in release pipelines for deployment purposes. Usually there is retention period on the build artifacts.

What deployment strategies are you familiar with or have used?

There are several deployment strategies:

  • Rolling
  • Blue green deployment
  • Canary releases
  • Recreate strategy

What is configuration drift? What problems is it causing?

Configuration drift happens when in an environment of servers with the exact same configuration and software, a certain server or servers are being applied with updates or configuration which other servers don't get and over time these servers become silently different than all others.

This situation might lead to bugs which hard to identify and reproduce.

How to deal with a configuration drift?

Configuration drift can be avoided with desired state configuration (DSC) implementation. Desired state configuration can be a declarative file that defined how a system should be. There are tools to enforce desired state such as a terraform or azure dsc. There are incremental or complete strategies.

Do you have experience with testing cross-proejct changes? (aka cross-dependency)

Note: cross-dependency is when you have two or more changes to separate projects and you would like to test them in mutual build instead of testing each change separately.

SRE

What SRE team is responsible for?

One can argue whether it's per company definition or a global one but at least according to a large companies, like Google for example, the SRE team is responsible for availability, latency, performance, efficiency, change management, monitoring, emerging response, and capacity planning of their services.

Orestis Pantazos

Orestis Pantazos

DevOps Engineer