Infrastructure as Code
IaC is used to manage infrastructure being deployed to the cloud. Infrastructure in the cloud is configurable.
Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using the same versioning as the DevOps team uses for source code. Like the principle that the same source code generates the same binary, an IaC model generates the same environment every time it is applied. IaC is a key DevOps practice and is used in conjunction with continuous delivery.
Infrastructure as code (IaC) tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. IaC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share.
Terraform is HashiCorp’s infrastructure as code tool. It lets you define resources and infrastructure in human-readable, declarative configuration files, and manages your infrastructure’s lifecycle. Using Terraform has several advantages over manually managing your infrastructure:
Terraform can manage infrastructure on multiple cloud platforms.
The human-readable configuration language helps you write infrastructure code quickly.
Terraform’s state allows you to track resource changes throughout your deployments.
You can commit your configurations to version control to safely collaborate on infrastructure.
Can upgrade code, infrastructure, or configuration separately. It uses the same process regardless of the change.
IaC solves real problems
Infrastructure as Code evolved to solve the problem of environment drift in the release pipeline. Without IaC, teams must maintain the settings of individual deployment environments. Over time, each environment becomes a snowflake, that is, a unique configuration that cannot be reproduced automatically. Inconsistency among environments leads to issues during deployments. With snowflakes, administration and maintenance of infrastructure involves manual processes which were hard to track and contribute to errors.
Manage any Infrastructure
Terraform plugins called providers let Terraform interact with cloud platforms and other services via their application programming interfaces (APIs). HashiCorp and the Terraform community have written over 1,000 providers to manage resources on Amazon Web Services (AWS), Azure, Google Cloud Platform (GCP), Kubernetes, Helm, GitHub, Splunk, and DataDog, just to name a few. You can find providers for many of the platforms and services you already use in the Terraform Registry.
Seagen has not created a new provider and is not anticipating doing so.
We currently use the Azure, Google, Databricks, and Oracle Cloud Infrastructure Providers. Start with the Seagen Terraform Registry to search for a provider or module you may want to use and one that is supported by Seagen DevOps.
Recommendation: When deploying a resource or module using Terraform, after deploying that or those resources, do not update the configuration manually as this will corrupt your state file.
Module Support
Modules are are packages of resources that have been preset to a specific configuration for one or more use cases at Seagen. Seagen has dozens of modules that are used for the various support providers. These modules are basic and give enough support to get started. Modules are used to help you quickly get an environment configured and deployed.
Recommendation: If a module fits, use it. If it does not fit, deploy individual resources instead. This allows you update individual resources. Seagen ETS will be removing the ability of teams to manually configure or manage resources in Azure and Google by the end of 2022.
What happens if a module does not contain as part of the provider package the parameters or configuration that you need?
Modules are self-service and DevOps would like the community at large to contribute back. When a module is updated using self-service, DevOps would like to make that module available to the community. When an update occurs, the Engineering Governance Committee will review and approve the change. This is in lieu of creation of a DevOps Community of Practice.
What happens if the module does not provide the parameters or configuration needed?
The DevOps Community of Practice will update the modules and contribute them back to the registry. Changing a module could break existing deployments. The GDCT team has created a process for managing versioning so we don’t break the existing IaC. As changes are made, the documentation will be updated by the person or team making the update. (resource required)
TODO: Need to assign someone to regularly review the Terraform modules and registry to capture the latest changes.
What happens when an update to the Seagen Registered Module is done?
The DevOps Community of Practice will review this update and commit it.
Terraform Resources
Resources are the individual infrastructure or services that will be deployed to the cloud. Whereas modules combine resources, resources live on their own. When new infrastructure or services are needed, your team will need to decide to either use a module with one to many resources or use individual resources.
Advantages to Using Resources
Fine grained control
Unique uses cases can be implemented without affecting others
Access to all the resource values
Highly customizable
Easier to debug
Can be contributed to the community
You own unique resource
Disadvantages to Using Resources
More variables and inputs to control
Need significant understanding to start
Modules provide templates to enable compliance
Need to understand cross resource dependencies
Less support than modules due to the customization
You own the unique resource
Still need to involve DevOps Community of Practice when bugs exist in Terraform
TODO: Define L1 and ongoing support for bugs introduced by individual contributors
Resources That are Not Allowed
There are some resources that are not allowed as defined by Enterprise Architecture and the Engineering Governance Committee. There are also some configurations of those resources that we do not allow to be configured by default.
Process for Making Changes
If a module or resource are not allowed today, a request can be made to the Engineering Governance Committee for review. If the EGC agrees with the change, they will take it to the Architecture Review Committee and update the enterprise standards.
When a module or resource is created, the code will be scanned and will not allow changes to the following;
Network changes
Authentication changes
Tokens or credentials in code
The DevOps Community of Practice will be responsible for reviewing new resources or modules and ensure the code is up to par.
TODO: work with Veeramani to stand up the DevOps Community of Practice. This COP will contain 2-3 reviewers who will review the new code. All reviews will be done within 4 hours of the request. This activity should be part of their annual review process to make sure they are keeping the standards updated, adding or updating standards. Reviewers change each year.
Standardize your deployment workflow
Terraform’s configuration language is declarative, meaning that it describes the desired end-state for your infrastructure, in contrast to procedural programming languages that require step-by-step instructions to perform tasks. Terraform providers automatically calculate dependencies between resources to create or destroy them in the correct order.
Idempotence is the property that a deployment command always sets the target environment into the same configuration, regardless of the environment’s starting state. Idempotence is achieved by either automatically configuring an existing target or by discarding the existing target and recreating a fresh environment.
NOTE: Please be aware that Terraform is designed to control drift in an environment. When a base configuration is deployed, Terraform creates a state.
What happens when I manually change the deployed configuration after deploying with Terraform?
When running Terraform again, Terraform will look at the resource and determine the state. If the state has changed by a manual change in the cloud, Terraform will overwrite the configured resource to restore its state. The developer will need to add the changes to the resources and redeploy or update those resources. This is because ETS wants developers to use automation to manage and create resources in the cloud and not do so via the cloud provider UI.
NOTE: When changes are made through automation, Terraform will update the resource appropriately or destroy and recreate as needed. To understand when something is updated or deleted and redeployed, use the Terraform registry for that provider.
**BUG: As of October 31, 2022, Seagen does permit making changes manually in the cloud provider UI. This will cause Terraform to error out in some scenarios. When this happens, the state file needs to be fixed. See the GDCT DevOps team for assistance.**
IaC delivers real benefits
Infrastructure as Code enables DevOps teams to test applications in production-like environments early in the development cycle. These teams expect to provision multiple test environments reliably and on demand. Infrastructure represented as code can also be validated and tested to prevent common deployment issues. At the same time, the cloud dynamically provisions and tears down environments based on IaC definitions.
Teams who implement IaC can deliver stable environments rapidly and at scale. Teams avoid manual configuration of environments and enforce consistency by representing the desired state of their environments via code. Infrastructure deployments with IaC are repeatable and prevent runtime issues caused by configuration drift or missing dependencies. DevOps teams can work together with a unified set of practices and tools to deliver applications and their supporting infrastructure rapidly, reliably, and at scale.
Prefer declarative definitions
The preferred approach to IaC is to use declarative definition files where possible. A definition file specifies the what an environment requires and not necessarily the how. In other words, it may define the specific version and configuration of a server component as a requirement, but does not specify the process for installing and configuring it. This abstraction allows for greater flexibility in the middle, such as optimized techniques the infrastructure provider may employ. It also helps reduce the technical debt of maintaining imperative code, such as deployment scripts, that can accrue over time.
There is no single standard syntax for declarative IaC. Different platforms support different, and often multiple, file formats, such as YAML, JSON, and XML. As a result, the decision to select a syntax for describing IaC usually comes down to the requirements of the target platform.
Using IaC on Azure
Azure provides native support for IaC via the Azure Resource Manager. Teams can define declarative templates that specify the infrastructure required to deploy their solutions.
Teams can also use the IaC supported by popular third party platforms, such as Terraform, Ansible, Chef, and Pulumi to manage automated infrastructure.
Vision
Seagen will utilize Terraform Cloud, GitHub Enterprise, and GitHub Actions to deliver Infrastructure to the cloud. There are some core tenants that Seagen DevOps follows.
All infrastructure should be delivered via pipelines and Terraform modules or resources
All code is promotable
Terraform Cloud is used for IaaS, PaaS, DBaas, FaaS, Serverless architectures
End state goal is zero downtime deployment with immutable infrastructure
To deploy infrastructure with Terraform:
Scope - Identify the infrastructure for your project.
Author - Write the configuration for your infrastructure.
Initialize - Install the plugins Terraform needs to manage the infrastructure.
Plan - Preview the changes Terraform will make to match your configuration.
Apply - Make the planned changes.
These steps are done in a GitHub Action.
Out of Scope
Deploying resources on-premises with Terraform Cloud is out of scope for the DevOps team. The primary reason is that Seagen is a cloud first company and we should evaluate the cloud options first. Even if a company has to deploy to VMs, those VMs should be deployed to the cloud if possible using the VM configuration and Integration Management Strategies as a guide.