Taking DevSecOps to the Next Level: Automating DevOps Tools with Python API for Developer Experience
In DevOps we blend software development, security and IT operations to shorten the development lifecycle and to provide continuous delivery with high software quality. As we seek to further enhance these processes, automating DevOps tools through APIs is a powerful strategy. The integration of automation tools plays a pivotal role both in enhancing security, operational efficiencies, and improving developer satisfaction.
- Arunkumar Soundar
My name is Arunkumar Soundar, and I am Tech Lead for the OB Platform team in DNB. In this article, I would like to share our experience creating a Python API using FastAPI, designed to interface seamlessly with prominent DevOps tools such as GitLab, AWS, Dynatrace, and ServiceNow. We'll explore the advantages of using FastAPI, how it can be employed to automate interactions with these tools, and the impact on the developer experience.
This article will not only serve as a how-to guide but also inspire innovative uses of FastAPI in the DevOps realm. By creating APIs, you can also consume them to build an Internal Developer platform or integrate with other systems to enhance the developer experience.
The Role of Automation in DevOps:
Automation is fundamental to DevOps, facilitating faster and more reliable software development cycles by streamlining workflows, enhancing quality, and improving operational efficiency.
Why Python for DevOps?
Python is a popular choice for DevOps due to its simplicity, flexibility, and robust ecosystem.
Key benefits include:
- Ease of Learning and Use: Python's clear syntax makes it accessible for beginners and efficient for experienced programmers.
- Rich Libraries and Frameworks: Extensive libraries such as Ansible for automation, Django for web development, and Pandas for data analysis support diverse DevOps tasks.
- Wide Community Support: A large community provides extensive resources, shared knowledge, and troubleshooting help.
- Integration Friendly: Python easily integrates with other systems and tools, facilitating automation across technologies.
- Scripting and Automation: Python's capabilities make it ideal for writing scripts and automating workflows, essential for efficient DevOps processes.
Python's versatility and comprehensive toolset make it a powerful ally in implementing and streamlining DevOps practices.
Let's start by building a 'Hello World' API to understand the basics:
Fast API – A modern, fast web framework for building APIs with Python
Installation:
Install FastAPI and uvicorn using pip:
pip install fastapi uvicorn
Creating a Simple API:
Explanation:
1. Import FastAPI module.
2. Create an instance of FastAPI.
3. Define a route with a decorator.
4. Create a function that returns a dictionary automatically converted to JSON.
Running the API:
Run the API using uvicorn:
uvicorn myapi:app --reload
Access your API at http://127.0.0.1:8000/
Testing Your API:
Test your API using a web browser or curl:
curl http://127.0.0.1:8000/
http://127.0.0.1:8000/docs
We've seen a basic program using Python's FastAPI. Now let's build a real-world use case where the Python API integrates with GitLab, AWS, Dynatrace, and ServiceNow.
Project Structure:
my_fastapi_app/
│
├── app/ # Main application package
│ ├── main.py # FastAPI app creation and configuration
│ ├── dependencies.py # Dependencies for routes (e.g., API keys, configs)
│ ├── routers/ # Different routers for each service
│ │ ├── gitlab.py # Router for GitLab API endpoints
│ │ ├── dynatrace.py # Router for Dynatrace API endpoints
│ │ ├── aws.py # Router for AWS API endpoints
│ │ └── serviceNow.py # Router for another serviceNow API endpoints
│ │
│ ├── models/ # Pydantic models for request and response data
│ │ ├── gitlab_models.py # Models specific to GitLab
│ │ ├── dynatrace_models.py # Models specific to Dynatrace
│ │ ├── aws_models.py # Models specific to AWS
│ │ └── serviceNow_models.py # Models for another serviceNow
│ │
│ └── utils/ # Utility functions and classes
│ ├── gitlab_utils.py # Utilities for interacting with GitLab API
│ ├── dynatrace_utils.py # Utilities for interacting with Dynatrace API
│ ├── aws_utils.py # Utilities for interacting with AWS API
│ └── serviceNow_utils.py # Utilities for another serviceNow
│
├── config/ # Configuration files or environment management
│ ├── settings.py # Settings management using pydantic BaseSettings
│
└── requirements.txt # Project dependencies
Components Description
- main.py: This is the entry point of your FastAPI app where you configure and instantiate your application. You also include middleware, exception handlers, and mount your routers here.
- dependencies.py: Here, you define any dependencies your routes might need (e.g., database connections, API tokens, configuration settings).
- routers/: Each service (GitLab, Dynatrace, AWS, and your other service) has its own router file. This helps in organizing endpoints related to each service neatly and makes them easy to manage.
- models/: Define Pydantic models for your API. These models will help in validating the data coming into your endpoints and formatting data going out to clients.
- utils/: Utility modules contain functions and classes that help interact with each specific API. This might include functions to handle API requests, process data, etc.
- config/settings.py: Manages configuration and settings, potentially using environment variables or config files to manage sensitive information like API keys securely.
Sample FastAPI App Initialization (main.py)
- main.py: Initializes the FastAPI application and includes the routers for each service.
dependencies.py
- dependencies.py: Defines a dependency that checks for a valid token in the headers of incoming requests.
Routers (gitlab.py)
routers (e.g., gitlab.py): Defines API endpoints related to each service. In this example, we have a simple endpoint to list GitLab projects
Models (gitlab_models.py)
- models (e.g., gitlab_models.py): Defines data models using Pydantic which are used to validate data.
Utils (gitlab_utils.py)
- utils (e.g., gitlab_utils.py): Contains utility functions, such as fetching data from an external API. This could include making HTTP requests.
Config (settings.py)
- settings.py: Uses Pydantic's BaseSettings to manage configurations and environment variables, making the application easier to configure.
requirements.txt
- requirements.txt: Lists all the dependencies needed to run the application.
But we are not going to use this project structure now instead we will do the logic in the gitlab.py.
Here’s a breakdown of each import and its typical use case:
- from fastapi import FastAPI, HTTPException: FastAPI: Creates and manages the web application.
- HTTPException: Allows handling specific HTTP errors within FastAPI. from pydantic import BaseModel:
- BaseModel: Used for data validation and settings management in FastAPI.
- import gitlab:Accesses GitLab's API for repository and project management.
- import os: Provides a way of using functionality that is dependent on the operating system, like reading environment variables and handling files.
- import logging: Used for tracking events that happen when some software runs, especially useful for debugging.
- import shutil: Offers high-level file operations such as copying and deleting files.
- import subprocess: Used for running external commands and processes from Python.
These imports collectively enable the development of a FastAPI application that can interact with web services, handle data securely, manage files, and execute system-level commands.
The ProjectCreate class is a Pydantic model used to validate and manage data for creating new projects. Here's a quick overview of its fields:
- product_name: str - The name of the product.
- template_project_id: int - ID for a template project used as a base.
- aws_profile_dev: str - AWS profile name for development.
- gitlab_runner_tag: str - Tag for specifying the GitLab CI/CD runner.
- gitlab_group: str - GitLab group name where the project is located.
This model ensures that all necessary project creation data is provided correctly and conforms to expected types.
The FastAPI endpoint defined by @app.get("/api/v1/gitlab/list_templates", ...) retrieves a list of boilerplate templates from a GitLab group. Here's a concise overview:
Endpoint Details
- URL: /api/v1/gitlab/list_templates
- Method: GET
- Response Model: Expects a list of Project models, each containing project data.
- Tags: Categorized under "GITLAB" for API documentation organization.
- Summary: Provides a brief description, "List the boilerplate templates".
Function: list_gitlab_templates()
- Purpose: Connects to GitLab using the API and lists all projects within a specific group identified by template_group_id (16016 in this case).
- Process:
- Initializes a gitlab.Gitlab instance with credentials sourced from environment variables (GITLAB_URL and GITLAB_PRIVATE_TOKEN).
- Retrieves a group by its ID and lists all projects within this group.
- Formats the projects into a list of dictionaries, each containing a project's name and id.
- Error Handling: Captures and logs exceptions, raises an HTTP 500 error if the process fails, indicating an "Error listing templates".
This endpoint is useful for applications needing to display or use boilerplate templates from a specific GitLab group programmatically.
The FastAPI endpoint @app.post ("/api/v1/gitlab/create_project", ...) is designed to create a new CI & CD project on GitLab based on specified template details. Here's a quick overview:
Endpoint Details
- URL: /api/v1/gitlab/create_project
- Method: POST
- Response Model: Expects a list of Project models (presumably returning the newly created project details).
- Tags: Categorized under "GITLAB" for API documentation organization.
- Summary: "Create a CI & CD project".
Function: create_cd(product_create: ProductCreate)
- Purpose: Takes project creation parameters via a ProductCreate object and uses them to create a new CI & CD project in GitLab.
- Process:
- Initializes a connection to GitLab using credentials from environment variables.
- Logs the start of the process and the attempt to create a new project.
- Constructs a dictionary with new project details, including the name, namespace ID, template usage configuration, and the specific template project ID from the ProductCreate object.
- Creates the project on GitLab using the specified details.
- Logs successful project creation.
For GitLab, you can generate an access token via the settings in the sub-group of your projects, intended for API authentication. This token is used as GITLAB_PRIVATE_TOKEN in this program to authenticate with GitLab.
The code returns project details in a JSONResponse after creating projects on GitLab but incorrectly uses duplicate dictionary keys for messages. It should structure project details distinctly for clarity. It also handles errors specifically for GitLab project creation failures by logging and raising HTTP exceptions based on the error type: an HTTP 400 for GitLab-specific errors and an HTTP 500 for all other unexpected errors.
The command uvicorn main:app --reload is used to run a FastAPI application using Uvicorn, which is an ASGI server. Here’s what each part of the command means:
- uvicorn: The command to run the Uvicorn server.
- main:app: Tells Uvicorn to look for an object named app (which is your FastAPI instance) inside a Python file named main.py.
- --reload: Enables auto-reloading of the server; the server will restart whenever code changes are detected. This is particularly useful during development for seeing updates without manually restarting the server.
The URL http://127.0.0.1:8000/docs#/ refers to the documentation page for a FastAPI application, accessible when running locally. This address points to the auto-generated interactive API documentation provided by FastAPI, which uses Swagger UI. Here, you can see all the available endpoints of your API, their expected inputs, and outputs, and you can also interactively test these endpoints directly from your browser. This feature is especially useful for debugging and developing APIs, as it provides a visual and practical interface for understanding how the API functions.
We have completed the API development; now let's test the APIs. Instead of using curl commands, we can utilize the Swagger UI, which is included by default in FastAPI. This interface displays all the APIs, and you can execute them by clicking the 'Try it out' button.
Let's execute the API to list the templates from GitLab and view the available templates.
We successfully executed the API and received a response from GitLab with the list of templates.
Now, let's make a POST request to create a new 'java-api' project by inputting the template ID.
Let's execute the API after filling in the required inputs in JSON format.
The POST request was successfully executed, and we received a response containing the project ID and project URL to access the newly created project.
A new project is created based on the selected template, which includes skeleton code to kickstart our development.
The new project template also initiates a pipeline to create initial development, staging, and production environments with a 'Hello World' setup. This streamlines the process for developers by reducing the time needed to configure the basic application setup.
Uvicorn server logs will capture and record all API requests.
Note: To keep this brief, I will skip the API integration for AWS, Dynatrace, and ServiceNow. However, the integration process would be similar, aggregating APIs from each tool. Each tool has its own publicly available API documentation that can be referenced.
So, we have now completed the APIs - What have we achieved?
We have made the APIs available to developers, who can find and use templates to initiate their new projects. This includes bootstrapping environments without writing Infrastructure as Code (IaC) and provides a configuration file (.gitlab-ci.yml) to run the pipeline.
How will this bring value to developers?
Templating: We implement templating of code manifests with basic elements, customized based on our DNB requirements. Infrastructure as code (IaC) modules help bootstrap new projects, allowing developers to start building their applications within minutes.
Managing the Environment: Once a project is created, environment management for testing and QA in lower environments like dev and sit, and promoting from uat to prod, can also improve application quality. The infrastructure modules in the template aid in bootstrapping new environments and tearing them down once testing is completed. This facilitates testing different versions in parallel. Additionally, automating the Change Request (CRQ) process helps developers accelerate application releases.
Operate: After the application is deployed into production, monitoring its stability is crucial. The modules in the template for setting up dashboards in Dynatrace are already configured in a standard way to cover most scenarios.
Improving Developer Experience: These are some instances where developer experience can be enhanced, and how these APIs can be instrumental. To access the templates, initiate a POST call to create new projects. This will quickly locate places for automation for every tool, whether integrated in the web UI, CLI, or a Gitlab pipeline. This flexibility is crucial for developers.
Conclusion:
Summarize the transformative potential of using Python APIs to automate DevOps tools, emphasizing the improvements in efficiency, reduction in manual errors, and enhancement of the developer experience. This structure provides a comprehensive framework for discussing how Python can take DevOps automation to the next level, enhancing the overall developer experience and productivity.