It was the quarterly business review when the CFO asked a question nobody on the engineering team could answer.

"Our fraud loss rate has been climbing for six months. Is the model still working?"

The model was working. It was running. It was accepting requests and returning predictions. But was it still accurate? Nobody knew. The model had been trained eight months ago on data from twelve months before that. Transaction patterns had shifted. New fraud vectors had emerged. The feature distributions the model was trained on looked nothing like the data flowing through it in production.

The model was confident. The predictions were smooth. The fraud was getting through.

This is the silent failure mode of ML in production. Not a crash. Not an error. Just a gradual, invisible degradation that nobody catches until the business metric reveals it.

I built a pipeline that catches it automatically.

📺 Watch the Full Build on YouTube

I recorded this entire project live from SSH into EC2 via MobaXterm. Every command. Every file written with heredoc. Every error and fix. If you learn by watching before reading, start here.

Watch the Full Build on YouTube

The written breakdown below covers every architectural decision, every debugging story, and every lesson I would only learn by building this for real.

What This Pipeline Actually Does

A fraud detection model for financial transactions is only useful if it stays accurate over time. Transaction patterns change seasonally. New merchant categories emerge. Fraud techniques evolve. The feature distributions a model was trained on drift gradually away from what the model sees in production.

This pipeline addresses that problem at every stage of the ML lifecycle:

🔗 Full code: github.com/Eaglewings966/mlops-pipeline

This is Project 14 in my 150 Enterprise DevOps Projects challenge.

Why Fraud Detection Is the Right Use Case for MLOps

Fraud detection is the canonical MLOps use case for three reasons that make it harder than a standard classification problem.

Severe class imbalance. The IEEE-CIS Fraud Detection dataset has a 3.5% fraud rate. A model that labels every transaction as legitimate achieves 96.5% accuracy and catches zero fraud. AUPRC (area under the precision-recall curve) is the correct metric — it measures performance across the full range of classification thresholds and is sensitive to minority class performance.

Concept drift. Fraud patterns change constantly. Fraudsters adapt to detection models. New fraud vectors emerge. Seasonal patterns shift. A model trained in Q1 may be significantly less effective by Q3 without any change to the code or infrastructure. Drift detection is not optional — it is essential.

Business stakes. False negatives cost money in fraud losses. False positives cost money in declined legitimate transactions and customer churn. The threshold between them is a business decision that must be monitored and adjusted as model performance drifts.

Every enterprise fintech that processes payments at scale has this problem. Every bank has this problem. The pipeline I built addresses all three dimensions.

The Architecture

Why XGBoost Instead of a Neural Network

XGBoost is the right model for tabular fraud detection data for three reasons:

Neural networks on tabular data with this feature set and volume consistently underperform well-tuned gradient boosted trees. The research literature on this is unambiguous.

Why Optuna Instead of GridSearch or RandomSearch

Optuna uses Tree-structured Parzen Estimator (TPE) for sampling. TPE builds a probabilistic model of which hyperparameter regions have produced good results and samples from promising regions preferentially. This converges to good configurations faster than random search and is more efficient than grid search on large hyperparameter spaces.

The Hyperband pruner kills unpromising trials early based on intermediate results — reducing compute cost by approximately 40%:

study = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=42),
    pruner=optuna.pruners.HyperbandPruner()
)
study.optimize(objective, n_trials=50)

Why SMOTE Instead of Class Weighting

XGBoost has a scale_pos_weight parameter that handles class imbalance. I tested both approaches. SMOTE produced consistently better AUPRC scores across cross-validation folds for this specific dataset.

SMOTE generates synthetic fraud samples in feature space, forcing the model to learn more robust decision boundaries around the minority class. Class weighting makes the model pay more attention to existing fraud samples but does not increase the diversity of the fraud examples it sees.

smote = SMOTE(
    random_state=42,
    sampling_strategy=0.1  # 10:1 ratio after oversampling
)
X_train_resampled, y_train_resampled = smote.fit_resample(
    X_train, y_train
)

⚠️ Critical rule: SMOTE must be applied after the train/test split. Applying it before the split leaks synthetic samples into the test set. I will cover the exact debugging session where I caught this mistake in the challenges section — it cost me 3 hours.

Implementation — The Code That Matters Most

Great Expectations Data Validation

The validation stage answers one question before every training run: is this data actually valid for training a fraud detection model?

# Five checks that must all pass before training proceeds

# Check 1 — Target column is binary
unique_values = df["isFraud"].dropna().unique()
assert set(unique_values).issubset({0, 1})

# Check 2 — Transaction amounts are positive
assert (df["TransactionAmt"] > 0).all()

# Check 3 — Fraud rate is in expected range
fraud_rate = df["isFraud"].mean()
assert 0.005 <= fraud_rate <= 0.05

# Check 4 — No completely empty rows
assert df.isnull().all(axis=1).sum() == 0

# Check 5 — Minimum row count for statistical validity
assert len(df) >= 10000

The validation runs in under 30 seconds on 100,000 rows. If any check fails the pipeline raises a ValueError with the specific check name. No error message says "something went wrong." Every error says exactly what went wrong.

MLflow Experiment Tracking

Every training run logs the complete experimental context:

with mlflow.start_run(run_name="fraud-detection-xgboost") as run:

    # Dataset metadata
    mlflow.log_param("dataset_rows", len(df))
    mlflow.log_param("feature_count", len(feature_cols))
    mlflow.log_param("train_fraud_rate", float(y_train.mean()))

    # All params.yaml values
    mlflow.log_params(params["xgboost_defaults"])

    # After training — metrics
    mlflow.log_metrics({
        "average_precision": 0.8234,
        "roc_auc": 0.9187,
        "f1_score": 0.7412,
        "precision": 0.8124,
        "recall": 0.6821
    })

    # Artifacts — confusion matrix, SHAP plots
    mlflow.log_artifact("/tmp/confusion_matrix.png")
    mlflow.log_artifact("/tmp/shap_summary.png")

    # Register model with signature
    mlflow.xgboost.log_model(
        model,
        "fraud-detection-model",
        registered_model_name="fraud-detection-xgboost",
        signature=infer_signature(X_test[:100], probas)
    )

Every experiment is reproducible from the MLflow UI. Click any run, see every parameter that was used, every metric that was produced, every artifact that was generated. The confusion matrix from run 7 versus run 12 is one click away.

Evidently AI Drift Detection

The drift detector runs daily and compares the current production data distribution against the reference dataset captured at training time:

report = Report(metrics=[
    DatasetDriftMetric(),                              # Share of drifted columns
    DataDriftTable(),                                  # Per-column drift table
    ColumnDriftMetric(column_name="TransactionAmt"),   # Amount distribution
    ColumnDriftMetric(column_name="C1"),               # Card feature
])

report.run(
    reference_data=reference_df,
    current_data=current_df,
    column_mapping=column_mapping
)

# Extract drift result
drift_detected = share_drifted > config.drift_threshold

if drift_detected:
    # Send SNS alert
    sns.publish(
        TopicArn=config.sns_topic_arn,
        Subject="DRIFT ALERT — Model Requires Retraining",
        Message=detailed_drift_report
    )
    # GitHub Actions workflow automatically triggers retraining

The Evidently report is an HTML document showing the distribution of every feature in the reference data versus the current data, with a traffic-light indicator for drift status per column.

The FastAPI Serving API

The serving API loads the Production-stage model from the MLflow registry once at startup and holds it in memory for all predictions:

@asynccontextmanager
async def lifespan(app: FastAPI):
    global model, model_version

    model_uri = f"models:/fraud-detection-xgboost/Production"
    model = mlflow.xgboost.load_model(model_uri)

    client = mlflow.MlflowClient()
    versions = client.get_latest_versions(
        "fraud-detection-xgboost", stages=["Production"]
    )
    model_version = versions[0].version
    yield

When a new model is trained and promoted to Production, the next request after a server restart serves the new model automatically. No redeployment. No configuration change. The model registry is the source of truth.

@app.post("/predict")
async def predict(request: TransactionRequest):
    X = prepare_features(request)
    fraud_probability = float(model.predict_proba(X)[0, 1])

    return {
        "is_fraud": fraud_probability >= 0.5,
        "fraud_probability": round(fraud_probability, 4),
        "risk_level": get_risk_level(fraud_probability),
        "latency_ms": round((time.time() - start_time) * 1000, 2)
    }

Challenges and Debugging — The Parts That Actually Hurt

Challenge 1 — SMOTE Before Split: The Silent Data Leakage Bug

This is the most expensive debugging session of the entire project because the mistake produced results that looked correct.

On my first implementation I applied SMOTE before splitting the data:

# WRONG — data leakage
X_resampled, y_resampled = smote.fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(
    X_resampled, y_resampled, test_size=0.2
)

The model trained this way achieved AUPRC of 0.91 on the test set. Excellent result. I logged it to MLflow and was ready to move on.

Then I noticed something unusual. The model performed significantly worse on a holdout validation dataset I had separately set aside before touching anything.

A gap of 0.18 is not noise. SMOTE generates synthetic samples by interpolating between existing samples in feature space. When applied before the split, synthetic samples generated from real training data points end up in the test set. The test set is contaminated.

The fix:

# CORRECT — split first, then apply SMOTE only to training data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y
)

smote = SMOTE(random_state=42, sampling_strategy=0.1)
X_train_resampled, y_train_resampled = smote.fit_resample(
    X_train, y_train
)
# X_test is NEVER touched by SMOTE

After fixing: AUPRC on test set dropped to 0.82. AUPRC on holdout: 0.80. The two numbers now agree. The model's real performance was 0.82, not 0.91.

💡 Always validate ML performance on a holdout dataset set aside before any preprocessing is applied. If test and holdout scores disagree significantly, look for data leakage.

Challenge 2 — MLflow Model Loading Fails in Serving

After training the model and registering it, the FastAPI serving API failed to start with:

mlflow.exceptions.MlflowException:
The model at models:/fraud-detection-xgboost/Production
was not found. No model with name 'fraud-detection-xgboost'
was found in the registry.

The model was definitely registered — visible in the MLflow UI. The problem was that the serving EC2 instance was using a different MLFLOW_TRACKING_URI than the training instance.

# On the serving server — point to training server's private IP
export MLFLOW_TRACKING_URI=http://TRAINING_SERVER_PRIVATE_IP:5000

After fixing the URI, loading still failed with:

OSError: [Errno 2] No such file or directory:
'/tmp/tmpXXXXXX/fraud-detection-model/xgboost.model'

MLflow was trying to download model artifacts from S3 but the serving EC2 instance was missing the S3 read permission for the MLflow artifacts bucket.

aws iam put-role-policy \
  --role-name mlops-pipeline-serving-role \
  --policy-name mlflow-artifacts-read \
  --policy-document '{
    "Statement": [{
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::mlops-pipeline-mlflow-*",
                   "arn:aws:s3:::mlops-pipeline-mlflow-*/*"]
    }]
  }'

⚠️ Two separate problems, same confusing error message. IAM permission issues in MLflow surface as file-not-found errors because the S3 download silently fails and MLflow looks for the file locally.

Challenge 3 — Optuna Trials Timing Out on EC2

Running 50 Optuna trials with 5-fold cross-validation on 590,000 rows exceeded the GitHub Actions job budget. Each trial took approximately 4 minutes. 50 trials × 4 minutes = over 3 hours.

Three fixes combined to solve this:

Fix 1 — Run Optuna on 20% of training data:

# Hyperparameter search on 20% sample
X_sample = X_train[:int(len(X_train) * 0.2)]
y_sample = y_train[:int(len(y_train) * 0.2)]
best_params, study = run_hyperparameter_tuning(X_sample, y_sample)

# Final model trained on full dataset with best params
model = xgb.XGBClassifier(**best_params)
model.fit(X_train_resampled, y_train_resampled)

Fix 2 — Enable the Hyperband pruner. Unpromising trials are stopped after 2 folds instead of running all 5. Average trial time: 4 minutes → 2.4 minutes.

Fix 3 — Add --no-tuning flag. For automated retraining triggered by drift, use the best parameters from the previous run stored in SSM Parameter Store rather than running 50 new trials from scratch.

⏱️ Time for 50 trials after all three fixes: 82 minutes. Fits within the CI budget.

Challenge 4 — Evidently Drift Report Fails on Missing Columns

ValueError: Column 'V35' is in column_mapping but not in current data

The reference data had all 339 feature columns after feature engineering. The production data was only sending 52 columns. Evidently requires matching columns when column_mapping is specified explicitly.

The fix was to only include columns present in both datasets:

feature_cols = [
    c for c in config.feature_columns
    if c in reference_df.columns and c in current_df.columns
]

column_mapping = ColumnMapping(
    target=target_col,
    numerical_features=[c for c in feature_cols if ...],
    categorical_features=[c for c in feature_cols if ...]
)

💡 This also made the drift report more meaningful — a feature that is missing from production data cannot drift. It is simply absent.

Challenge 5 — DVC Push Failing Due to S3 Permissions

ERROR: failed to push data to the cloud
AuthorizationError checking head object
An error occurred (403) when calling the HeadObject operation: Forbidden

The IAM role had S3 permissions scoped to a specific prefix, but DVC needs to check bucket existence at the root level before writing to the cache prefix. The S3 permissions model requires both separately:

{
  "Effect": "Allow",
  "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
  "Resource": ["arn:aws:s3:::mlops-pipeline-dvc-*"]
},
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
  "Resource": ["arn:aws:s3:::mlops-pipeline-dvc-*/*"]
}

⚠️ Bucket-level actions apply to the bucket ARN without a trailing slash. Object-level actions apply to the ARN with /* appended. Both are needed for DVC to function correctly.

Validation — Proving the Pipeline Works

Validate Training Metrics

cat data/processed/training_metrics.json

Expected output:

{
  "average_precision": 0.82,
  "roc_auc": 0.92,
  "f1_score": 0.74,
  "precision": 0.81,
  "recall": 0.68,
  "test_fraud_count": 4123,
  "test_total": 118000
}

Validate Model Registry

python3.11 << 'EOF'
import mlflow
import os

mlflow.set_tracking_uri(
    os.environ.get("MLFLOW_TRACKING_URI", "http://localhost:5000")
)
client = mlflow.MlflowClient()

versions = client.get_latest_versions(
    "fraud-detection-xgboost",
    stages=["Production"]
)
for v in versions:
    print(f"Version: {v.version}")
    print(f"Stage: {v.current_stage}")
    print(f"Run ID: {v.run_id}")
    print(f"Source: {v.source}")
EOF

Validate API Prediction

# Test a high-risk transaction — should return CRITICAL or HIGH
curl -s -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "TransactionAmt": 8500.0,
    "TransactionHour": 3,
    "TransactionDay": 6
  }' | python3.11 -m json.tool

# Test a low-risk transaction — should return LOW
curl -s -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "TransactionAmt": 25.0,
    "TransactionHour": 14,
    "TransactionDay": 2
  }' | python3.11 -m json.tool

Validate Drift Detection

python3.11 -c "
from src.monitoring.drift_detector import detect_drift
import pandas as pd

# Load and modify reference data to simulate drift
ref = pd.read_csv('data/reference/reference_data.csv')
current = ref.copy()
current['TransactionAmt'] = current['TransactionAmt'] * 3.5
current.to_csv('/tmp/drifted_current.csv', index=False)

results = detect_drift(
    'data/reference/reference_data.csv',
    '/tmp/drifted_current.csv'
)
print(f'Drift detected: {results[\"drift_detected\"]}')
print(f'Share drifted: {results[\"share_drifted_columns\"]:.1%}')
"

Expected: drift_detected = True

Key Lessons From This Build

AUPRC is the only metric that matters for imbalanced fraud data. A model with 97% accuracy that catches zero fraud is worse than useless in production — it provides false confidence. AUPRC measures exactly the thing you care about: across all possible classification thresholds, how well does the model separate fraud from legitimate transactions?

MLflow model stages are not optional in production. Registering a model is not the same as deploying it. The Production stage is the signal the serving API uses to know which model version to load. When you need to roll back, you archive the bad version and promote the previous one. This workflow is only possible if you use stages consistently from the beginning.

Drift detection is only as good as your reference data. I initially used the full training dataset as reference — including SMOTE-generated synthetic samples, which gave Evidently a distorted view of the real data distribution. The correct reference dataset is the pre-SMOTE training data — real samples only.

DVC pipeline caching can mask stale results. If you forget to list a file in a stage's deps section and then modify it, DVC will serve stale output without warning. I modified a feature engineering function once without adding it to the deps list. The model trained on stale features and the metrics looked normal. I only caught it when I ran the pipeline from scratch on a clean machine.

Security Posture at a Glance

Production Gap Analysis

Feature Store. The preprocessing pipeline currently runs inline at training time. In production, features should be computed once and stored in a feature store (Feast or AWS SageMaker Feature Store) to ensure training and serving use identical feature computation logic.

Champion/Challenger Testing. Currently the pipeline promotes every new model to Production automatically if metrics exceed the previous version. A more robust approach sends 10% of traffic to the new model, compares metrics over 24 hours, then promotes or rolls back.

Online Retraining. The current pipeline runs batch retraining when drift is detected — creating a lag between when drift appears and when the new model is deployed. Online learning (incrementally updating the model on new transactions) reduces this lag significantly for high-velocity fraud scenarios.

Model Explainability in the API. The current API returns a fraud probability and a risk level. Regulated financial environments require explanations: "This transaction was flagged because the amount is 4 standard deviations above the mean for this card type and it occurred at 3am." Adding SHAP values to the /predict response enables this.

Cleanup

bash scripts/destroy_all.sh

This empties all S3 buckets, stops all running processes, runs terraform destroy, and terminates EC2 instances.

⚠️ Always run this. The RDS instance, ElastiCache, and EC2 instances will continue billing until explicitly terminated.

What's Next

Project 15 is a RAG (Retrieval Augmented Generation) pipeline — document ingestion, self-hosted embedding model, Qdrant vector database, and a FastAPI query interface backed by Claude.

Follow me at emmanuelubani.hashnode.dev for every project as it ships.

References

• MLflow Documentation

• DVC Documentation

• Evidently AI Documentation

• Great Expectations Documentation

• XGBoost Documentation

• Optuna Documentation

• SMOTE — imbalanced-learn

• FastAPI Documentation

• IEEE-CIS Fraud Detection Dataset

• SHAP Documentation

• Why AUPRC for Imbalanced Classification

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

I was running a routine check across our AWS account when I noticed something that made my stomach drop. An S3 bucket with public access not fully blocked. One of 47 buckets. The kind of thing nobody checks manually because there are 47 of them and checking each one by hand takes an afternoon.

But that was not what made me stop.

What made me stop was realizing I did not know how many other findings like this were sitting in the account:

• Unencrypted EBS volumes

• IAM access keys unused for 14 months

• EC2 instances without any IAM role

• RDS databases publicly accessible

• Resources with zero tags making cost attribution impossible

None of these were catastrophic individually. Combined they represented a compliance audit waiting to happen.

I spent two days building a tool that finds all of it. In under 60 seconds.

📺 Watch the Full Build on YouTube

I recorded this entire build live from an SSH session on EC2 via MobaXterm. Every command. Every file. Every error. If you learn better by watching before reading, start here.

Watch the Full Build on YouTube

The rest of this article covers every file and every decision in writing with extra context that did not fit in the video.

What This Tool Does

The AWS Resource Inventory CLI is a production-grade Python tool that runs entirely from an SSH session on EC2. You type one command. It:

🔗 Full code: github.com/Eaglewings966/aws-resource-inventory

This is Project 13 in my 100 Enterprise DevOps Projects challenge. The previous project was a full Privileged Access Management platform. This one is the visibility layer that tells you what is in your account before someone else finds it first.

Why This Matters for Fintech and Enterprise Teams

Every fintech organization above a certain scale has the same problem. Infrastructure grows faster than governance. An engineer spins up an EC2 instance for a proof of concept. The PoC becomes a feature. The feature ships. The instance stays running for 18 months with no tags, no IAM role, and an unencrypted root volume. Nobody is negligent. The system just has no automated visibility layer.

The first time most teams see their real infrastructure state is during a compliance audit. That is the worst possible time to find out.

PCI-DSS, SOC2, and ISO 27001 all require continuous monitoring of your cloud environment. This tool does not replace a proper CSPM solution at enterprise scale, but for teams building toward compliance it is a working implementation of the principle that you cannot secure what you cannot see.

The Architecture — How the Tool Is Structured

The tool is organized as a Python package with three layers.

Layer 1 — Scanners

Each AWS service has its own scanner module: EC2, S3, RDS, Lambda, EKS, VPC, IAM, CloudWatch, Security Groups, SNS, DynamoDB, ElastiCache, ALB, and a cross-service Tagging Compliance scanner that uses the AWS Resource Groups Tagging API to check every taggable resource in the account simultaneously.

Each scanner returns a standardized dictionary with four keys: service name, region, a list of resource objects, a list of findings with severity and recommendation, and a summary object with counts. This consistent structure means the output layer can consume any scanner's results without knowing which service produced them.

Layer 2 — Outputs

Six output modules each receive the combined results from all scanners and produce a different artifact:

Each output module is independent. Failing to send a Slack message does not prevent the HTML report from being generated.

Layer 3 — CLI

Typer provides the CLI interface with Rich for beautiful terminal output. Progress spinners show each scanner running. A color-coded findings table shows severity counts on completion. File paths are printed clearly so you know exactly where every output landed.

The Implementation — Every Decision Explained

Why SSH Into EC2 Instead of Running Locally

Every command in this project runs from inside an SSH session on EC2 via MobaXterm. This is not a personal preference. It is the correct execution model for tools that need IAM permissions to scan AWS resources.

When you run this tool on EC2 with an IAM instance role attached, the credentials come from the instance metadata service:

• No access keys stored anywhere

• No ~/.aws/credentials file

• No environment variables with sensitive values

Running this locally requires storing AWS credentials somewhere on your development machine. At some point those credentials end up in a git commit, a dotfile backup, or a screenshot. EC2 instance roles eliminate that entire risk class.

The IAM Least Privilege Design

The IAM policy for the runner instance is split into seven permission groups, each covering exactly the API calls each scanner needs:

• EC2 scanning needs DescribeInstances — not CreateInstances or TerminateInstances

• IAM scanning needs ListUsers and ListRoles — not CreateUser or AttachRolePolicy

# The policy document is created with explicit Action lists
# Not managed policies like ReadOnlyAccess which are too broad
aws iam create-policy \
  --policy-name inventory-runner-policy \
  --policy-document file:///tmp/inventory-policy.json

This matters because the inventory tool has read access to sensitive data. If the EC2 instance were compromised, an overly permissive IAM role would let an attacker use the tool's permissions to make destructive changes. Least privilege limits the blast radius to read operations only.

The IAM Scanner — The Most Valuable Scanner in the Tool

The IAM scanner uses the IAM Credential Report — a CSV document that AWS generates showing the complete security posture of every IAM user in the account. Root account MFA status. Access key ages. Last usage dates. Password rotation status. It is the document a security auditor asks for first.

# Generate the credential report
iam.generate_credential_report()
time.sleep(3)

# Parse it as CSV
report_response = iam.get_credential_report()
report_csv = report_response["Content"].decode("utf-8")
reader = csv.DictReader(io.StringIO(report_csv))

The scanner processes every row and flags:

# Check for stale access keys
if key_active and last_used_dt < stale_threshold:
    days_stale = (now - last_used_dt).days
    result["findings"].append({
        "severity": "HIGH",
        "resource_id": row["user"],
        "resource_type": f"IAM Access Key {key_num}",
        "finding": f"Access key unused for {days_stale} days",
        "recommendation": f"Deactivate or delete this key"
    })

The Tagging Scanner — Visibility Across Everything

The tagging scanner uses the AWS Resource Groups Tagging API — the only API that can return all taggable resources across all services in a single set of API calls. EC2 instances, RDS databases, Lambda functions, S3 buckets, EKS clusters, ElastiCache clusters — one paginator covers all of them.

paginator = tagging.get_paginator("get_resources")
for page in paginator.paginate():
    for resource in page["ResourceTagMappingList"]:
        existing_tags = {
            t["Key"]: t["Value"]
            for t in resource.get("Tags", [])
        }
        missing = [t for t in required_tags if t not in existing_tags]
        if missing:
            non_compliant += 1

The required tags are configurable via environment variable. The default set is Project, Environment, Owner, and ManagedBy. These four tags are the minimum needed to answer the three questions every cloud finance team asks: what is this? what environment is it in? who owns it?

💡 The first time I ran the tagging scanner on a real account, the compliance number was 34%. Every engineer on the team believed they were tagging resources correctly. The number was 34%. Without an automated check, tagging discipline degrades invisibly over time.

The HTML Report — Why Dark Theme and Why Jinja2

The HTML report uses a dark theme matching the Tech with Emma brand colors:

The reason I used Jinja2 for templating instead of f-strings or string concatenation is separation of concerns. The template defines how the report looks. The Python code defines what data goes into it. If I want to change the report layout I edit the Jinja2 template without touching the scanner logic.

The Slack Output — Why Blocks API

The Slack notification uses the Blocks API rather than simple text formatting. Blocks give you structured layout — a header section, a fields section with key-value pairs in a grid, and a context section for attribution.

payload = {
    "attachments": [{
        "color": color,  # Red for critical, orange for high, green for clean
        "blocks": [
            {"type": "header", "text": {...}},
            {"type": "section", "fields": [...]},
            {"type": "context", "elements": [...]}
        ]
    }]
}

The color on the attachment border changes based on severity:

Engineers can see the status at a glance from the channel list without opening the message.

Challenges and Debugging — What Actually Went Wrong

Challenge 1 — The Credential Report 503 Error

The IAM credential report generation is an asynchronous operation. On the first attempt:

botocore.exceptions.ClientError: An error occurred (ReportNotPresent)
when calling the GetCredentialReport operation:
No credential report exists. Starting a new report generation task

Added sleep(2) and ran again:

botocore.exceptions.ClientError: An error occurred (ReportInProgress)
when calling the GetCredentialReport operation:
Report generation is still in progress

Two seconds was not enough. For accounts with many IAM users the generation can take 10–15 seconds. The fix was a polling loop:

def wait_for_credential_report(iam_client, max_wait: int = 30) -> str:
    """Poll until the credential report is ready."""
    start = time.time()
    iam_client.generate_credential_report()

    while time.time() - start < max_wait:
        try:
            response = iam_client.get_credential_report()
            return response["Content"].decode("utf-8")
        except iam_client.exceptions.ReportInProgress:
            time.sleep(2)
        except iam_client.exceptions.ReportNotPresent:
            time.sleep(1)

    raise TimeoutError("Credential report did not complete in time")

⏱️ Time spent debugging: 25 minutes. The fix itself took 3 minutes.

Challenge 2 — The S3 Public Access Check False Positive

The S3 scanner was picking up buckets from all regions but only successfully checking the ones in us-east-1. Buckets in eu-west-1 were returning:

botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException)
when calling the GetBucketPublicAccessBlock operation:
The specified location-constraint is not valid

The fix was to get the bucket's actual region first and create a region-specific client for each public access check:

# Get bucket region
location = s3.get_bucket_location(Bucket=name)
bucket_region = location["LocationConstraint"] or "us-east-1"

# Create region-specific client for this bucket
regional_s3 = get_client("s3", bucket_region)
pub = regional_s3.get_bucket_public_access_block(Bucket=name)

⏱️ This adds approximately 400ms per bucket due to the additional API call. For an account with 100 buckets that is 40 extra seconds — acceptable for a daily scan.

Challenge 3 — Moto Tests Breaking on Rich Console Output

The first test run output was filled with Rich terminal formatting codes that made the test output unreadable. The Rich handler was firing even during tests because the logger was configured at module import time.

The fix was to check for a test environment flag:

def get_logger(name: str) -> logging.Logger:
    logger = logging.getLogger(name)
    if not logger.handlers:
        if os.environ.get("TESTING"):
            handler = logging.StreamHandler()
        else:
            handler = RichHandler(...)

# In test configuration
export TESTING=1
python3 -m pytest tests/ -v

Challenge 4 — SES Sandbox Rejection

botocore.exceptions.ClientError: An error occurred (MessageRejected)
when calling the SendEmail operation:
Email address is not verified. The following identities failed
the check in region US-EAST-1: devopsalert3@gmail.com

AWS SES starts every account in sandbox mode. Both the sender and the recipient must be verified. The fix:

aws ses verify-email-identity \
  --email-address devopsalert3@gmail.com \
  --region us-east-1

Click the verification link in the email that arrives. For production use, request SES production access which removes the sandbox restriction.

Challenge 5 — DynamoDB ListTables Pagination

On an account with 340 tables, the scanner only reported 100. The list_tables API uses a different cursor mechanism than most AWS list APIs:

# Wrong — only gets first 100 tables
response = dynamodb.list_tables()
tables = response["TableNames"]

# Correct — paginates through all tables
tables = []
last_evaluated_name = None
while True:
    kwargs = {"Limit": 100}
    if last_evaluated_name:
        kwargs["ExclusiveStartTableName"] = last_evaluated_name

    response = dynamodb.list_tables(**kwargs)
    tables.extend(response["TableNames"])

    last_evaluated_name = response.get("LastEvaluatedTableName")
    if not last_evaluated_name:
        break

⚠️ The pagination token for list_tables is a table name, not an opaque token. Always check the API documentation for the specific pagination mechanism rather than assuming all AWS APIs behave identically.

Validation — Proving the Tool Works

Run the Tool and Verify Each Output

# Run the full scan
python3 -m inventory.main scan --region us-east-1

# Verify JSON output was created
ls -lh reports/inventory_*.json

# Verify CSV findings were created
ls -lh reports/findings_*.csv
cat reports/findings_*.csv | head -10

# Verify HTML report was created
ls -lh reports/report_*.html
wc -l reports/report_*.html

# Verify S3 archive
aws s3 ls s3://${INVENTORY_S3_BUCKET}/inventory-reports/ \
  --recursive --region us-east-1

Verify a Known Finding Is Detected

# Create an unencrypted EBS volume — should trigger HIGH finding
aws ec2 create-volume \
  --availability-zone us-east-1a \
  --size 1 \
  --encrypted false \
  --region us-east-1 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=test-unencrypted}]'

# Run the scanner
python3 -m inventory.main scan \
  --services ec2 \
  --no-slack \
  --no-email \
  --no-s3

# Grep the findings CSV for the expected finding
grep "not encrypted" reports/findings_*.csv

# Clean up the test volume
VOLUME_ID=$(aws ec2 describe-volumes \
  --filters "Name=tag:Name,Values=test-unencrypted" \
  --query 'Volumes[0].VolumeId' \
  --output text --region us-east-1)

aws ec2 delete-volume \
  --volume-id ${VOLUME_ID} \
  --region us-east-1

Expected grep output:

HIGH,EC2,EBS Volume,vol-0abc123def,EBS volume is not encrypted,Enable EBS encryption

The HIGH finding appeared exactly as expected.

Security Posture at a Glance

Key Lessons From This Build

The credential report is the most valuable single API call in all of AWS. It answers every question a security auditor asks about IAM user posture in one call. If you build nothing else from this article, build the part that generates, downloads, and parses the credential report daily.

The tagging compliance number will be lower than you expect. The first real account came back at 34%. Running the tagging scanner weekly and Slack-alerting on any decrease in the compliance percentage changed the team's behaviour more than any policy document or training session.

moto tests are worth the setup time. The moto tests caught two issues that would have produced silent incorrect results in production: the DynamoDB pagination bug and a condition in the IAM scanner where keys with a last-used date of exactly the threshold day were being classified incorrectly.

Production Gap Analysis

Cross-region scanning. The current implementation scans a single region. A production implementation should accept a list of regions and run scanners in parallel using concurrent.futures.ThreadPoolExecutor.

Findings deduplication. If you run this daily, findings present yesterday and still present today should be tracked as a single ongoing finding, not reported as a new finding each day. DynamoDB is the right store for this — keyed by resource ARN and finding type with first-seen and last-seen timestamps.

Auto-remediation for low-risk findings. Unassociated Elastic IPs and unattached EBS volumes are pure cost waste. For findings like these where remediation carries no risk, add an --auto-remediate flag that releases unassociated EIPs and snapshots then deletes unattached volumes older than 30 days.

Ticketing system integration. A CRITICAL finding should open a P1 in your ticketing system automatically with the resource ID, the finding, and the exact AWS CLI command to fix it. The Jira API and GitHub Issues API are both workable targets.

Cleanup

bash scripts/destroy-all.sh

This script empties the S3 bucket, terminates the EC2 runner, and removes the IAM role and policy.

# Quick verification
aws ec2 describe-instances \
  --filters "Name=tag:Project,Values=aws-resource-inventory" \
           "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[*].InstanceId' \
  --output text --region us-east-1

Expected output: empty. No running instances.

What's Next

Project 14 is a self-managed, highly available PostgreSQL cluster on Kubernetes with production load testing — watching zero transactions drop when the primary node is killed during a 500-user pgBench run.

Follow me at emmanuelubani.hashnode.dev for every new project as it ships.

References

• AWS IAM Credential Report Documentation

• AWS Resource Groups Tagging API

• Typer CLI Framework

• Rich Terminal Library

• Jinja2 Templating

• Moto AWS Mocking Library

• Boto3 Documentation

• AWS SES Sandbox and Production Access

• AWS S3 Public Access Block

• PCI-DSS Cloud Security Requirements

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

It Was 2am When the Security Audit Report Landed in My Inbox

The subject line read: "CRITICAL FINDINGS — Immediate Action Required."

I opened it half-asleep. The first table stopped me cold.

847 active SSH keys across 63 production servers. 312 of them belonged to engineers who had left the company more than six months ago. 47 of those engineers had left over a year ago. One key was traced back to a contractor engagement that ended in 2019.

Every single one of those keys could still authenticate to a production server right now.

The security team gave us three options:

This article is the complete implementation guide.

📺 Watch the Full Build on YouTube

I recorded the complete build walkthrough for this project. If you prefer watching over reading, or want to see every terminal command and every error in real time, the full video is on YouTube.

Watch the Full Build on YouTube

The written breakdown continues below with every command, every configuration file, every mistake I made, and the lessons I would only learn by building this in a real environment.

What Is Privileged Access Management and Why Most Teams Get It Wrong

The standard approach goes like this. An engineer joins the team. Someone generates an SSH key pair for them. The public key is added to the authorized_keys file on every server they need access to. The engineer has access. Everything works.

Then the engineer leaves. Someone removes their Slack access. Someone disables their GitHub account. Someone deactivates their Okta login. But the SSH key? It sits in that authorized_keys file until someone specifically hunts it down and removes it. Which never happens systematically. Which is how you end up with 847 active keys belonging to people who no longer work for you.

The correct solution is not better key management. It is making persistent SSH keys architecturally impossible.

In this project I built a PAM platform with three components working together:

• HashiCorp Boundary — session brokering and access control

• HashiCorp Vault SSH CA — credential issuance with a 3-minute TTL

• Okta OIDC — identity and group-based authorization

When an engineer leaves and their Okta account is deprovisioned, their access to every server disappears in the same instant. No key rotation. No access review. One deprovisioning action covers everything.

🔗 Full code: github.com/Eaglewings966/privileged-access-management

The Architecture — What Gets Built and Why Each Component Exists

The system has six layers. Understanding why each layer exists makes the configuration decisions make sense rather than feeling like arbitrary complexity.

Layer 1 — Identity (Okta OIDC)

Okta is the source of truth for who you are and what groups you belong to. Group membership in Okta maps directly to access policies in Boundary:

• pam-admins group → access to production targets

• developers group → access to development targets only

When someone is removed from a group, their access disappears immediately. The reason I chose Okta over AWS IAM for identity is portability — engineers already have Okta accounts, their laptops are managed by Okta device trust, their VPN uses Okta. Adding PAM to that same identity layer means one less credential set and one less deprovisioning step when someone leaves.

Layer 2 — Session Brokering (HashiCorp Boundary)

Boundary is not a bastion host. This distinction matters enormously.

I deployed Boundary on EKS EC2 with a PostgreSQL RDS backend for state and ElastiCache Redis for session storage. Boundary's entire access model — which targets exist, which users can access them, which sessions are active — lives in PostgreSQL. If you use an in-memory store, you lose all of this on restart.

Layer 3 — Credential Issuance (Vault SSH CA)

This is the most important component in the entire platform. Vault acts as a Certificate Authority for SSH. When Boundary approves an access request, it calls Vault and asks for a signed SSH certificate for the requesting user. Vault issues that certificate with a TTL of exactly 180 seconds.

The certificate contains:

• The user's identity (their username)

• The principals they can authenticate as (ec2-user)

• The exact validity window (now to now + 3 minutes)

• A cryptographic signature from the Vault CA

The target server has one line added to its sshd_config:

TrustedUserCAKeys /etc/ssh/vault-client-ca.pub

That single line means the server will accept any SSH certificate signed by the Vault CA. It will not accept static SSH keys. The authorized_keys file is empty. Static key authentication is disabled entirely.

Layer 4 — Target Instances (EC2 in Private Subnets)

Target EC2 instances sit in private subnets with no inbound rules from the internet. Port 22 is only open from within the VPC. The only way to reach them is through Boundary. Even if an attacker somehow obtained a valid Vault SSH certificate, they would still need network access to the private subnet to use it.

Layer 5 — Session Recording (S3)

Every session brokered by Boundary is recorded and stored in an S3 bucket with server-side KMS encryption. The recording captures every command typed, every output returned, and full timestamps. The recording is immutable — it cannot be deleted by the engineer who created it.

S3 lifecycle policy: STANDARD_IA after 30 days → GLACIER after 90 days. Long-term compliance retention at minimal cost.

Layer 6 — Audit and Alerting (Python + CloudWatch + SNS)

A Python script reads the access log and S3 session metadata and generates a weekly HTML audit report showing every access event per user per target per environment. A CloudWatch alarm fires an SNS email whenever an access request occurs outside business hours (8am–6pm WAT, Monday–Friday).

The weekly report that used to require a security engineer to manually correlate SSH logs across 63 servers now runs in 4 seconds.

The Implementation — Every Command in Sequence

Creating the EKS EC2 Cluster

Everything runs from an SSH session into an EC2 bastion or directly from AWS CloudShell.

# SSH into your EC2 bastion first
ssh -i bastion-key.pem ubuntu@<BASTION_IP>

# Create EKS cluster with EC2 node groups
eksctl create cluster \
  --name pam-cluster \
  --region us-east-1 \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 2 \
  --nodes-max 5

# Configure kubectl
aws eks update-kubeconfig \
  --region us-east-1 \
  --name pam-cluster

# Verify
kubectl get nodes

Wait for all three nodes to show Ready status before proceeding. On EC2 node groups this takes approximately 5 minutes.

Provisioning Infrastructure with Terraform

The Terraform configuration provisions nine categories of resources: RDS PostgreSQL, ElastiCache Redis, S3 for session recordings, EC2 target instances, IAM roles, security groups, SNS topics, CloudWatch alarms, and Kubernetes namespaces.

cd privileged-access-management/terraform

# Initialize with S3 backend
terraform init \
  -backend-config="bucket=your-tfstate-bucket" \
  -backend-config="key=pam/terraform.tfstate" \
  -backend-config="region=us-east-1"

terraform fmt
terraform validate
terraform plan -out=tfplan
terraform apply tfplan

⏱️ The apply takes approximately 12 minutes. The longest component is the ElastiCache Redis cluster which takes around 8 minutes even for a single cache.t3.micro node.

After apply, save the outputs:

terraform output

PROD_IP=$(terraform output -raw target_prod_private_ip)
DEV_IP=$(terraform output -raw target_dev_private_ip)
BUCKET=$(terraform output -raw session_recordings_bucket)
RDS_ENDPOINT=$(terraform output -raw rds_endpoint)
REDIS_ENDPOINT=$(terraform output -raw redis_endpoint)

Installing Vault and Configuring the SSH CA

# Add the HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Install Vault
helm install vault hashicorp/vault \
  --namespace vault \
  --version 0.27.0 \
  -f vault/values.yaml

# Wait for the pod to be Running
kubectl get pods -n vault --watch

Once the pod shows 1/1 Running, initialize Vault:

# Port-forward Vault API
kubectl port-forward svc/vault -n vault 8200:8200 &
sleep 3

export VAULT_ADDR=http://localhost:8200

# Initialize with 3 key shares, threshold of 2
kubectl exec -it vault-0 -n vault -- \
  vault operator init \
    -key-shares=3 \
    -key-threshold=2 \
    -format=json > vault-init.json

# Save the output immediately
cat vault-init.json

# Extract and export root token
ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')
UNSEAL_1=$(cat vault-init.json | jq -r '.unseal_keys_b64[0]')
UNSEAL_2=$(cat vault-init.json | jq -r '.unseal_keys_b64[1]')

export VAULT_TOKEN=${ROOT_TOKEN}

# Unseal Vault
vault operator unseal ${UNSEAL_1}
vault operator unseal ${UNSEAL_2}

# Verify
vault status

Now configure the SSH CA:

# Set the target IPs before running the script
export PROD_TARGET_IP=${PROD_IP}
export DEV_TARGET_IP=${DEV_IP}
export SSH_CERT_TTL=180

# Run the full Vault SSH CA configuration
bash vault/config/auth-setup.sh

The auth-setup.sh script performs eight configuration steps: enables the ssh-client-signer and ssh-host-signer secrets engines, generates CA key pairs for both, creates signing roles for pam-admin and pam-developer with a 180-second TTL, creates the Vault policies, enables Kubernetes auth for Boundary, and writes the target metadata to KV v2.

After the script completes, verify the SSH CA is working:

# Test certificate signing
ssh-keygen -t ed25519 -f /tmp/test-key -N "" -q

vault write ssh-client-signer/sign/pam-admin \
  public_key=@/tmp/test-key.pub \
  valid_principals="ec2-user"

# Show the certificate details
vault write -field=signed_key \
  ssh-client-signer/sign/pam-admin \
  public_key=@/tmp/test-key.pub > /tmp/test-cert.pub

ssh-keygen -L -f /tmp/test-cert.pub

The ssh-keygen -L output is the most important verification in this entire project. Look at the Valid line:

Valid: from 2024-01-15T14:23:07 to 2024-01-15T14:26:07

The difference between from and to is exactly 3 minutes. That is your proof that the SSH CA is configured correctly.

# Clean up test keys
rm -f /tmp/test-key /tmp/test-key.pub /tmp/test-cert.pub

Configuring Target Instances to Trust Vault CA

Each target EC2 instance needs one configuration change: it must be told to trust SSH certificates signed by the Vault CA. The setup-ssh-ca.sh script handles this for all targets:

bash vault/ssh-ca/setup-ssh-ca.sh \
  "${PROD_IP},${DEV_IP}" \
  targets-key.pem \
  ec2-user

What this script does step by step:

1. Fetches the CA public key from Vault:

CA_PUBLIC_KEY=$(vault read \
  -field=public_key \
  ssh-client-signer/config/ca)

2. Uploads the CA key and configures sshd on each target:

ssh -i targets-key.pem ec2-user@${TARGET_IP} \
  "sudo bash -c '
    echo \"${CA_PUBLIC_KEY}\" > /etc/ssh/vault-client-ca.pub
    echo \"TrustedUserCAKeys /etc/ssh/vault-client-ca.pub\" \
      >> /etc/ssh/sshd_config
    systemctl restart sshd
  '"

3. Removes all existing authorized_keys files:

ssh -i targets-key.pem ec2-user@${TARGET_IP} \
  "sudo find /home /root \
    -name authorized_keys \
    -delete 2>/dev/null"

⚠️ After this step, the targets-key.pem file cannot be used to access these servers anymore. The only valid authentication method is a Vault-signed certificate.

Verify the configuration:

# Try to use the static key — this should FAIL
ssh -i targets-key.pem ec2-user@${PROD_IP}
# Expected: Permission denied (publickey)

# Now get a Vault certificate — this should SUCCEED
bash scripts/access-request.sh ${PROD_IP} ec2-user pam-admin

Testing the Full Access Flow

bash scripts/access-request.sh \
  ${PROD_IP} \
  ec2-user \
  pam-admin

Watch the output:

================================================
PAM Access Request
================================================
Target:     10.0.1.50
User:       ec2-user
Vault Role: pam-admin
Certificate TTL: 3 minutes

Temporary key pair generated
Requesting certificate from Vault...

Certificate issued:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Key ID: "vault-pam-admin-1705329787"
Valid: from 2024-01-15T14:23:07 to 2024-01-15T14:26:07
Principals: ec2-user
Extensions: permit-pty, permit-port-forwarding

Connecting to 10.0.1.50...
Certificate expires in 3 minutes. Complete your work before then.
================================================

[ec2-user@ip-10-0-1-50 ~]$

You are now connected via a certificate that will expire in 3 minutes. No static key was used at any point.

Generating the Audit Report

pip3 install boto3

python3 scripts/generate-audit-report.py \
  --bucket ${BUCKET} \
  --region us-east-1 \
  --output reports/audit-report.html \
  --days 7

Output:

============================================================
PAM Audit Report Generator
============================================================
Period: 2024-01-08 to 2024-01-15
Bucket: pam-session-recordings-123456789

Loading session data...
  Found 12 sessions from access log
  Found 12 sessions from S3 recordings
  Total: 24 sessions

Generating HTML report...
Report generated: reports/audit-report.html
============================================================

4 seconds. The report that used to take three days now runs in 4 seconds.

Challenges and Debugging — The Parts Nobody Warns You About

Challenge 1 — The SSH Lockout That Cost Me a Full Instance

This is the most expensive mistake in the entire build. I am documenting it in detail because I have seen this exact error destroy hours of work for multiple engineers.

I was configuring the Ansible role that hardens the SSH config on target instances. The role included this task:

- name: Restrict SSH to specific users
  lineinfile:
    path: /etc/ssh/sshd_config
    line: "AllowUsers ec2-user"
    state: present
  notify: Restart sshd

The playbook ran against a mixed inventory that included both Amazon Linux instances (default user: ec2-user) and Ubuntu instances (default user: ubuntu). The handler fired. sshd restarted on the Ubuntu instance with AllowUsers ec2-user. The user ec2-user does not exist on Ubuntu. I lost SSH access permanently.

Recovery via AWS SSM Session Manager:

aws ssm start-session \
  --target i-0abc123def456 \
  --region us-east-1

SSM worked because it bypasses sshd entirely. I corrected the config:

sudo sed -i '/AllowUsers ec2-user/d' /etc/ssh/sshd_config
sudo systemctl restart sshd

The fix — always parameterize the SSH username per OS:

vars:
  ssh_user: "{{ 'ec2-user' if ansible_distribution == 'Amazon'
                else 'ubuntu' }}"

tasks:
  - name: Restrict SSH to correct user
    lineinfile:
      path: /etc/ssh/sshd_config
      line: "AllowUsers {{ ssh_user }}"

⚠️ Time lost: 45 minutes plus one terminated and reprovisioned instance. Before running any sshd configuration change, verify that AWS SSM Session Manager is enabled and working. SSM is your emergency parachute when SSH configuration changes go wrong.

Challenge 2 — Vault SSH CA Not Trusted After sshd Restart

After configuring the first target instance, I tested a certificate connection and got:

sign_and_send_pubkey: signing failed: agent refused operation
ec2-user@10.0.1.50: Permission denied (publickey)

The certificate was valid. The CA key was on the server. The TrustedUserCAKeys line was in sshd_config. But authentication was failing.

I ran sshd in debug mode to find the actual cause:

sudo /usr/sbin/sshd -d -p 2222

The output showed:

debug1: Certificate type 1 not permitted by TrustedUserCAKeys

Certificate type 1 is a host certificate. Type 0 is a user certificate. My Vault signing role had accidentally set allow_host_certificates=true. The fix:

vault write ssh-client-signer/roles/pam-admin \
  allow_user_certificates=true \
  allow_host_certificates=false \
  key_type="ca" \
  default_user="ec2-user" \
  ttl="180s"

💡 Always run sshd in debug mode on a test port when SSH certificate authentication fails. The debug output tells you exactly which check is failing and why.

Challenge 3 — ElastiCache Redis TLS Connection Error

Terraform provisioned ElastiCache with transit_encryption_enabled=true. When Boundary tried to connect to Redis it failed with:

ERR This Redis cluster does not support non-TLS connections

The Boundary configuration was using a plain Redis URL. ElastiCache with transit encryption requires the rediss:// scheme (double s for SSL). Two fixes required:

# Update the Redis URL scheme
REDIS_URL="rediss://pam-sessions.xxxxx.cache.amazonaws.com:6379"

# Add the AWS certificate bundle
curl -o /etc/ssl/certs/aws-elasticache-bundle.pem \
  https://www.amazontrust.com/repository/AmazonRootCA1.pem

💡 Any time you enable transit encryption on an AWS managed service, update every client connection string to use the encrypted scheme. rediss:// for Redis, jdbc:postgresql:// with sslmode=require for PostgreSQL.

Challenge 4 — OIDC Redirect URI Mismatch With Okta

The first login attempt failed with an Okta error:

The redirect URI in the request did not match a registered redirect URI.

The difference between what Boundary sent and what Okta had registered:

# What Boundary sent:
http://localhost:7171/v1/auth-methods/oidc:authenticate:callback

# What Okta had registered:
http://localhost:7171/v1/auth-methods/oidc:authenticate:callback/

One trailing slash. Okta treats these as completely different URIs.

⚠️ OIDC redirect URI matching is exact string comparison. No trailing slash difference, no http vs https difference, no subdomain difference is tolerated. Copy and paste from the Boundary documentation. Never type it manually.

Challenge 5 — Boundary Database Init Must Run Before Controller Starts

The first time the Boundary controller pod started it immediately entered CrashLoopBackOff:

[ERROR] controller: error connecting to database: 
  ERROR: relation "boundary_schema_version" does not exist

Boundary requires the database schema to be initialized before the controller can start. The fix was a Kubernetes Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: boundary-db-init
  namespace: boundary
spec:
  template:
    spec:
      containers:
      - name: boundary-init
        image: hashicorp/boundary:latest
        command:
          - boundary
          - database
          - init
          - -config
          - /boundary/config/boundary.hcl
        env:
          - name: BOUNDARY_POSTGRES_URL
            valueFrom:
              secretKeyRef:
                name: boundary-db-credentials
                key: connection_string
      restartPolicy: OnFailure

💡 Any Boundary deployment must include a database initialization Job that completes before the controller Deployment starts. The CrashLoopBackOff error message is not helpful — always check the full log output for the actual cause.

Validation — Proving Every Component Works

Validating Static Key Authentication Is Completely Disabled

ssh -i targets-key.pem ec2-user@${PROD_IP}
# Expected: Permission denied (publickey)

Static keys are rejected. The target server will only accept certificates signed by the Vault CA.

Validating the 3-Minute TTL Enforcement

# Generate and use a certificate
ssh-keygen -t ed25519 -f /tmp/demo-key -N "" -q

vault write -field=signed_key \
  ssh-client-signer/sign/pam-admin \
  public_key=@/tmp/demo-key.pub > /tmp/demo-cert.pub

# Connect successfully
ssh -i /tmp/demo-key \
    -i /tmp/demo-cert.pub \
    -o StrictHostKeyChecking=no \
    ec2-user@${PROD_IP}

# Wait 4 minutes then try the SAME certificate
sleep 240

ssh -i /tmp/demo-key \
    -i /tmp/demo-cert.pub \
    -o StrictHostKeyChecking=no \
    ec2-user@${PROD_IP}
# Expected: Permission denied (publickey)

The certificate expired. The cryptographic signature is now outside its validity window. The server rejects it.

Validating Session Recording Is Working

aws s3 ls s3://${BUCKET}/sessions/ --recursive --region us-east-1

Expected output:

2024-01-15 14:23:07    4096 sessions/2024/01/15/session-abc123.cast
2024-01-15 14:31:44    8192 sessions/2024/01/15/session-def456.cast

Validating CloudWatch Alarm for After-Hours Access

aws cloudwatch put-metric-data \
  --namespace PAMPlatform \
  --metric-name AccessOutsideBusinessHours \
  --value 1 \
  --unit Count \
  --region us-east-1

Within 1–2 minutes the CloudWatch alarm should transition to ALARM state and send an SNS email notification.

Security Posture at a Glance

Key Lessons From This Build

Production Gap Analysis

Boundary worker in every VPC. A Boundary worker deployed into each VPC or subnet where targets live eliminates the need for peered network paths and improves latency for engineers in distant regions.

Session timeout at the Boundary level. Vault's 3-minute TTL only blocks new connections. An idle SSH session stays open indefinitely after the certificate expires. Configure Boundary session connection limits and maximum duration to enforce automatic termination after inactivity.

Vault audit logging. Every certificate issuance, every policy evaluation, every authentication event should be logged to a persistent audit device and shipped to your SIEM. This gives you a complete record of every credential issuance request independent of Boundary's own audit trail.

Vault namespaces for multi-team isolation. If multiple teams share the same Vault cluster, Vault Enterprise namespaces provide complete isolation at the policy, auth method, and secrets engine level.

Break-glass access procedures. Define, document, and test quarterly the procedure for accessing systems when Boundary is unavailable. The procedure must exist, be tested, and be known by more than one person.

Full Source Code

The complete code is available at:

🔗 github.com/Eaglewings966/privileged-access-management

When you are done testing, clean up your environment:

bash scripts/destroy-all.sh

The destroy script empties the S3 session recordings bucket, uninstalls Vault and Boundary Helm releases, deletes Kubernetes namespaces, runs terraform destroy, deletes the EKS cluster via eksctl, and removes local temporary key files.

After running, verify in the AWS console that the following are all gone: RDS instance, ElastiCache cluster, EC2 target instances, S3 bucket, EKS cluster, IAM roles, security groups, SNS topic, and CloudWatch alarm.

# Quick verification via AWS CLI
aws ec2 describe-instances \
  --filters "Name=tag:Project,Values=pam" \
           "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[*].InstanceId' \
  --output text \
  --region us-east-1

aws rds describe-db-instances \
  --query 'DBInstances[?contains(DBInstanceIdentifier,`pam`)].DBInstanceIdentifier' \
  --output text \
  --region us-east-1

Both commands should return no output.

What's Next

Project 13 is the Immutable Infrastructure Pipeline — Packer golden AMIs, Ansible pre-baked hardening, InSpec compliance testing before AMI promotion, and automated drift detection.

References

• HashiCorp Boundary Documentation

• Vault SSH Secrets Engine

• Vault SSH Certificate Authority

• Okta OIDC Integration Guide

• AWS ElastiCache TLS Documentation

• AWS SSM Session Manager

• eksctl Documentation

• Terraform AWS Provider

• OpenSSH Certificate Authentication

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

The Audit That Cost $3.2 Million

In 2022, a fintech startup in Berlin failed their SOC2 Type II audit. The findings were not sophisticated:

• Root login was enabled over SSH on 34 of their 41 EC2 instances

• Audit logging was not configured on any instance

• World-writable files existed in production directories

• The password policy allowed single-character passwords that never expired

None of this was the result of an attack. It was the result of years of infrastructure growth without compliance monitoring. Nobody had ever checked. Not because the engineers were careless. Because there was no automated system watching.

The compliance remediation took four months of manual work. During those four months, a $3.2 million enterprise contract went to a competitor who could show a clean compliance report on demand. The startup's engineers were skilled. Their infrastructure was working. But they could not prove it was secure.

Compliance is not a one-time event. It is a continuous state. Every new instance provisioned, every configuration change applied, every package update installed can introduce drift. Manual audits catch drift weeks after it happens. This pipeline catches it within hours and fixes it automatically.

Who I Am and Why I Built This

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. I transitioned into tech in January 2024 after working as a zoo volunteer, an automotive technician apprentice, and a beauty and cosmetics store manager.

I built this project because compliance automation is one of the most underbuilt areas in cloud engineering. Teams spend enormous time building CI/CD pipelines for application delivery but leave the compliance state of their infrastructure to quarterly manual audits. This project treats compliance the same way mature teams treat code quality — as something that is continuously measured, automatically reported, and capable of blocking a deployment when it falls below an acceptable threshold.

What This Pipeline Does in Plain Terms

A fresh EC2 instance running Amazon Linux 2023 scores approximately 47% against the CIS Level 2 benchmark. This means 53% of the security controls that CIS considers essential for a production server are not configured by default.

The improvement from 47% to above 85% happens in a single automated pipeline run. No manual intervention. No checklist. No quarterly review. A score that would fail a SOC2 audit becomes a score that passes it — in under 30 minutes, documented and auditable at every step.

Implementation

Step 1 — OpenSCAP Scanning With Ansible

OpenSCAP is the open-source tool used by the US government, the NSA, and enterprises worldwide for SCAP-based compliance scanning. It reads a SCAP content file containing the CIS benchmark rules and evaluates the system against every rule.

⚠️ The two most important things to know about oscap exit codes:

• Exit code 2 means non-compliant — the scan ran successfully and found issues

• Exit code 1 means the scan itself failed

If you write failed_when: oscap_result.rc != 0, your playbook will fail on every non-perfect system. That means every system — because no fresh EC2 instance is 100% CIS compliant.

- name: Run OpenSCAP CIS Level 2 scan
  command: >
    oscap xccdf eval
    --profile {{ cis_l2_profile }}
    --results {{ reports_dir }}/cis_level2_results.xml
    --report {{ reports_dir }}/cis_level2_report.html
    {{ scap_content_base }}/{{ scap_file }}
  register: oscap_result
  failed_when: oscap_result.rc not in [0, 2]
  changed_when: false

Step 2 — Score Extraction

After the scan, the compliance score is buried in the XML results file. This extracts it and saves it as a fact for every subsequent pipeline stage:

- name: Extract CIS Level 2 compliance score
  shell: >
    grep -oP 'score.*?\K[\d.]+(?=%)' \
    {{ reports_dir }}/cis_level2_results.xml | head -1
  register: level2_score_raw
  changed_when: false

- name: Set CIS Level 2 score fact
  set_fact:
    cis_level2_score: "{{ level2_score_raw.stdout | float | round(2) }}"

The score is saved as a fact and written to a JSON summary file that every subsequent pipeline stage reads.

Step 3 — Auto-Remediation With Ansible

The remediation role applies fixes for every automatically remediable finding. The two most impactful remediation tasks:

SSH hardening covers 15–20 CIS rules in a single role:

- name: "[CIS 5.2] Harden SSH configuration"
  lineinfile:
    path: /etc/ssh/sshd_config
    regexp: "^#?{{ item.key }}"
    line: "{{ item.key }} {{ item.value }}"
    state: present
    backup: yes
  loop:
    - { key: "PermitRootLogin", value: "no" }
    - { key: "MaxAuthTries", value: "3" }
    - { key: "Protocol", value: "2" }
    - { key: "X11Forwarding", value: "no" }
    - { key: "LogLevel", value: "VERBOSE" }
  notify: Restart SSH

Kernel sysctl hardening covers another 10–15 rules:

- name: "[CIS 3.1] Apply kernel sysctl hardening"
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
    reload: yes
    sysctl_file: /etc/sysctl.d/99-cis-hardening.conf
  loop:
    - { key: "net.ipv4.ip_forward", value: "0" }
    - { key: "net.ipv4.conf.all.log_martians", value: "1" }
    - { key: "net.ipv4.tcp_syncookies", value: "1" }
    - { key: "kernel.randomize_va_space", value: "2" }

Step 4 — The Compliance Gate

This is the piece that makes the entire system matter for CI/CD. The compliance gate is a Python script that reads the JSON summary files, evaluates every instance against the threshold, and exits with code 1 if anything fails:

def evaluate_compliance(reports, threshold, profile, fail_on_critical):
    results = []
    all_passed = True
    score_key = f"{profile.replace('-', '_')}_score"

    for report in reports:
        score = float(report.get(score_key, 0))
        critical = int(report.get("critical_findings", 0))
        passed = score >= threshold
        critical_block = fail_on_critical and critical > 0

        result = {
            "hostname": report.get("hostname", "unknown"),
            "score": score,
            "passed": passed and not critical_block,
        }
        results.append(result)
        if not result["passed"]:
            all_passed = False

    return all_passed, results

In GitHub Actions, this script runs as its own job. If it exits with code 1, all downstream jobs including the deploy job are blocked. The pipeline cannot proceed. The deployment cannot happen.

Step 5 — CloudWatch Metrics

The Python metrics publisher pushes scores to a custom CloudWatch namespace. This creates the compliance trend chart over time:

cloudwatch.put_metric_data(
    Namespace="ComplianceScanner",
    MetricData=[{
        "MetricName": "ComplianceScore",
        "Dimensions": [
            {"Name": "InstanceId", "Value": instance_id},
            {"Name": "Profile", "Value": "cis-level2"}
        ],
        "Timestamp": datetime.now(timezone.utc),
        "Value": float(report["cis_level2_score"]),
        "Unit": "Percent"
    }]
)

The CloudWatch dashboard shows these metrics over time with a horizontal annotation line at 85% — the deployment threshold. Any dip below that line is immediately visible.

What Broke During This Build

Debugging Story 1 — The SSH Lockout

On my first run of the remediation playbook, the playbook failed halfway through the SSH hardening tasks. The handler had not yet fired to restart SSH, so I had a modified sshd_config that had not been applied yet.

The next time I connected, everything worked fine — the old config was still active. Then I ran the playbook again. This time the handler fired. sshd restarted with the new config. The new config had AllowUsers ec2-user — but the instance user was actually ubuntu.

I lost SSH access permanently.

⚠️ Two lessons from this:

1. Always test remediation on a throwaway instance before touching anything with real data

2. Never add AllowUsers to sshd_config in a playbook without parameterizing the username per OS type

Debugging Story 2 — The Dynamic Inventory Discovery Problem

The aws_ec2 dynamic inventory plugin requires boto3 on the control machine and the amazon.aws Ansible collection to be installed. On the GitHub Actions runner, both were missing. The pipeline failed with a cryptic error about the inventory plugin not being found.

The fix was adding these two steps before running any ansible command in the pipeline:

- name: Install Ansible and dependencies
  run: |
    pip install ansible boto3 botocore
    ansible-galaxy collection install amazon.aws

💡 The error message when this is missing does not clearly state what is wrong. It says the aws_ec2 plugin cannot be loaded. The actual cause is that the amazon.aws collection is not installed.

The Real Lesson

Here is the thing about compliance automation that most engineers do not say directly.

The goal of compliance automation is not to pass audits. The goal is to make the audit irrelevant.

When an auditor asks to see your CIS compliance posture, the correct answer is not a spreadsheet from last quarter. The correct answer is opening a CloudWatch dashboard that shows:

• The compliance score of every instance for the last 90 days

• A GitHub Actions run history showing every deployment passed the compliance gate

• An HTML report generated 12 minutes ago

The Berlin startup that lost the $3.2 million contract failed because they could not show current compliance state on demand. Not because they were non-compliant. Because they did not know whether they were compliant or not without spending four months checking manually.

This pipeline makes the compliance state observable at any moment. The dashboard shows it. The reports prove it. The CI/CD gate enforces it. An auditor who arrives without warning gets a live answer, not a promise that you will get back to them next week.

That is the real value of compliance automation. Not passing audits. Eliminating the audit as a stressful event by turning compliance into a continuously monitored, automatically enforced property of your infrastructure.

Security Posture at a Glance

Production Gap Analysis

This pipeline covers the CIS benchmark remediation pattern fully. A production implementation would additionally require the following.

Drift detection on a schedule. New instances are provisioned continuously. Every new instance starts at baseline compliance. A scheduled pipeline run weekly ensures no instance drifts below threshold without being caught.

OpenSearch for historical compliance data. CloudWatch retains metrics for 15 months but the querying is limited. OpenSearch provides richer querying for compliance trend analysis across fleets of hundreds of instances.

Automated evidence packages for auditors. SOC2 auditors want documented evidence in specific formats. A production pipeline generates structured evidence packages per audit period, not just HTML reports.

Windows instance support. This pipeline covers Linux only. Enterprise environments typically have Windows instances that require different tooling — CIS-CAT Pro or Microsoft's STIG tooling — for equivalent coverage.

How This Helps Companies

The Berlin startup spent four months doing manually what this pipeline does in 30 minutes. They spent those four months not closing their \(3.2 million contract. The engineering time cost alone — four engineers for four months — was approximately \)280,000 at market rates.

This pipeline is not a luxury for mature organizations. It is the baseline that any organization handling customer data should have before they try to sign enterprise contracts. Enterprise clients ask about compliance. Every time. Without exception.

Full Source Code

The complete code is available at:

🔗 github.com/Eaglewings966/openscap-ansible-compliance

When you are done testing, clean up your environment:

cd terraform && terraform destroy --auto-approve
rm -f compliance-key.pem

What's Next

Project 12 covers Privileged Access Management — HashiCorp Boundary replacing bastion hosts, Vault SSH CA for just-in-time certificates, and full session recording.

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

In April 2023, a financial services company in London received a phone call from their bank. Unusual transactions had been detected on their payments processing account.

The investigation that followed took three weeks and cost $4.3 million in forensics, remediation, and regulatory fines.

Here is what the investigators found.

An attacker had compromised a developer's laptop through a phishing email. On that laptop was a kubeconfig file with admin access to the production Kubernetes cluster. The attacker used that kubeconfig to exec into a running pod. The pod was a simple logging service with no business-sensitive data. But it was inside the cluster.

And once inside the cluster, there was nothing stopping lateral movement.

The logging pod could make HTTP requests to the payments service. There was no network policy preventing it. There was no mTLS requiring mutual authentication between services. There was no authorization policy restricting which services could talk to which. There was no runtime detection watching for unusual connections from a logging pod to a payments endpoint.

The attacker moved from the logging pod to the payments service in eleven minutes. The cluster's entire internal network was flat. Every pod implicitly trusted every other pod.

The data breach lasted six days before it was detected.

Every security failure in that incident had a specific technical solution that was not in place. This project implements all of them.

Who I Am and Why This Project Matters

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. Before I transitioned into tech in January 2024, I was a zoo volunteer, an automotive technician apprentice, and a beauty and cosmetics store manager.

I build these projects not to check portfolio boxes but to understand the systems deeply enough to operate them under pressure. Zero-trust is one of those concepts that sounds simple in theory and reveals enormous complexity in practice.

This article walks through every component of the platform, every architectural decision, and every error I hit along the way. My goal is that after reading this, you can build this platform yourself and explain every decision to a CISO or a CTO with confidence.

What Zero-Trust Actually Means in a Kubernetes Context

Zero-trust is not a product you buy. It is an architectural principle you implement across multiple layers.

The core idea is simple: trust nothing by default, verify everything explicitly, assume the perimeter has already been breached. Design your systems so that a compromised component cannot be used to compromise other components.

The attacker who compromised the London company had kubeconfig access. In a zero-trust cluster, that exec attempt would have triggered an immediate CRITICAL alert from Falco. The lateral movement would have been impossible because there was no path between the logging pod and the payments service.

The Five Security Layers

Layer 1 — Cilium eBPF (Network Visibility)

Cilium replaces kube-proxy entirely using eBPF technology. eBPF allows programs to run inside the Linux kernel without modifying kernel source code. Cilium uses this to provide L7 network visibility, policy enforcement, and load balancing at the kernel level.

💡 Install Cilium before Istio. Cilium provides the networking foundation that the service mesh runs on. Installing out of order causes Istio's sidecar injection webhook to fail with misleading timeout errors.

helm install cilium cilium/cilium \
  --version 1.14.5 \
  --namespace cilium-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=YOUR_EKS_ENDPOINT \
  --set k8sServicePort=443 \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

The kubeProxyReplacement=strict setting is critical. It tells Cilium to completely replace kube-proxy rather than running alongside it, giving Cilium full control over all service routing decisions at the eBPF level.

Layer 2 — Istio mTLS (Encryption + Identity)

Istio injects an Envoy sidecar proxy into every pod in labeled namespaces. This proxy intercepts all inbound and outbound network traffic. When two pods communicate, their Envoy sidecars perform a TLS handshake that proves both pod identities using certificates issued by the Istio certificate authority.

The key configuration is the PeerAuthentication policy:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: global-mtls
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

STRICT mode means Istio will reject any connection that is not mTLS. A pod without an Istio sidecar cannot talk to a pod with one.

🔐 This is the configuration that would have stopped the London attack. The attacker's connection from the exec session would not have had an Istio certificate. The payments service would have rejected it at the network layer.

Layer 3 — Istio AuthorizationPolicy (Request Authorization)

mTLS proves who you are. AuthorizationPolicy decides what you are allowed to do. The zero-trust default is deny everything — then add explicit allow rules only for the communication paths that are actually needed.

# Deny everything in the namespace first
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: zero-trust-apps
spec: {}
---
# Then explicitly allow only what is needed
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: zero-trust-apps
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/zero-trust-apps/sa/frontend"
      to:
        - operation:
            ports: ["8080"]
            methods: ["GET", "POST"]

The principal field is the SPIFFE identity of the frontend service account. Even if an attacker gets into a different pod with a different service account, they cannot call the backend. Their SPIFFE identity will not match the allow rule.

Layer 4 — OPA Gatekeeper (Admission Control)

Gatekeeper enforces policies at the moment a resource is created. Before any pod reaches a node, Gatekeeper validates it against constraint templates written in Rego. A pod that violates any constraint is rejected before it ever runs.

The most important constraint for zero-trust is the one that blocks privileged containers:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: DisallowPrivileged
metadata:
  name: disallow-privileged-zero-trust
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - zero-trust-apps

A privileged container has root access to the host node. In a zero-trust model, this is never acceptable. Gatekeeper blocks it before the container ever starts.

Layer 5 — Falco (Runtime Detection)

Falco uses eBPF to watch every system call made by every process in every container. When a pattern matches a rule, Falco generates an alert.

The three most important rules for zero-trust:

- rule: Privileged Exec in Zero-Trust Pod
  condition: >
    spawned_process and
    k8s.ns.name = "zero-trust-apps" and
    proc.name in (bash, sh, zsh) and
    proc.pname in (runc, containerd-shim)
  output: >
    Shell spawned in zero-trust pod via exec
    (namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: CRITICAL

Implementation — Building the Platform

The Installation Order Problem

This is the most important thing I can tell you about building this platform. The installation order matters. Not slightly. Critically.

Cilium → Istio → Gatekeeper → Constraint Templates → Constraints → Falco → Vault

On my first attempt I installed Istio before Cilium. Istio's control plane came up but the sidecar injection webhook kept failing. The error message pointed to a timeout in the webhook call but the actual cause was that Cilium had not finished taking over service routing from kube-proxy. Istio's webhook traffic was not being routed correctly.

The fix was uninstalling Istio, waiting for Cilium to fully initialize, then reinstalling Istio. This cost me ninety minutes.

The 3/3 Verification

After deploying the application, run:

kubectl get pods -n zero-trust-apps

You should see 3/3 Ready for every pod — three containers running simultaneously:

The Health Check Problem

When I applied the deny-all AuthorizationPolicy, all pods immediately failed their readiness probes and entered a crash loop.

The readiness probe makes an HTTP request from the kubelet. The kubelet does not have an Istio sidecar. So its requests do not have mTLS certificates. So they are rejected by the deny-all policy. So the pod fails its health check. So Kubernetes restarts it. So it fails again. Infinite loop.

The fix was adding explicit allow rules for the health check paths:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-health-checks
  namespace: zero-trust-apps
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
    - to:
        - operation:
            paths: ["/health", "/ready"]

⚠️ This was the most operationally painful debugging session of the entire project. The error from kubectl describe pod just showed the pod failing its readiness probe — no indication that the cause was an Istio authorization policy rejecting the kubelet's health check request.

Key Lessons From This Build

These are the four things that cost me the most time and that no tutorial warned me about:

The Real Lesson

Here is the thing about zero-trust that every article glosses over.

Zero-trust does not make your cluster secure. Zero-trust makes your cluster auditable.

The London financial services attack succeeded because nothing was watching. There were no logs of which pod talked to which. There were no alerts on unusual connection patterns. There was no record of which service account made which API call.

When you implement this zero-trust stack, you gain something more valuable than blocking power. You gain visibility:

• Every mTLS handshake creates a certificate log

• Every authorization policy decision creates an access log

• Every Falco rule match creates an alert

• Every Gatekeeper denial creates an event

The attacker who gets into your cluster will leave a trail you can follow. The lateral movement attempt will trigger a Falco alert before it succeeds. The anomalous service-to-service call will be blocked by authorization policy and logged. The exec attempt into a production pod will generate a CRITICAL alert that pages on-call within sixty seconds.

You cannot prevent every breach. You can make sure that every breach is detectable, containable, and attributable. That is what zero-trust actually delivers.

The London company did not have logging. They did not know they had been breached for six days. This platform would not have prevented the initial compromise. But it would have made the lateral movement impossible and detected the breach attempt within minutes.

Production Gap Analysis

This platform demonstrates the complete zero-trust security model. A production deployment at enterprise scale would additionally require the following.

Istio Gateway with cert-manager provides external traffic ingress with automatic TLS certificate provisioning and rotation. This replaces any direct pod exposure with a controlled entry point that participates in the mesh.

Falco Sidekick routes alerts to PagerDuty, Slack, and SIEM systems simultaneously. Console logs are not a production alerting mechanism — alerts need to reach the right people through the right channels within seconds of detection.

CiliumNetworkPolicy adds L3/L4 network policies on top of Istio's L7 authorization. Defense in depth at both layers eliminates an entire class of service mesh bypass techniques that operate at the TCP level below Istio's visibility.

Vault HA with 5-node Raft cluster replaces the single-replica Vault in this project. A single Vault pod is a single point of failure for all secret injection in the cluster. See Project 8 for the full Vault HA implementation.

OPA policy unit testing with Conftest and Gator ensures constraint template Rego logic is correct before it reaches production. An incorrect Rego policy that blocks legitimate workloads is an operational emergency. Test your policies before deploying them.

Security Posture at a Glance

How This Helps Companies

The financial services company that paid $4.3 million was running a standard Kubernetes cluster with flat networking. Not because their engineers were careless. Because zero-trust Kubernetes is complex to implement correctly and the documentation for doing it end-to-end is fragmented across five different tool ecosystems.

This project integrates all five layers into a single coherent platform. The implementation patterns demonstrated here apply directly to any organization running Kubernetes workloads that handles sensitive data, serves regulated industries, or has enterprise clients who require security attestation before signing contracts.

Full Source Code

The complete code is available at:

🔗 github.com/Eaglewings966/istio-zero-trust-eks

When you are done testing, clean up your environment:

kubectl delete -f apps/ -R --ignore-not-found=true
helm uninstall vault falco gatekeeper istiod cilium \
  -n vault,falco,gatekeeper-system,istio-system,cilium-system
eksctl delete cluster \
  --name emmanuel-zt-platform --region us-east-1

What's Next

Project 11 covers the Automated SOC2/CIS Compliance Pipeline — OpenSCAP scanning, Ansible auto-remediation, and a CI/CD gate that blocks deployments when the compliance score drops below 85%.

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

The Attack That Changed How the World Thinks About Software

In December 2020, security researchers at FireEye discovered something that made the entire industry stop and reconsider everything they thought they knew about software security.

SolarWinds, a company that makes network monitoring software used by 33,000 organizations including the US Pentagon, NASA, and Fortune 500 companies, had been compromised. Attackers had inserted malicious code into SolarWinds' own build pipeline. The malicious version was compiled, packaged, and signed by SolarWinds' legitimate signing process. The update was distributed through official channels. 18,000 organizations downloaded and installed it.

The attack was undetected for nine months.

The reason it was undetected for so long is the same reason traditional code signing failed to catch it. The software was signed. The signature verified correctly. Signature verification told every customer that the software was legitimate — when it was not. The signing process itself had been compromised.

The investigation and remediation cost exceeded $40 million. The scope of data exfiltrated is still not fully known.

The solution to the SolarWinds attack model is not better signing. It is transparent signing. Every signature must be recorded in a publicly auditable log that anyone can query. Any deviation from the expected signing identity is immediately detectable. No private signing key is stored anywhere that an attacker can find it.

That is what this project builds.

Who I Am and Why I Built This

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. I transitioned into tech in January 2024 after working as a zoo volunteer, an automotive technician apprentice, and a beauty and cosmetics store manager.

I came into this field understanding what counterfeiting looks like. In Lagos markets, you learn quickly that a product can look authentic, have the right label, and still be fake. The question is not whether it looks real. The question is whether there is a verifiable chain of custody that proves it is real.

Container image supply chain security is the same problem. An image can be signed and still be malicious if the signing process was compromised. What you need is a chain of custody — cryptographic proof of exactly who signed what, when, from which pipeline, and a public record that anyone can audit.

This article walks through every component of that chain. I will explain the architecture, the implementation decisions, and the problems I ran into along the way.

What Is Container Image Supply Chain Security

Before I show you what I built, I want to ground this in plain language because the terminology in this space can feel opaque.

A container image is a packaged version of your application. Everything it needs to run is inside it. When Kubernetes deploys your application, it pulls this image and runs it.

The supply chain is every step between when a developer writes code and when that code runs in production. The source code is part of the supply chain. The build pipeline is part of the supply chain. The container registry is part of the supply chain. Every step is a potential point of compromise.

Supply chain security means establishing cryptographic proof at every step and enforcing that proof before anything runs.

💡 Think of it like the supply chain for pharmaceutical drugs. A medicine package has a barcode, a lot number, a manufacturer certificate, and a chain of custody document. A pharmacist does not just check whether the medicine looks like medicine — they verify the chain. Every step from manufacturing to dispensing is documented and verifiable. Container image supply chain security is that process, implemented in software, enforced automatically, and recorded in a public log that anyone can audit.

What I Built

A complete container image supply chain security platform with four integrated components.

Architecture Decisions

Why Keyless Signing Over Key-Based Signing

Traditional code signing requires a private key. That private key must be stored somewhere — in GitHub Secrets, in a key management system, or in a hardware security module. Wherever it lives, it is a target.

SolarWinds had a signing key. The attackers used it.

Keyless signing changes the threat model fundamentally. Instead of a private key, the signing credential is a short-lived OIDC token issued by GitHub Actions for the duration of a single job. The token proves the signing happened from a specific GitHub repository, a specific workflow file, and a specific git branch. When the job ends, the token expires. There is nothing to steal.

If an attacker compromises the build pipeline and tries to sign a malicious image, the OIDC subject in the certificate will not match the expected workflow path. Verification will fail. The anomalous entry in Rekor is publicly visible.

Why Dual Enforcement With Kyverno and OPA Gatekeeper

Two admission controllers for one cluster sounds redundant. It is not.

Kyverno handles image signature verification natively. Its policy language is YAML-based and purpose-built for Kubernetes resource validation. It is the right tool for checking whether an image was signed by the correct identity.

OPA Gatekeeper handles registry allowlisting using Rego policies. Rego is a general-purpose policy language that can express constraints that Kyverno's YAML-based policies cannot — complex logic about which registries are approved, which image name patterns are permitted, and what combinations of attributes are required.

Together they form a defense-in-depth admission chain:

For an admission control layer that is the last line of defense before untrusted code runs in the cluster, redundancy is not optional.

Why SBOM in Both SPDX and CycloneDX

Generating both adds thirty seconds to the pipeline. It ensures compatibility with every downstream consumer regardless of their toolchain preference. There is no reason not to generate both.

Implementation

Step 1 — The Signing Pipeline

Here is the core of the supply chain pipeline. The signing job uses Cosign's keyless mode with COSIGN_EXPERIMENTAL enabled:

- name: Sign ECR image with Cosign keyless
  env:
    COSIGN_EXPERIMENTAL: "1"
  run: |
    cosign sign --yes \
      ${{ needs.build.outputs.ecr-image }}

    echo "Signature recorded in Rekor transparency log"

The --yes flag accepts the Rekor upload without prompting. The COSIGN_EXPERIMENTAL: "1" flag enables keyless signing mode. When this command runs, it:

1. Requests an OIDC token from GitHub

2. Sends it to Fulcio to get a short-lived certificate

3. Signs the image digest with that certificate

4. Records both the signature and the certificate in Rekor

By the time this job completes, the signature is in a public transparency log that anyone in the world can query.

Step 2 — SBOM Generation and Attachment

- name: Generate SBOM for ECR image
  run: |
    syft ${{ needs.build.outputs.ecr-image }} \
      -o spdx-json=sbom-ecr.spdx.json \
      -o cyclonedx-json=sbom-ecr.cyclonedx.json

    echo "Total packages: $(cat sbom-ecr.spdx.json | jq '.packages | length')"

- name: Attach SBOM to image with Cosign
  env:
    COSIGN_EXPERIMENTAL: "1"
  run: |
    cosign attach sbom \
      --sbom sbom-ecr.spdx.json \
      ${{ needs.build.outputs.ecr-image }}

Syft scans the image layers and produces a machine-readable inventory of every package. The SBOM is then attached to the image using Cosign, stored as an OCI artifact in the same registry alongside the image. Anyone who pulls the image can also pull and verify the SBOM without any additional tooling beyond Cosign.

Step 3 — Kyverno Policy Enforcement

The ClusterPolicy that enforces signature verification:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [demo-app]
      verifyImages:
        - imageReferences:
            - "ghcr.io/eaglewings966/*"
            - "*.dkr.ecr.us-east-1.amazonaws.com/*"
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/Eaglewings966/cosign-image-supply-chain/.github/workflows/supply-chain-pipeline.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"

The subject field is the most important part of this policy. It specifies the exact GitHub Actions workflow path and branch that is permitted to sign images. An image signed by any other workflow, repository, or branch will fail verification and the pod will be blocked.

🔐 This is what would have caught the SolarWinds attack. The attackers used SolarWinds' own signing credentials. But they could not have had the specific GitHub Actions OIDC identity for a specific workflow file at a specific git ref. The subject mismatch would have been caught before any image ran.

Step 4 — Testing the Enforcement

Attempting to deploy an unsigned image:

# Try to deploy an unsigned nginx image
kubectl apply -f unsigned-test-deployment.yaml

# Expected response from Kyverno:
# Error from server: error when creating "unsigned-test-deployment.yaml":
# admission webhook "mutate.kyverno.svc-fail" denied the request:
# resource Deployment/demo-app/unsigned-test was blocked
# due to the following policies: verify-image-signatures:
# image nginx:latest failed signature verification

The admission webhook fires before the deployment reaches the Kubernetes scheduler. The unsigned image never touches a node. It is rejected at the API server level.

Then deploying the signed image:

kubectl apply -f apps/demo-app/deployment.yaml

kubectl get pods -n demo-app
# NAME                        READY   STATUS    RESTARTS   AGE
# demo-app-7d9f8b6c4-xk2pq   1/1     Running   0          45s

The signed image deploys successfully. Kyverno verified the signature, confirmed the OIDC subject matched the expected workflow, and allowed the pod to schedule.

What Broke During This Build

Debugging Story 1 — The Subject Mismatch

My first attempt at the Kyverno policy used a simplified subject pattern:

subject: "https://github.com/Eaglewings966/cosign-image-supply-chain"

Every signed image was blocked. Kyverno returned a verification failure with a cryptic error about the attestor chain not matching.

The actual subject recorded in the Rekor entry is the full workflow path including the specific workflow file and the git ref:

https://github.com/Eaglewings966/cosign-image-supply-chain/.github/workflows/supply-chain-pipeline.yml@refs/heads/main

The Kyverno policy subject must match this exactly. The way to find the exact subject is to run cosign verify on a signed image and look at the subject field in the JSON output. That subject is what goes into the policy.

⚠️ The error message from Kyverno does not tell you what subject it found versus what it expected. You have to query Rekor directly to see the actual subject. Forty minutes of verification failures before I found the exact subject string.

Debugging Story 2 — Gatekeeper Webhook Registration Race

After deploying Gatekeeper on Fargate, the constraint templates failed to install with:

Error from server (InternalError): error when creating
"require-signed-images-template.yaml": Internal error occurred:
failed calling webhook "webhook.gatekeeper.sh": failed to call
webhook: Post https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit:
dial tcp: connection refused

Gatekeeper's own webhook was not ready when I tried to install the constraint templates. On Fargate, the cold start time for Gatekeeper pods is longer than on EC2. The webhook server was still initializing when kubectl apply was attempting to send resources through it.

The fix was waiting for all Gatekeeper pods to show Running and their health endpoints to return 200 before applying any constraint templates.

💡 In production, replace the sleep 60 with a proper readiness check against the webhook endpoint.

Debugging Story 3 — SBOM Attach on Immutable Tag

When I tried to attach the SBOM to the ECR image, cosign attach returned an error about the image tag being immutable.

ECR's immutable tag feature prevents any push to an existing tag. Cosign's attach command attempts to push an OCI artifact referencing the same tag — ECR treats this as a tag modification attempt and blocks it.

The solution is to attach the SBOM using the image digest instead of the tag:

# Resolve tag to digest first
DIGEST=\((crane digest \){IMAGE_TAG})

# Attach SBOM by digest
cosign attach sbom --sbom sbom.spdx.json \({IMAGE_REGISTRY}/\){REPO}@${DIGEST}

The digest is immutable by nature. Attaching by digest creates a new OCI artifact in the registry that references the image digest without modifying the existing tag.

The Real Lesson

Here is the thing about supply chain security that most tutorials skip entirely.

Signing an image proves that you signed it. It does not prove that the image is safe.

Traditional signing tells you who signed something. Transparent signing tells you who signed something, when, from which exact build context, and records that information in a log that anyone can audit for anomalies.

The difference is audibility at scale.

If someone compromises your build pipeline and signs a malicious image with a private key stored in GitHub Secrets, the traditional signature says the image is legitimate. A verifier cannot tell the difference between a legitimate build and a compromised build if both used the same key.

With keyless signing tied to OIDC identity, the compromised build would have signed with a different OIDC subject, or from a different workflow file, or from a different branch. Any of those deviations is visible in Rekor. A monitoring system that watches Rekor for unexpected signing events from your repository would catch it.

The SolarWinds attackers had the signing key. They could not have had the correct OIDC identity for a specific workflow file at a specific git ref. That specificity is what transparent signing gives you that traditional signing does not.

Supply chain security is not about trust. It is about verifiability. Build systems where every assertion can be independently verified by anyone without trusting you.

Production Gap Analysis

This platform demonstrates the core supply chain security model. A production deployment at enterprise scale would additionally require the following.

SLSA Level 3 provenance via the slsa-github-generator action provides cryptographic proof of the full build process including the source code commit, the build inputs, and the build environment. SLSA Level 3 is the minimum required for US federal software procurement under Executive Order 14028.

Sigstore Policy Controller replaces the Kyverno image verification component with a dedicated controller designed specifically for Cosign verification. Policy Controller has deeper integration with the Sigstore ecosystem and supports more complex verification policies including multi-party signing requirements.

Rekor monitoring via a dedicated service that watches the Rekor transparency log for unexpected signing events from your repositories. An unexpected signing event is an early indicator of build pipeline compromise.

Dependency-Track integration with the SBOM provides continuous vulnerability monitoring of every package in every deployed image. When a new CVE is published that affects a package in a deployed image's SBOM, Dependency-Track generates an alert without requiring a new image build.

Security Posture at a Glance

How This Helps Companies

The SolarWinds attack was not unique. Supply chain attacks have increased by over 700% since 2020. The attack vector is effective because it bypasses traditional perimeter security entirely. If you trust the build pipeline, you trust the output.

This platform establishes cryptographic proof at every step of the delivery process and enforces that proof before any code runs in the cluster. It does not eliminate supply chain attacks. It makes them detectable, auditable, and ultimately ineffective against a properly configured enforcement layer.

Full Source Code

The complete code is available at:

🔗 github.com/Eaglewings966/cosign-image-supply-chain

When you are done testing, clean up your environment:

helm uninstall policy-reporter -n kyverno-policy-reporter
helm uninstall gatekeeper -n gatekeeper-system
helm uninstall kyverno -n kyverno
cd terraform && terraform destroy --auto-approve
eksctl delete cluster \
  --name supply-chain-cluster --region us-east-1

What's Next

Project 10 in this series covers the Zero-Trust Kubernetes Platform — Istio mTLS, OPA Gatekeeper, Falco runtime security, and Vault secrets injection.

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

The Breach That Started With a Hardcoded Secret

In March 2020, an engineer at a US healthcare company pushed a configuration file to a private GitHub repository. The file contained an AWS access key. The key had been there for four months. GitHub's secret scanning had not been enabled. The repository was private but not air-gapped from the internet.

A credential harvesting bot found it within eleven minutes.

By the time the security team discovered the breach two weeks later, the attacker had exfiltrated 3.1 million patient records using the access key's S3 permissions. The key had never been rotated. It had a two-year expiry. Nobody knew it existed in that file until the breach investigation surfaced it.

The HIPAA fine was $6.85 million.

The access key was not special. It was the same kind of static credential that lives in thousands of Kubernetes secrets, CI/CD pipelines, environment variables, and configuration files across the industry right now. The only difference between that company and most others is that their breach was discovered.

Static credentials are not a configuration problem. They are an architectural problem. The solution is not better rotation policies. The solution is eliminating static credentials entirely and replacing them with dynamic ones that expire automatically.

That is what this project builds.

Who I Am and Why I Built This

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. I came into tech in January 2024 after working as a zoo volunteer, an automotive technician apprentice, and a beauty store manager.

I built this project because secrets management is one of the most underbuilt parts of cloud infrastructure. Teams spend weeks on CI/CD pipelines and days on monitoring stacks but leave database passwords in Kubernetes secrets and AWS access keys in GitHub Actions variables.

This article walks through the full implementation of a production-grade Vault HA cluster. I will explain every architectural decision, every configuration detail, and every problem I encountered. Whether you are implementing this for the first time or evaluating whether your current Vault setup matches production standards, this article gives you both the what and the why.

What Static Credentials Actually Cost

Before I show you what I built, I want to make the business case concrete — because this is a conversation you may need to have with a CTO or an engineering manager.

Static credentials have four failure modes that compound over time.

Theft. A credential stored anywhere can be stolen from there. It can be found in git history, leaked through logs, extracted from a compromised pod, or obtained through a phishing attack on the engineer who created it.

Rotation failure. Manual rotation is a human process. Humans forget. Humans rotate in the wrong environment. Humans rotate without updating all the systems that depend on the credential. Every manual rotation is a potential outage.

Scope creep. A credential created for one purpose gets reused for another. The S3 read key becomes the key that also writes. The database read user becomes the one running migrations. Permissions accumulate because creating new credentials is inconvenient.

Attribution loss. Six months after an access key is created, nobody remembers what it was for, who created it, or what it has access to. It sits in the account as a permanent attack surface that nobody wants to revoke because nobody knows what will break.

Dynamic credentials eliminate all four failure modes at the architectural level:

What I Built

A five-node HashiCorp Vault HA cluster on AWS EKS Fargate with four secrets engines and full sidecar injection for all workloads.

The Cluster

Five Vault server pods using Raft integrated storage for consensus. No external storage dependency. Vault manages its own distributed state. AWS KMS auto-unseal — when any Vault pod restarts, it calls the KMS key to unseal itself automatically. No human intervention. No on-call engineer woken up to type unseal keys.

The Four Secrets Engines

The Injection Model

Vault Agent injector handles sidecar injection for all workloads. Applications read secrets from files in /vault/secrets/. They never call the Vault API. They never handle token renewal. The agent sidecar does all of it.

Why Five Replicas

The most common question I get about this architecture is why five replicas instead of three.

Raft requires a quorum of floor(n/2)+1 nodes to operate:

With three nodes, lose one and you are at exactly the minimum. One more failure and Vault stops accepting writes. You are one bad pod restart away from a split-brain scenario.

With five nodes, you can lose two nodes simultaneously and continue operating normally. During planned maintenance this means you can take down one node for an upgrade while being fully protected against an unplanned failure of a second node at the same time.

For a secrets platform — the infrastructure that everything else depends on — the extra two pods are not overhead. They are the difference between a system that can tolerate real production conditions and one that can only tolerate lab conditions.

Implementation

Step 1 — Provision Infrastructure with Terraform

The Terraform configuration creates four categories of resources: a KMS key for auto-unseal, an IRSA role for the Vault service account, an RDS PostgreSQL instance for the dynamic credentials demonstration, and the Vault Helm release itself.

Here is the KMS key configuration:

resource "aws_kms_key" "vault_unseal" {
  description             = "Vault auto-unseal key"
  deletion_window_in_days = 7
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "EnableRootAccess"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "AllowVaultUnseal"
        Effect = "Allow"
        Principal = {
          AWS = aws_iam_role.vault_server.arn
        }
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:DescribeKey"
        ]
        Resource = "*"
      }
    ]
  })
}

The key policy scopes KMS access to exactly the Vault server role. No other IAM principal can call this key. If an attacker compromises a different role in the account, they cannot use it to unseal Vault.

Step 2 — Configure Vault HA with Raft

The Vault Helm values configure the five-node Raft cluster. The most important section is the Raft storage configuration with all five retry_join addresses:

storage "raft" {
  path    = "/vault/data"
  node_id = "{{ .NodeID }}"

  retry_join {
    leader_api_addr = "http://vault-0.vault-internal:8200"
  }
  retry_join {
    leader_api_addr = "http://vault-1.vault-internal:8200"
  }
  retry_join {
    leader_api_addr = "http://vault-2.vault-internal:8200"
  }
  retry_join {
    leader_api_addr = "http://vault-3.vault-internal:8200"
  }
  retry_join {
    leader_api_addr = "http://vault-4.vault-internal:8200"
  }
}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "${VAULT_AWSKMS_SEAL_KEY_ID}"
}

Every node tries to join all other nodes on startup. Whichever node comes up first becomes the initial leader. The others join as followers. When the leader goes down, Raft holds an election and one of the followers becomes the new leader in five to ten seconds. This is transparent to the Vault clients.

Step 3 — Initialize Vault

Initialization happens once and only once per cluster. With KMS auto-unseal, Vault uses recovery keys instead of unseal keys. These recovery keys are only needed if KMS itself becomes unavailable.

kubectl exec -it vault-0 -n vault -- \
  vault operator init \
    -key-shares=5 \
    -key-threshold=3 \
    -recovery-shares=5 \
    -recovery-threshold=3 \
    -format=json

🔐 The recovery keys must be stored offline. In a physical safe. Not in AWS Secrets Manager. Not in another Kubernetes secret. If AWS KMS is the emergency — the only scenario where you need these keys — you cannot reach AWS to retrieve them. Paper. Safe. Off-site backup. This is not optional.

Step 4 — Configure Dynamic Database Credentials

This is the most operationally significant configuration in the entire project. After this step, no human can log into the PostgreSQL database. Only Vault can.

# Configure the database connection
vault write database/config/demo-postgres \
  plugin_name="postgresql-database-plugin" \
  allowed_roles="demo-app-db-role" \
  connection_url="postgresql://{{username}}:{{password}}@${RDS_ENDPOINT}:5432/vaultdemo" \
  username="vaultadmin" \
  password="${DB_PASSWORD}"

# Rotate root credentials — Vault now owns the password
vault write -force database/rotate-root/demo-postgres

# Create the dynamic role
vault write database/roles/demo-app-db-role \
  db_name="demo-postgres" \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

After rotate-root, the original master password no longer works. Vault generated a new random password and stored it internally. Every application that needs database access now requests dynamic credentials from Vault. Each request creates a new PostgreSQL role with a unique name and password, valid for one hour. When the TTL expires, Vault drops the role automatically.

Step 5 — Vault Agent Sidecar Injection

The Vault Agent injector intercepts pod creation through a Kubernetes mutating admission webhook. When it sees specific annotations on a pod, it injects an init container that fetches secrets and a sidecar container that keeps them fresh.

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "demo-app"

  # KV secret injected as JSON file
  vault.hashicorp.com/agent-inject-secret-config.json: >-
    secret/data/apps/demo-app/config

  # Dynamic database credentials injected as env file
  vault.hashicorp.com/agent-inject-secret-db-creds: >-
    database/creds/demo-app-db-role
  vault.hashicorp.com/agent-inject-template-db-creds: |
    {{- with secret "database/creds/demo-app-db-role" -}}
    DB_USERNAME="{{ .Data.username }}"
    DB_PASSWORD="{{ .Data.password }}"
    {{- end }}

The application container reads /vault/secrets/db-creds as a file. It never calls Vault. It never handles token renewal. When the database credential TTL approaches expiry, the Vault Agent sidecar fetches a new credential and overwrites the file. The application reads the file on its next database connection. The entire rotation is transparent to the application code.

What Broke During This Build

Debugging Story 1 — The Raft Pods That Would Not Join

After deploying the Helm release, four of the five Vault pods were stuck in 0/1 Not Ready. Only vault-0 showed Ready.

The issue was that the other pods were trying to join the Raft cluster but vault-0 was not yet initialized. Raft requires the initial node to be initialized before followers can join. The retry_join configuration handles this by retrying indefinitely, but the pods showed as not ready because the readiness probe was failing while they waited.

The fix was waiting for vault-0 to become ready before running the initialization script. Once vault-0 was initialized and unsealed via KMS, the other four pods joined the Raft cluster within seconds.

💡 Raft cluster formation is sequential on first deployment. All pods showing not-ready before initialization is expected behaviour, not a configuration error.

Debugging Story 2 — Database Root Rotation Locked Me Out

During my first attempt at configuring the database secrets engine, I ran vault write -force database/rotate-root/demo-postgres before testing that vault read database/creds/demo-app-db-role actually worked.

The root rotation succeeded. But the dynamic credential generation failed because I had a syntax error in the creation_statements SQL template. The role could not be created. And now I could not fix it by logging into the database directly because Vault had rotated the root password and I no longer had it.

The only recovery path was destroying the RDS instance and creating a new one with a known password.

⚠️ Test dynamic credential generation completely before rotating root credentials. Once you rotate, there is no going back. The root password exists only inside Vault's encrypted storage.

Debugging Story 3 — IRSA Trust Policy Mismatch

The Vault pods were starting but immediately failing with permission errors on the KMS API calls. The IAM role existed. The KMS key policy allowed the role. But the calls were being denied.

The issue was in the IRSA trust policy condition. The service account name in the condition was vault-server but the actual service account created by Helm was vault. The OIDC subject did not match, so AWS rejected the AssumeRoleWithWebIdentity call silently.

The fix was updating the trust policy condition to match the exact service account name.

💡 The error message from AWS does not tell you what the condition mismatch was — it just returns AccessDenied. Debugging IRSA issues requires comparing the actual service account name and namespace against the exact strings in the trust policy condition.

Production Gap Analysis

This cluster demonstrates the full Vault capability set. A production deployment at enterprise scale would additionally require the following.

Vault Enterprise namespaces provide multi-tenancy within a single Vault cluster. Each team or business unit gets their own namespace with isolated policies, auth methods, and secrets engines. Without namespaces, a single Vault cluster becomes a shared credential store where one team can potentially access another team's secrets through policy misconfiguration.

Vault Sentinel policies provide fine-grained governance beyond what HCL policies support. Sentinel can enforce conditions like requiring MFA for destructive operations, blocking credential generation outside business hours, and enforcing minimum TTL values on all issued credentials.

Cross-region Vault replication via Vault Enterprise Performance Replication provides read-scalability and disaster recovery across regions. The primary cluster handles all writes. Secondary clusters serve reads from local replicas. If the primary region becomes unavailable, a secondary can be promoted.

Break-glass procedures need to be documented, tested, and stored in a physical secure location before the cluster goes to production. The recovery keys printed during initialization are the only way to recover if both AWS KMS and the Vault cluster state become unavailable simultaneously.

Security Posture at a Glance

How This Helps Companies

The healthcare company that paid a $6.85 million HIPAA fine was not technically incompetent. They were architecturally unprepared for the risk profile of static credentials. The access key that caused the breach was created by a developer solving a specific problem. It worked. Nobody thought about what happened if it was found.

Vault does not make developers more careful. It makes the dangerous pattern impossible. When credentials expire in an hour, there is nothing to find. When root database passwords are unknown even to the team that set them up, there is nothing to steal. When every credential request is logged with identity, timestamp, and lease ID, there is a full audit trail for every access event.

Full Source Code

The complete code is available at:

🔗 github.com/Eaglewings966/hashicorp-vault-eks

When you are done testing, clean up your environment:

helm uninstall vault -n vault
kubectl delete namespace demo-app vault --ignore-not-found=true
cd terraform && terraform destroy --auto-approve
eksctl delete cluster --name vault-eks-cluster --region us-east-1

What's Next

Project 9 in this series covers Container Image Supply Chain Security — Cosign keyless signing, Syft SBOM generation, and Kyverno admission control enforcing signatures before any unsigned image reaches the cluster.

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

In September 2021, a fintech company in Singapore deployed a Docker image to production. The image contained a critical vulnerability in its base OS layer. The vulnerability had been publicly known for three weeks. There was a patch available. Nobody had scanned the image.

The company had a CI pipeline. It ran unit tests. It checked code coverage. It validated formatting. It did not check the security of the container it was building.

Six weeks after the deployment, an attacker exploited the vulnerability. The breach exposed 2.3 million customer records. The regulatory fine was $4.2 million. The reputational damage was irreversible.

A container vulnerability scanner in the CI pipeline would have blocked that deployment on the day the image was built. The scan takes under two minutes. The difference between running it and not running it was $4.2 million and 2.3 million compromised customers.

This is the problem I built a solution for.

Who I Am and Why Security Matters to Me

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. Before I transitioned into tech in January 2024, I was a zoo Tourist Guard volunteer, an automotive technician apprentice, and a beauty store manager.

I tell you this because I came into cloud engineering understanding what it means to operate with consequences. In my previous work, mistakes had real costs. A misdiagnosed engine problem meant a customer paying for repairs that did not fix anything.

The mindset I carry into DevSecOps is the same. A missed vulnerability in a pipeline is not a theoretical risk. It is a timer counting down to an incident.

The pipeline I am describing in this article implements security scanning at every stage of the software delivery lifecycle. Source code, dependencies, infrastructure, container image, and running application. Five different attack surfaces. Five different tools. One integrated pipeline.

What Shift Left Actually Means

Before I walk through the implementation, I want to explain the concept of shift left in terms that make the business case clear for both engineers and non-technical readers.

Security vulnerabilities exist in software at every stage of the delivery process. They can be found in source code, in the libraries the code depends on, in the infrastructure configuration, in the container image that packages the code, or in the running application itself.

The earlier in the pipeline you find a vulnerability, the cheaper it is to fix:

Shift left means moving security checks as early in the pipeline as possible. The fintech company in Singapore had security checks at zero stages. This pipeline has security checks at five stages before the image ever reaches production.

What I Built

An eight-stage DevSecOps pipeline using GitHub Actions that scans every attack surface before allowing an image to reach the container registry. Everything provisioned with Terraform. OIDC authentication to AWS — no static credentials anywhere.

Architecture Decisions

Why Trivy Blocks on CRITICAL and HIGH Only

The blocking threshold is the most consequential policy decision in a DevSecOps pipeline. Get it wrong in one direction and the pipeline becomes a rubber stamp. Get it wrong in the other direction and engineers bypass it.

• CRITICAL (9.0–10.0 CVSS): Public exploits exist, actively targeted, patching is non-negotiable

• HIGH (7.0–8.9 CVSS): Significant risk with exploits that are likely or proven

Blocking on CRITICAL and HIGH is the threshold used at Stripe, Monzo, and Cloudflare. MEDIUM and LOW findings are tracked in a security backlog and addressed on a scheduled remediation cycle.

⚠️ If you block on MEDIUM in a production environment, engineers will look for ways around the pipeline within a week. A policy that gets bypassed is worse than no policy because it creates false confidence.

Why OIDC Instead of Static AWS Credentials

Static AWS access keys stored in GitHub Secrets are a persistent attack surface. They must be rotated. They can be leaked through log output. They can be extracted through a compromised runner. They have a lifespan measured in months or years.

OIDC federation works differently:

• GitHub Actions requests a short-lived token from AWS's OIDC provider for each job run

• The token is valid only for the duration of that specific job

• No long-lived credentials to store, rotate, or protect

• The IAM role is scoped to the specific repository — only jobs from Eaglewings966/aws-devsecops-pipeline can assume the role

• A leaked token from one repository cannot be used to access another repository's AWS resources

Why ECR Image Tag Immutability

Mutable image tags are a silent security risk. If the latest tag can be overwritten, an attacker who gains push access to the registry can replace a known-good image with a malicious one without changing the tag. The deployment pulls latest, gets the malicious image, and there is no detectable deployment event to investigate.

Immutable tags make this attack impossible. Once an image is pushed with a specific tag, that tag cannot be reused. Any attempt to overwrite it fails at the API level.

Why DAST After Push Rather Than During Build

OWASP ZAP requires a running application. Running the container from the ECR image after push validates that the same image that will be deployed to production is the one being tested. Not a locally built version. Not a development build with different environment variables. The exact production image.

Implementation

Step 1 — Provision ECR and OIDC with Terraform

The Terraform configuration creates three resources: an ECR repository with immutable tags and scan on push enabled, an IAM OIDC identity provider for GitHub Actions, and an IAM role that GitHub Actions assumes via OIDC with ECR push permissions.

Here is the OIDC provider and IAM role:

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  ]
}

resource "aws_iam_role" "github_actions" {
  name = "devsecops-pipeline-github-actions-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          "token.actions.githubusercontent.com:sub" =
            "repo:Eaglewings966/aws-devsecops-pipeline:*"
        }
      }
    }]
  })
}

The StringLike condition on the subject claim scopes the role to a specific repository. This is what prevents other GitHub repositories from assuming this role even if they obtain a valid OIDC token.

cd terraform
terraform init
terraform apply --auto-approve
terraform output github_secrets_to_set

Step 2 — The Pipeline Trigger and Permissions

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

permissions:
  contents: read
  security-events: write
  id-token: write
  pull-requests: write

The id-token: write permission is what enables OIDC authentication. Without it, GitHub Actions cannot request the OIDC token needed to assume the AWS IAM role. The security-events: write permission allows SARIF results to be uploaded to the GitHub Security tab.

Step 3 — Semgrep SAST

sast-semgrep:
  name: SAST — Semgrep Code Scanning
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v4

    - name: Run Semgrep SAST scan
      uses: semgrep/semgrep-action@v1
      with:
        config: >-
          p/nodejs
          p/security-audit
          p/owasp-top-ten
          p/secrets
        generateSarif: "1"
      env:
        SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}

    - name: Upload SARIF results
      uses: github/codeql-action/upload-sarif@v3
      if: always()
      with:
        sarif_file: semgrep.sarif
        category: semgrep-sast

⚠️ The if: always() on the SARIF upload is critical. If Semgrep finds a vulnerability and fails the job, the upload step must still run to post the findings to the Security tab. Without always(), a failed scan produces no visible output for the engineering team.

Step 4 — Trivy Image Scan With Blocking Policy

image-scan-trivy:
  name: Image Scan — Trivy CVE Scanning
  needs: build

  steps:
    - name: Run Trivy — CRITICAL and HIGH block
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: \({{ env.REGISTRY }}/\){{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}
        format: sarif
        output: trivy-results.sarif
        severity: CRITICAL,HIGH
        exit-code: '1'
        ignore-unfixed: true
        vuln-type: os,library

    - name: Run Trivy — full report all severities
      uses: aquasecurity/trivy-action@master
      if: always()
      with:
        image-ref: \({{ env.REGISTRY }}/\){{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}
        format: table
        severity: CRITICAL,HIGH,MEDIUM,LOW
        exit-code: '0'

Two Trivy runs are intentional:

This gives the security team complete visibility into MEDIUM and LOW findings without stopping the deployment for issues that don't warrant it.

Step 5 — OWASP ZAP DAST

dast-owasp-zap:
  needs: push-to-ecr

  steps:
    - name: Start application container
      run: |
        docker run -d \
          --name target-app \
          -p 3000:3000 \
          \({{ env.REGISTRY }}/\){{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}

        for i in $(seq 1 30); do
          if curl -sf http://localhost:3000/health; then
            echo "Application is ready"; break
          fi
          sleep 2
        done

    - name: Run OWASP ZAP baseline scan
      uses: zaproxy/action-baseline@v0.10.0
      with:
        target: 'http://localhost:3000'
        fail_action: false

The readiness loop polls the health endpoint before starting the ZAP scan. A scan against an application that is still starting up produces false positives from connection refused errors. Waiting for the health check to return 200 before scanning ensures ZAP is testing the running application, not the startup sequence.

What Broke During This Build

Debugging Story 1 — OIDC Token Permission Missing

After setting up the OIDC provider and IAM role in Terraform and configuring the GitHub Actions workflow, the AWS authentication step failed with:

Error: Credentials could not be loaded, please check your
action inputs: Could not load credentials from any providers

The workflow had permissions: contents: read at the top level but was missing id-token: write. Without id-token: write, GitHub Actions does not generate the OIDC token that the AWS action needs to assume the IAM role. Adding the permission to the workflow permissions block resolved it immediately.

Debugging Story 2 — Trivy Scanning a Non-Existent Image

The Trivy scan job ran after the build job but scanned an image that was not available in the runner environment. The build job created the image but the Trivy job ran on a fresh runner with no local images.

The fix was uploading the Docker image as a GitHub Actions artifact after the build job and downloading it in the Trivy job before scanning. The artifact upload and download added about ninety seconds to the pipeline but made the image available across jobs without requiring an intermediate registry.

Debugging Story 3 — Checkov Failing on Valid Terraform

Checkov flagged the ECR repository for missing Lambda environment variable encryption — a rule that does not apply to ECR resources. The check was a false positive from Checkov applying a Lambda-specific rule to a non-Lambda resource.

The fix was adding the specific check ID to the skip list in the Checkov action configuration with a documented reason.

💡 Every skipped check must be documented with why it was accepted. A Checkov skip without documentation is a security debt that future engineers cannot evaluate.

Security Posture Improvements at a Glance

Production Gap Analysis

This pipeline covers the five most critical security scan layers. A production implementation at enterprise scale would additionally require the following.

Software Bill of Materials (SBOM) generation using Syft or Grype creates a machine-readable inventory of every package in the container image. SBOM generation is becoming a regulatory requirement in several jurisdictions and is required for US federal software procurement under Executive Order 14028.

Image signing with Cosign and verification at deployment time ensures that only images signed by the CI pipeline can be deployed to the Kubernetes cluster. A Gatekeeper admission webhook enforces this at the cluster level.

Defect Dojo or a similar vulnerability management platform aggregates SARIF results from multiple pipelines, tracks vulnerability state over time, assigns remediation ownership, and measures mean time to remediation. Without aggregation, security findings from individual pipelines have no organizational memory.

GitLeaks or TruffleHog as dedicated secret scanners provide deeper secret detection than Semgrep's secret rules. They detect secrets in git history in addition to current code — catching credentials that were committed and deleted but are still visible in the repository history.

How This Helps Companies

The fintech company in Singapore did not lack engineering talent. They lacked an automated security gate in their delivery pipeline. The vulnerability they shipped was in a public CVE database. A scanner would have found it in ninety seconds.

The pipeline in this repository implements the security controls that prevent that scenario. Not as a one-time audit. As an automated gate that runs on every push and every pull request.

These skills apply directly to any engineering team that ships containerized applications and wants to move from reactive security — finding vulnerabilities after they are exploited — to proactive security — catching them before they are deployed.

Full Source Code

The complete code is available at:

🔗 github.com/Eaglewings966/aws-devsecops-pipeline

When you are done testing, clean up your environment:

aws ecr batch-delete-image \
  --repository-name devops-demo-app \
  --image-ids "$(aws ecr list-images \
    --repository-name devops-demo-app \
    --query 'imageIds[*]' --output json)" \
  --region us-east-1

cd terraform && terraform destroy --auto-approve

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

In the spring of 2022, a healthtech startup in Amsterdam was three weeks away from closing a $12 million Series B round.

The lead investor required a security audit as a condition of closing. The startup engaged an external auditor for a two-week AWS security review. What the auditor found was not sophisticated. It was not the result of a targeted attack. It was the result of years of moving fast without governance.

The findings read like a checklist of everything that goes wrong when engineering teams manage AWS accounts without centralized security controls:

• Root account credentials had been used to create access keys twelve times in the previous eighteen months

• CloudTrail had been disabled in the staging account to reduce logging costs and never re-enabled

• Three IAM users had programmatic access keys that had not been rotated in over two years

• GuardDuty was not enabled in any of the four accounts

• Resources had been provisioned in six different AWS regions with no inventory of what was running where

• No centralized audit log. No compliance monitoring. No threat detection.

The investor pulled the term sheet two days before closing. The reason cited in the withdrawal letter was inadequate security posture for a company handling protected health information.

The engineering team spent the next four months implementing the controls that should have been in place from day one. The company raised at a lower valuation in a subsequent round nine months later.

Every finding in that audit is preventable with the architecture I am describing in this article. Not partially preventable. Completely preventable. With Terraform code that deploys in under fifteen minutes.

Who I Am and Why This Architecture Matters to Me

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. I transitioned into tech in January 2024 after working as a zoo volunteer, an automotive technician apprentice, and a beauty store manager.

I tell you this because I came into cloud engineering without the luxury of learning security practices gradually through a senior engineer's mentorship. I had to understand from first principles why these controls exist, what they prevent, and what it costs when they are absent.

The architecture in this article is not theoretical. It is the practical implementation of the controls that enterprises require before they sign contracts, investors require before they fund, and auditors require before they certify. I built it, I understand every line of it, and I can explain the reasoning behind every decision.

What This Landing Zone Builds

A complete multi-account governance foundation on AWS with two accounts, six security controls, and zero ability to disable any of them from within a member account.

The Six Service Control Policies

1. Deny root account usage in all member accounts

2. Deny disabling CloudTrail — cannot be stopped from inside the account

3. Deny disabling GuardDuty — detector cannot be deleted

4. Deny leaving the Organization — accounts cannot escape governance

5. Restrict activity to approved regions — no shadow infrastructure

6. Deny sensitive IAM operations without MFA — credential abuse protection

Understanding Service Control Policies

Before I walk through the implementation, I want to make sure the concept of Service Control Policies is clear for every reader at every experience level.

An IAM policy controls what an identity can do. If you have an IAM admin in an account, they can modify their own IAM policies. They can grant themselves new permissions. They can remove restrictions that their organization put in place. IAM policies are enforced within an account but can be changed by anyone with IAM admin access.

An SCP is different in a fundamental way. An SCP is applied at the Organizational Unit level by the management account administrator. It sets the maximum permissions boundary for every account in that OU. An IAM admin in a member account cannot modify the SCP. They cannot remove it. They cannot create a permission that the SCP denies.

💡 Think of it like this: An IAM policy is the rules posted on the wall inside a room. Anyone in the room with enough authority can change the rules. An SCP is the architecture of the building. The people inside cannot change the architecture of the building they are standing in.

This distinction is why SCPs are the foundation of any serious multi-account security architecture. They establish controls that survive administrative compromise. Even if an attacker gains IAM admin access to a member account, they cannot disable CloudTrail, cannot disable GuardDuty, cannot move the account out of the Organization's governance model.

The SCP holds.

Architecture Decisions

Why Organization-Level CloudTrail

A per-account CloudTrail can be disabled by an admin in that account. The SCP that prevents cloudtrail:StopLogging helps, but the defense-in-depth approach is to also centralize the trail at the organization level.

An organization-level trail is created in the management account. It captures API activity from all current and future member accounts automatically. New accounts that join the Organization are immediately covered without any additional configuration. And because the trail resource lives in the management account, member account admins have no access to modify or delete it.

The trail also enables log file validation. Every log file gets a cryptographic hash. If any log file is tampered with after delivery, the validation process detects the corruption. For compliance and forensics purposes, this gives you provable log integrity.

Why GuardDuty With All Detections Enabled

The base GuardDuty configuration detects threats from VPC flow logs, DNS logs, and CloudTrail events. This covers network-based and credential-based attacks. But it misses two increasingly common attack vectors:

• S3 data exfiltration through unusual API patterns — detected only when S3 protection is enabled

• Kubernetes cluster compromise through pod privilege escalation — detected only when Kubernetes audit log analysis is enabled

• Malware on EC2 instances that exfiltrates data through disk-level operations — detected only when malware protection is enabled

Enabling all three increases the GuardDuty monthly cost slightly. In an account handling any form of sensitive data, the cost of a missed detection is orders of magnitude higher.

Why Six Config Rules as the Baseline

These six rules appear in the CIS AWS Foundations Benchmark, the AWS Well-Architected Framework security pillar, and every enterprise cloud security questionnaire I have reviewed.

They are not comprehensive coverage. They are the minimum that any organization should have before they describe their AWS environment as having compliance monitoring.

Implementation

Step 1 — Create the Organization Structure

resource "aws_organizations_organization" "main" {
  aws_service_access_principals = [
    "cloudtrail.amazonaws.com",
    "config.amazonaws.com",
    "guardduty.amazonaws.com",
    "sso.amazonaws.com"
  ]

  feature_set = "ALL"

  enabled_policy_types = [
    "SERVICE_CONTROL_POLICY",
    "TAG_POLICY"
  ]
}

resource "aws_organizations_organizational_unit" "workloads" {
  name      = "Workloads"
  parent_id = aws_organizations_organization.main.roots[0].id
}

resource "aws_organizations_account" "workload" {
  name      = var.member_account_name
  email     = var.member_account_email
  parent_id = aws_organizations_organizational_unit.workloads.id
}

The aws_service_access_principals block enables trusted access for each service that will operate at the organization level. Without this, CloudTrail and Config cannot write to centralized resources in the management account from member accounts.

feature_set = "ALL" enables both consolidated billing and all governance features including SCPs. Without this, SCP policy types cannot be enabled.

Step 2 — Deploy the Service Control Policies

Here is the most critical SCP — the one that prevents any root account activity in member accounts:

resource "aws_organizations_policy" "deny_root_usage" {
  name = "landing-zone-deny-root-usage"
  type = "SERVICE_CONTROL_POLICY"

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid    = "DenyRootAccountUsage"
      Effect = "Deny"
      Action = ["*"]
      Resource = ["*"]
      Condition = {
        StringLike = {
          "aws:PrincipalArn" = ["arn:aws:iam::*:root"]
        }
      }
    }]
  })
}

resource "aws_organizations_policy_attachment" "deny_root_usage" {
  policy_id = aws_organizations_policy.deny_root_usage.id
  target_id = aws_organizations_organizational_unit.workloads.id
}

The SCP uses a wildcard Action with a Condition that matches only the root principal ARN pattern. Every API call made from the root account in any member account under the Workloads OU will be denied. Every call. No exceptions.

Here is the region restriction SCP with the critical global service exemptions:

resource "aws_organizations_policy" "deny_non_approved_regions" {
  name = "landing-zone-deny-non-approved-regions"
  type = "SERVICE_CONTROL_POLICY"

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid    = "DenyNonApprovedRegions"
      Effect = "Deny"
      NotAction = [
        "iam:*", "organizations:*", "route53:*",
        "budgets:*", "waf:*", "cloudfront:*",
        "sts:*", "support:*", "trustedadvisor:*"
      ]
      Resource = ["*"]
      Condition = {
        StringNotEquals = {
          "aws:RequestedRegion" = ["us-east-1", "us-west-2"]
        }
      }
    }]
  })
}

⚠️ The NotAction pattern is essential here. Using Action with a Deny would also deny IAM, Route53, and CloudFront operations because those services are global and do not operate within a specific region. NotAction says: deny everything except this list of global services when outside the approved regions. Omitting these exemptions breaks IAM operations in the account.

Step 3 — Deploy Organization-Level CloudTrail

resource "aws_cloudtrail" "org_trail" {
  name                          = "org-cloudtrail"
  s3_bucket_name                = aws_s3_bucket.org_logs.bucket
  s3_key_prefix                 = "cloudtrail"
  include_global_service_events = true
  is_multi_region_trail         = true
  is_organization_trail         = true
  enable_log_file_validation    = true
  cloud_watch_logs_group_arn    = "${aws_cloudwatch_log_group.cloudtrail.arn}:*"
  cloud_watch_logs_role_arn     = aws_iam_role.cloudtrail_cloudwatch.arn

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type   = "AWS::S3::Object"
      values = ["arn:aws:s3:::"]
    }
  }

  depends_on = [aws_s3_bucket_policy.org_logs]
}

Four flags make this trail production-grade:

Step 4 — Enable GuardDuty With Full Detection Coverage

resource "aws_guardduty_detector" "main" {
  enable = true

  datasources {
    s3_logs {
      enable = true
    }
    kubernetes {
      audit_logs {
        enable = true
      }
    }
    malware_protection {
      scan_ec2_instance_with_findings {
        ebs_volumes {
          enable = true
        }
      }
    }
  }
}

resource "aws_cloudwatch_event_rule" "guardduty_findings" {
  name = "landing-zone-guardduty-findings"

  event_pattern = jsonencode({
    source      = ["aws.guardduty"]
    detail-type = ["GuardDuty Finding"]
    detail = {
      severity = [{ numeric = [">=", 7] }]
    }
  })
}

The severity threshold of 7.0 captures HIGH (7.0–8.9) and CRITICAL (9.0–10.0) findings. MEDIUM findings (4.0–6.9) require investigation but not immediate response. In a production environment with a security operations team, you would route MEDIUM findings to a ticketing system and HIGH/CRITICAL findings to PagerDuty.

Step 5 — Deploy AWS Config With Six Compliance Rules

resource "aws_config_config_rule" "mfa_enabled_for_iam_console" {
  name = "mfa-enabled-for-iam-console-access"

  source {
    owner             = "AWS"
    source_identifier = "MFA_ENABLED_FOR_IAM_CONSOLE_ACCESS"
  }

  depends_on = [aws_config_configuration_recorder_status.main]
}

resource "aws_config_config_rule" "s3_bucket_public_access_prohibited" {
  name = "s3-bucket-public-access-prohibited"

  source {
    owner             = "AWS"
    source_identifier = "S3_BUCKET_PUBLIC_ACCESS_PROHIBITED"
  }

  depends_on = [aws_config_configuration_recorder_status.main]
}

All six Config rules use managed rule identifiers. AWS maintains the rule logic. You declare the rule and AWS evaluates your resources against it continuously. Any non-compliant resource triggers a finding in the Config console and can be configured to trigger an SNS notification via Config notification channels.

Step 6 — Deploy and Verify

cd terraform
terraform apply --auto-approve

# Verify everything is active
aws organizations describe-organization
aws cloudtrail get-trail-status \
  --name org-cloudtrail --region us-east-1
aws guardduty list-detectors --region us-east-1
aws configservice describe-configuration-recorder-status
aws organizations list-policies --filter SERVICE_CONTROL_POLICY

What Broke During This Build

Debugging Story 1 — The CloudTrail That Would Not Create

My first terraform apply failed with this error:

Error: creating CloudTrail Trail: InsufficientS3BucketPolicyException:
Bucket does not exist or insufficient policy

CloudTrail requires a specific bucket policy before it can write logs. The policy must allow the cloudtrail.amazonaws.com service principal to call s3:GetBucketAcl and s3:PutObject. Terraform was creating the CloudTrail resource before the bucket policy resource was fully applied.

The fix: Adding depends_on = [aws_s3_bucket_policy.org_logs] to the CloudTrail resource. This forces Terraform to complete the bucket policy creation before attempting to create the trail.

Debugging Story 2 — The Config Rules That Failed to Deploy

After the CloudTrail fix, the apply succeeded until it reached the Config rules. Three rules failed with:

Error: creating ConfigService ConfigRule: InvalidParameterValueException:
The configuration recorder does not exist.

AWS Config rules require the configuration recorder to be active before rules can be evaluated. But the recorder requires a delivery channel before it can be enabled. And the delivery channel requires the S3 bucket to exist.

The correct creation order is: S3 bucket → Config delivery channel → Config recorder → Config recorder status → Config rules

I had the recorder and recorder status in the wrong order relative to the delivery channel. Adding explicit depends_on references to enforce the correct sequence resolved the issue.

Debugging Story 3 — The SCP Region Restriction That Broke IAM

After deploying the region restriction SCP, I tried to run terraform plan from my management account to check for drift. The plan failed immediately with:

Error: error calling sts:GetCallerIdentity: AccessDenied

The region restriction SCP was denying STS calls because sts:* was not in my NotAction exemption list.

⚠️ Critical lesson: SCPs applied to an OU apply to all accounts in that OU — including the management account if you attach them to the root instead of to a child OU. Always attach region restriction SCPs to child OUs, never to the root, and always include all global services in the NotAction exemption list.

Security Posture Improvements at a Glance

Production Gap Analysis

This landing zone establishes the governance foundation. A production deployment at enterprise scale would additionally require the following.

AWS Security Hub provides a unified view of security findings from GuardDuty, Config, IAM Access Analyzer, and third-party integrations in a single dashboard. Security Hub also runs its own automated checks against the CIS AWS Foundations Benchmark and the AWS Foundational Security Best Practices standard. This is the tool that security teams use for day-to-day compliance monitoring.

GuardDuty delegated admin configuration designates a security account as the GuardDuty administrator for the entire Organization. This allows the security team to manage GuardDuty detectors, findings, and member accounts from a central point without accessing individual workload accounts.

AWS Config aggregator collects configuration and compliance data from all member accounts into a single aggregated view. Without an aggregator, Config data is siloed per account and requires navigating to each account individually to see the full compliance picture.

IAM Identity Center with SSO integration provides human access to member accounts through a centralized portal rather than requiring separate IAM users in each account. Engineers log in once with their corporate identity and assume roles in the accounts they need to access. Every access is logged, MFA is enforced centrally, and deprovisioning a user removes their access to all accounts simultaneously.

AWS Firewall Manager centrally manages WAF rules, Shield Advanced protections, and security group policies across all accounts in the Organization.

How This Helps Companies

The healthtech startup that lost their Series B term sheet was not unusual. The specific findings in their security audit appear in post-mortems, investor withdrawal letters, and regulatory enforcement actions across the industry.

The controls in this landing zone directly address every finding in that audit. Root account protection, CloudTrail integrity, GuardDuty coverage, region control, MFA enforcement, and centralized logging are not advanced security controls. They are the baseline that every organization managing sensitive data in AWS should have before they talk to enterprise customers or investors.

The engineering work to implement them is approximately one afternoon. The cost of not having them is measured in lost contracts, failed audits, and withdrawn term sheets.

Full Source Code

The complete Terraform code is available at:

🔗 github.com/Eaglewings966/aws-multi-account-landing-zone

When you are done testing, clean up your environment:

cd terraform && terraform destroy --auto-approve

Note: Member accounts created via Organizations cannot be deleted for 7 days after creation. This is an AWS constraint.

What's Next

Project 7 in this series covers a DevSecOps pipeline with container vulnerability scanning, infrastructure security scanning, and SAST integrated into GitHub Actions.

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

• 🔗 LinkedIn

• 💻 GitHub

• ✍️ Hashnode

• 📝 Medium

• 🌐 Portfolio

Before I Was a DevOps Engineer, I Was a Lot of Other Things

I have cleaned animal enclosures at a zoo. I have replaced brake pads and diagnosed engine faults as an automotive apprentice. I have managed stock, handled suppliers, and run the daily operations of a beauty and cosmetics store.

None of those jobs involved cloud infrastructure. None of them involved CI/CD pipelines, Docker containers, or IAM policies. I started learning DevOps and Cloud Engineering in January 2024 — no connections in tech, no prior experience, no roadmap from anyone who had done it before me.

I tell you this not for sympathy but because it shapes how I explain things. I came into this field as someone who had to learn everything from first principles. When I write about CI/CD pipelines, I write for the version of me from eighteen months ago who would have stared at a GitHub Actions YAML file and had no idea where to start.

This is Project 2 in my public DevOps portfolio series, built and deployed as part of my 30-day DevOps challenge. The full code is at github.com/Eaglewings966/github-actions-cicd-pipeline.

The Story That Made This Project Feel Necessary

A Lagos-based fintech startup was processing several thousand transactions per day. Three developers. A part-time DevOps consultant. Deployments worked like this: the lead developer wrote code, reviewed it himself, SSHed into the production server, pulled the latest commit from GitHub, ran npm install, and restarted the Node.js process with PM2.

It worked. Until it did not.

The lead developer went on a three-day trip to Abuja. A junior developer merged a critical bug fix for a payment gateway timeout issue. Nobody else had the SSH credentials. The fix sat merged in GitHub for 62 hours while users received failed transaction errors. The support queue built up. Two enterprise clients threatened to terminate contracts.

When the lead developer finally got back, the deployment took four minutes.

The business impact of those 62 hours was not four minutes.

When I audited their setup, I found the same five pain points I see consistently in early-stage startups:

• Manual deployments locked to one person with credentials

• No CI/CD — merged code went directly from personal judgment to production

• Inconsistent environments that made bugs impossible to reproduce

• Credential mismanagement — SSH keys shared informally, secrets in plain text .env files

• Zero observability — the team learned about outages when users tweeted about them

The solution I built addresses all five at the infrastructure level.

A fully automated CI/CD pipeline using GitHub Actions, Docker, Amazon ECR, and AWS ECS Fargate. Zero SSH access to production. Zero manual deployment steps. Every push to main passes automated quality gates and lands in a running container on AWS.

What Is a CI/CD Pipeline? (Plain English First)

Before the architecture — because if you have never worked with CI/CD before, the acronym alone can create false confidence that you understand what is actually happening.

CI (Continuous Integration) means every time a developer pushes code, an automated system checks it. Does it follow code style rules? Do the tests pass? Are there known security vulnerabilities in the dependencies? Problems are caught at the point of introduction, when they are cheapest to fix.

CD (Continuous Deployment) means once the code passes every check, the system automatically packages it, publishes it to a container registry, and updates the running service in production. No human touch required after the git push.

GitHub Actions is the automation engine. You write YAML files describing what to do — install Node.js, run ESLint, build a Docker image, push to ECR, update the ECS service — and GitHub runs them automatically every time something triggers the workflow.

The Pipeline Architecture

Before writing a single line of YAML, I drew the dependency chain. The structure of the pipeline is as important as the code inside it.

Developer pushes to main
         │
         ▼
   GitHub Actions triggered
         │
         ├─── Job 1: Lint (ESLint + Prettier)         ← runs immediately
         │
         ├─── Job 2: Security Scan (Trivy)             ← runs in parallel with Lint
         │
         ├─── Job 3: Test (Jest)                       ← needs Lint + Scan to pass
         │
         ├─── Job 4: Build and Push (Docker → ECR)     ← needs Test to pass
         │
         └─── Job 5: Deploy (ECS Fargate)              ← needs Build to succeed

Why this structure matters in real money

Lint and Security Scan run in parallel. They are independent quality gates — there is no reason to wait for one before starting the other. Running them simultaneously saves roughly two minutes on every pipeline run.

Test does not start until both pass. Build does not start until Test passes. Deploy does not start until the image is safely in ECR.

Without this chain, a failing lint check would still trigger a Docker build, an ECR push, and an ECS deployment update — burning five to eight minutes of compute and cloud storage on work you are about to throw away. With this chain, a failing lint check costs ten seconds of runner time.

💡 Fail fast. Fail cheap. The job dependency chain is an engineering cost decision, not just syntax.

Repository Structure

github-actions-cicd-pipeline/
│
├── .github/
│   └── workflows/
│       └── ci-cd.yml            ← the entire pipeline lives here
│
├── app/
│   ├── index.js                 ← Express app with /health endpoint
│   └── package.json
│
├── __tests__/
│   └── app.test.js              ← Jest test suite
│
├── docker/
│   ├── Dockerfile               ← multi-stage, non-root build
│   └── docker-compose.yml       ← full local stack
│
├── scripts/
│   ├── deploy.sh
│   └── nginx.conf
│
└── docs/
    └── pipeline-architecture.md

The .github/workflows/ path is not configurable — GitHub Actions reads this directory automatically. Every change to the pipeline is version-controlled and reviewable in pull requests like any other code change.

The docker/ directory separates containerisation concerns from application concerns. The docker-compose.yml brings up the full local stack — Node.js app, Nginx as a reverse proxy, and Redis for caching — so you can validate the complete environment locally before pushing anything.

Step 1 — The Application

The application is intentionally minimal. The pipeline is the project. But it must be real enough to lint, test, build, scan, and deploy with production-appropriate patterns.

// app/index.js
const express = require('express');
const helmet = require('helmet');
const app = express();
const PORT = process.env.PORT || 3000;

app.use(helmet());

app.get('/', (req, res) => {
  res.json({
    message: 'Tech with Emma — CI/CD Pipeline Running',
    author: 'Emmanuel Ubani',
    github: 'github.com/Eaglewings966',
    status: 'healthy',
    version: process.env.APP_VERSION || 'local'
  });
});

app.get('/health', (req, res) => {
  res.status(200).json({
    status: 'ok',
    uptime: process.uptime(),
    timestamp: new Date().toISOString()
  });
});

module.exports = app;

Why does /health exist? The Application Load Balancer sends health check requests to this endpoint every 30 seconds. If it does not return HTTP 200, the ALB stops routing traffic to that task and ECS replaces it automatically. The uptime and timestamp fields make the response useful for debugging as well as load balancer validation.

Why helmet()? It sets twelve HTTP security headers with one line — X-Content-Type-Options, X-Frame-Options, Strict-Transport-Security, and nine others. Baseline production security that costs nothing.

Validate the full stack locally before pushing:

# bash
cd docker
docker-compose up --build

Step 2 — The Dockerfile

# docker/Dockerfile

# ── Stage 1: Install dependencies ──────────────────────────────────────────────
FROM node:18-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# ── Stage 2: Production image ───────────────────────────────────────────────────
FROM node:18-alpine AS production

RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

COPY --from=builder /app/node_modules ./node_modules
COPY app/ ./

EXPOSE 3000

USER nodejs

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"

CMD ["node", "index.js"]

Three decisions here that carry real weight in production.

node:18-alpine produces an image around 110MB compared to 900MB for the full Node image. In an ECS cluster running across multiple tasks, that difference in pull time compounds on every deployment. Smaller images also mean a smaller attack surface — fewer packages, fewer potential vulnerabilities.

Multi-stage build means only production node_modules enter the final image. Jest, ESLint, and every other development dependency never reach production. This is both a security improvement and a size improvement.

USER nodejs runs the process as a non-root user. A container running as root that gets compromised gives an attacker root-level access inside the container and potentially to the host. A dedicated non-root user limits the blast radius. It costs zero effort and is a baseline container security requirement.

Step 3 — Configure GitHub Secrets

Do this before writing the workflow file. If secrets do not exist when the workflow runs, you get misleading error messages that point you everywhere except the actual problem.

Navigate to your repository: Settings → Secrets and variables → Actions. Create these six secrets:

Create the dedicated IAM user

Create a user named github-actions-deployer — not your personal credentials, not an administrator account. A user with exactly two capabilities: pushing to ECR and updating ECS services.

// IAM Policy — github-actions-deployer
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ECRAccess",
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload",
        "ecr:PutImage"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ECSDeployAccess",
      "Effect": "Allow",
      "Action": [
        "ecs:RegisterTaskDefinition",
        "ecs:DescribeTaskDefinition",
        "ecs:DescribeServices",
        "ecs:UpdateService",
        "iam:PassRole"
      ],
      "Resource": "*"
    }
  ]
}

🔒 If these credentials are ever compromised, the attacker can push container images and update ECS services. They cannot delete resources, access S3, modify IAM policies, or touch any other AWS service. Contained blast radius by design.

Step 4 — The Complete Workflow File

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
  ECS_SERVICE: ${{ secrets.ECS_SERVICE }}
  ECS_CLUSTER: ${{ secrets.ECS_CLUSTER }}
  IMAGE_NAME: devops-demo-app

jobs:

  # ── Job 1: Lint ──────────────────────────────────────────────────────────────
  lint:
    name: Code Quality
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
          cache-dependency-path: app/package-lock.json

      - name: Install dependencies
        run: npm ci
        working-directory: app

      - name: Run ESLint
        run: npx eslint . --max-warnings 0
        working-directory: app

      - name: Check Prettier formatting
        run: npx prettier --check .
        working-directory: app

  # ── Job 2: Security Scan (parallel with Lint) ────────────────────────────────
  security-scan:
    name: Security Scan
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Build image for scanning
        run: docker build -f docker/Dockerfile -t ${{ env.IMAGE_NAME }}:scan .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_NAME }}:scan
          format: table
          exit-code: '1'
          ignore-unfixed: true
          vuln-type: 'os,library'
          severity: 'CRITICAL,HIGH'

      - name: Upload Trivy scan results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: trivy-results
          path: trivy-results.sarif

  # ── Job 3: Test (waits for Lint + Security Scan) ─────────────────────────────
  test:
    name: Test Suite
    runs-on: ubuntu-latest
    needs: [lint, security-scan]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
          cache-dependency-path: app/package-lock.json

      - name: Install all dependencies
        run: npm ci
        working-directory: app

      - name: Run tests with coverage
        run: npm test -- --coverage --coverageThreshold='{"global":{"lines":80}}'
        working-directory: app

      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: app/coverage/

  # ── Job 4: Build and Push (waits for Test) ───────────────────────────────────
  build-and-push:
    name: Build and Push to ECR
    runs-on: ubuntu-latest
    needs: test
    outputs:
      image-uri: ${{ steps.build-image.outputs.image }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build, tag, and push image to ECR
        id: build-image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker buildx build \
            --platform linux/amd64 \
            --file docker/Dockerfile \
            --tag \(ECR_REGISTRY/\)ECR_REPOSITORY:$IMAGE_TAG \
            --tag \(ECR_REGISTRY/\)ECR_REPOSITORY:latest \
            --cache-from type=gha \
            --cache-to type=gha,mode=max \
            --push \
            .
          echo "image=\(ECR_REGISTRY/\)ECR_REPOSITORY:\(IMAGE_TAG" >> \)GITHUB_OUTPUT

  # ── Job 5: Deploy (main branch only) ─────────────────────────────────────────
  deploy:
    name: Deploy to ECS Fargate
    runs-on: ubuntu-latest
    needs: build-and-push
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Download task definition
        run: |
          aws ecs describe-task-definition \
            --task-definition ${{ env.IMAGE_NAME }} \
            --query taskDefinition \
            > task-definition.json

      - name: Update ECS task definition with new image
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: task-definition.json
          container-name: ${{ env.IMAGE_NAME }}
          image: ${{ needs.build-and-push.outputs.image-uri }}

      - name: Deploy to ECS service
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: ${{ steps.task-def.outputs.task-definition }}
          service: ${{ env.ECS_SERVICE }}
          cluster: ${{ env.ECS_CLUSTER }}
          wait-for-service-stability: true

Step 5 — Set Up AWS ECS Fargate

Create the ECR repository

# bash
aws ecr create-repository \
  --repository-name devops-demo-app \
  --region us-east-1 \
  --image-scanning-configuration scanOnPush=true \
  --encryption-configuration encryptionType=AES256

scanOnPush=true enables ECR's built-in vulnerability scanning on every push — independent of the Trivy scan in the pipeline. Two scanning engines, two chances to catch what one might miss.

Create the ECS cluster

# bash
aws ecs create-cluster \
  --cluster-name devops-demo-cluster \
  --capacity-providers FARGATE FARGATE_SPOT \
  --region us-east-1

Both FARGATE and FARGATE_SPOT capacity providers give you cost flexibility. FARGATE_SPOT instances can be 70% cheaper but can be interrupted with a two-minute warning. For production, use FARGATE for your baseline desired count and FARGATE_SPOT for burst capacity.

Create the ECS service

# bash
aws ecs create-service \
  --cluster devops-demo-cluster \
  --service-name devops-demo-service \
  --task-definition devops-demo-app \
  --desired-count 2 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-xxx,subnet-yyy],securityGroups=[sg-xxx],assignPublicIp=ENABLED}" \
  --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:...,containerName=devops-demo-app,containerPort=3000" \
  --deployment-configuration "minimumHealthyPercent=100,maximumPercent=200"

🚀 Zero-downtime is in that last flag. minimumHealthyPercent=100 with maximumPercent=200 means ECS starts new tasks first, waits for ALB health checks to confirm they are healthy, then drains and stops the old tasks. There is never a moment with fewer healthy tasks than your desired count serving traffic. This costs nothing extra — it just requires knowing the configuration exists.

The Security Architecture

Security in this pipeline is not a single feature. It is five independent controls, each addressing a different risk surface.

Why SHA image tags matter

When you tag images with latest, you lose traceability. If something breaks in production, you cannot look at the running container and trace it back to the exact code.

With SHA tagging, every running ECS task tells you its exact Git commit. You can look up that commit, read the code, and understand precisely what is deployed. A system you can reason about is a system you can fix quickly.

The Real Lesson

I want to tell you about the three hours I lost to a problem that took two seconds to fix.

During development, I hit this error:

Error: Credentials could not be loaded

I checked the secret names — they matched exactly. I checked the IAM user — the access key was active. I read the GitHub Actions documentation three times. Everything looked right.

The problem was one character. I had set AWS_REGION to us-east-2 instead of us-east-1. The credentials action was trying to validate against the wrong region endpoint and failing with a credentials error, not a region error. The actual mistake was invisible because I had no visibility into what values the pipeline was actually using.

The fix was changing one character. The diagnosis was three hours.

Here is what I add before every credentials step in any pipeline I build now:

# Add this debug step before any AWS credentials usage
- name: Debug environment
  run: |
    echo "Region:     ${{ env.AWS_REGION }}"
    echo "Cluster:    ${{ env.ECS_CLUSTER }}"
    echo "Service:    ${{ env.ECS_SERVICE }}"
    echo "Repository: ${{ env.ECR_REPOSITORY }}"

When the context is visible, the actual problem becomes obvious in seconds. One wrong character in one variable stands out immediately when you can see all the values together.

🧠 The most expensive debugging sessions are caused by the smallest mistakes. The defence is visibility, not cleverness.

Destroy Reminder

If you deployed to a live AWS account while following this walkthrough, clean up your resources when you are done. ECS Fargate tasks, Application Load Balancers, and NAT Gateways all accumulate charges while running.

# bash — Run these in order

# 1. Scale the service to zero, then delete it
aws ecs update-service \
  --cluster devops-demo-cluster \
  --service devops-demo-service \
  --desired-count 0

aws ecs delete-service \
  --cluster devops-demo-cluster \
  --service devops-demo-service

# 2. Delete the ECS cluster
aws ecs delete-cluster \
  --cluster devops-demo-cluster

# 3. Delete the ECR repository and all images inside it
aws ecr delete-repository \
  --repository-name devops-demo-app \
  --force \
  --region us-east-1

# 4. Delete the ALB and target group
# Navigate to: EC2 → Load Balancers in the AWS console

# 5. Remove VPC resources if you created them for this project

⚠️ Verify in your AWS Billing console after cleanup. Active resources you miss will continue billing silently.

Full code for this project: github.com/Eaglewings966/github-actions-cicd-pipeline

About the Author

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria, currently studying Computer Science at Imo State University. Before entering tech in January 2024, he volunteered at a zoo, trained as an automotive technician, and ran a beauty and cosmetics store. He writes about real infrastructure builds, non-traditional career transitions into tech, and lessons learned the hard way — under the brand Tech with Emma.

• 🔗 LinkedIn — linkedin.com/in/ubaniemmanuel

• 💻 GitHub — github.com/Eaglewings966

• ✍️ Hashnode — emmanuelubani.hashnode.dev

• 📝 Medium — medium.com/@emmaubani966

• 🌐 Portfolio — ops-run.lovable.app

If this walkthrough saved you time or helped something click, share it with one engineer who is learning CI/CD right now.

Three S3 buckets contained 2.1 terabytes of build artifacts, container image layers, and test data. Every object was stored in S3 Standard, the most expensive tier, despite none of the objects having been accessed in over thirty days. Two RDS instances that had been provisioned for a feature that was deprioritised were still running at full capacity, consuming compute and storage costs for a database that served zero queries.

There was no budget alert configured. There was no tagging policy. There was no automated idle detection. There was no monitoring of cost trends. The first signal the company received was the invoice at the end of the month. The CEO cancelled two planned hires to cover the overrun. Every single one of those cost drivers was detectable and preventable with automated controls. The tools exist in AWS. The engineering work to implement them is not complex. What was missing was the decision to build the system.

This article describes exactly how I built that system.

Who I Am and Why This Problem Matters to Me

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. Before I transitioned into tech in January 2024, I was a zoo volunteer, an automotive technician apprentice, and a beauty store manager.

I tell you this because I understand what it means to operate with limited resources. Every naira I spent on cloud learning had to be justified. I could not afford to leave resources running and discover the cost at the end of the month. That constraint taught me to build systems that watch spending automatically, because manual review is a discipline that fails under pressure.

The cost optimization engine I am describing in this article is the same architecture I would implement on day one at any engineering team that was not already running it. It is not a complex system. It is a responsible one.

I will explain every component, every architecture decision, and every implementation detail. Whether you are a senior engineer who wants to validate the approach or a junior engineer who wants to understand how automated cost controls work, this article gives you both.

What the System Does

Five automated controls working together as a coherent platform.

A Lambda function written in Python runs every 24 hours via an EventBridge schedule. It queries CloudWatch for EC2 CPU utilization metrics across all running instances. Any instance with average CPU below 5% for the past 24 hours is stopped automatically. A detailed report is published to an SNS topic and delivered to the engineering team by email.

AWS Budgets monitors the monthly spend against a $20 threshold. Alerts fire at 50%, 80%, and 100% of actual spend and at 80% of forecasted spend. The engineering team knows they are approaching the limit before they breach it, not after.

An S3 lifecycle configuration applies three rules to the cost optimizer bucket. Objects transition to Standard-IA after 30 days, to Glacier after 90 days, and are deleted after 365 days. Incomplete multipart uploads are aborted after 7 days. Old object versions are expired after 90 days. Storage costs reduce automatically without any ongoing manual action.

An IAM Deny policy blocks the creation of EC2, S3, and RDS resources that do not include all four required tags: Environment, Owner, Project, and CostCenter. Untagged resources cannot be created. Cost attribution is enforced at the point of creation.

A CloudWatch dashboard provides unified visibility into EC2 CPU utilization trends, Lambda invocation counts, Lambda error rates, and active cost alarms. The engineering team sees all cost signals in one place rather than navigating three separate AWS consoles.

Architecture Decisions Why Lambda Over AWS Instance Scheduler

AWS Instance Scheduler is a managed AWS solution for starting and stopping instances on a schedule. It requires CloudFormation and has a fixed reporting structure. I chose Lambda because it gives full control over the idle detection logic.

The Lambda approach checks actual CPU utilization metrics rather than stopping instances based on a time schedule. A scheduled stop might shut down an instance that is actively processing a long-running job at midnight. A CPU-based idle check only stops instances that are genuinely not being used. The distinction matters in a production environment where workloads do not always follow a predictable schedule.

The exemption tag pattern adds another dimension. Any instance tagged with CostOptimizer=exempt is skipped by the Lambda. This gives teams a way to protect specific instances without disabling the automation globally.

Why SNS as the Central Notification Hub

The Lambda publishes its report to an SNS topic rather than sending email directly. This design decision has a significant operational benefit. Additional notification channels can be added to the SNS topic without modifying the Lambda code.

A Slack integration can subscribe to the topic via an HTTP endpoint backed by another Lambda. A PagerDuty integration can subscribe for critical budget breach alerts. A ticketing system can subscribe to create an issue for every idle instance report. The Lambda code never changes. The notification architecture evolves by adding subscribers to the topic.

Why IAM Deny Policy Over AWS Config Rules

AWS Config can detect untagged resources after they are created and trigger remediation. But remediation runs after the fact. The resource exists in an untagged state for some period before Config evaluates it and remediates it.

An IAM Deny policy prevents the creation entirely. The API call fails before the resource exists. There is no remediation step because there is no untagged resource to remediate. For tagging enforcement, prevention is categorically more reliable than detection and remediation.

Why Terraform default_tags

The Terraform AWS provider supports a default_tags block at the provider level. Every resource created by the Terraform configuration automatically receives the tags defined there. This means if a new resource is added to main.tf and the engineer forgets to add the tag block, the resource still receives all four required tags from the provider default.

This is more reliable than repeating the tag block in every resource definition. It reduces configuration size and eliminates the most common source of tagging drift in large Terraform files.

Implementation — Building the Engine Step 1 — Set Up the Project Structure

mkdir aws-cost-optimization && cd aws-cost-optimization 
mkdir -p terraform lambda policies 
git init

Step 2 — The Lambda Function

The Lambda function is the heart of the system. It does four things in sequence. It retrieves all running EC2 instances, excluding any tagged as exempt. It queries CloudWatch for CPU utilization metrics for each instance over the past 24 hours. It stops any instance whose average CPU is below the threshold. And it publishes a structured report to SNS. Here is the core idle detection and stop logic

def get_average_cpu(instance_id, hours=24):
    end_time = datetime.now(timezone.utc)
    start_time = end_time - timedelta(hours=hours)

    response = cloudwatch_client.get_metric_statistics(
        Namespace="AWS/EC2",
        MetricName="CPUUtilization",
        Dimensions=[{"Name": "InstanceId", "Value": instance_id}],
        StartTime=start_time,
        EndTime=end_time,
        Period=int(hours * 3600),
        Statistics=["Average"]
    )

    datapoints = response.get("Datapoints", [])
    if not datapoints:
        return None

    return round(datapoints[0]["Average"], 2)


def lambda_handler(event, context):
    instances = get_running_instances()

    for instance in instances:
        avg_cpu = get_average_cpu(instance["instance_id"])

        if avg_cpu is not None and avg_cpu < CPU_IDLE_THRESHOLD:
            stop_instance(instance["instance_id"])
            stopped.append(instance)
        else:
            skipped.append(instance)
publish_report(stopped, skipped, no_data, errors)

The get_metric_statistics call uses a single period equal to the full 24-hour window. This returns one datapoint representing the average CPU across the entire evaluation period rather than a series of hourly averages. One datapoint means one comparison against the threshold. The logic stays simple.

Step 3 — AWS Budgets With Three Alert Thresholds

resource "aws_budgets_budget" "monthly_cost" {
  name         = "cost-optimizer-monthly-budget"
  budget_type  = "COST"
  limit_amount = "20"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator       = "GREATER_THAN"
    threshold                 = 50
    threshold_type            = "PERCENTAGE"
    notification_type         = "ACTUAL"
    subscriber_sns_topic_arns = [aws_sns_topic.cost_alerts.arn]
  }

  notification {
    comparison_operator       = "GREATER_THAN"
    threshold                 = 80
    threshold_type            = "PERCENTAGE"
    notification_type         = "ACTUAL"
    subscriber_sns_topic_arns = [aws_sns_topic.cost_alerts.arn]
  }

  notification {
    comparison_operator       = "GREATER_THAN"
    threshold                 = 100
    threshold_type            = "PERCENTAGE"
    notification_type         = "ACTUAL"
    subscriber_sns_topic_arns = [aws_sns_topic.cost_alerts.arn]
  }
}

Three actual spend notifications plus one forecasted spend notification at 80%. The forecasted alert is the most operationally valuable. It fires before the threshold is breached based on AWS's spend projection model. This gives the team time to investigate and act before the actual breach occurs.

Step 4 — The S3 Lifecycle Policy

Three rules combined into one lifecycle configuration:

resource "aws_s3_bucket_lifecycle_configuration" "assets_lifecycle" {
  bucket = aws_s3_bucket.cost_optimizer_assets.id

  rule {
    id     = "transition-to-cheaper-storage"
    status = "Enabled"
    filter { prefix = "" }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 90
      storage_class = "GLACIER"
    }
    expiration {
      days = 365
    }
  }

  rule {
    id     = "abort-incomplete-multipart"
    status = "Enabled"
    filter { prefix = "" }

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }

  rule {
    id     = "expire-old-versions"
    status = "Enabled"
    filter { prefix = "" }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
}

The incomplete multipart upload rule deserves special attention. Multipart uploads that fail or are abandoned leave partial object data in S3 that is charged at Standard tier rates but is never accessible and never completes. This is a silent cost driver in any bucket where large objects are uploaded. Aborting incomplete uploads after 7 days eliminates this.

Step 5 — IAM Tagging Enforcement

resource "aws_iam_policy" "require_tags" {
  name = "cost-optimizer-require-tags-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "DenyEC2WithoutRequiredTags"
        Effect = "Deny"
        Action = ["ec2:RunInstances"]
        Resource = ["arn:aws:ec2:*:*:instance/*"]
        Condition = {
          "Null" = {
            "aws:RequestTag/Environment" = "true"
            "aws:RequestTag/Owner"       = "true"
            "aws:RequestTag/Project"     = "true"
            "aws:RequestTag/CostCenter"  = "true"
          }
        }
      }
    ]
  })
}

The Null condition with value "true" evaluates to true when the tag key is absent from the request. The Deny statement fires when any of the four required tags are missing. This means all four tags are required for the resource creation to succeed. One missing tag blocks the entire request.

Step 6 — Deploy Everything

cd terraform 
terraform init 
terraform apply --auto-approve 

# Wait for apply to complete then invoke Lambda manually 
aws lambda invoke \ 
--function-name cost-optimizer-stop-idle-instances \ 
--region us-east-1 \ 
--log-type Tail \ 
response.json 

cat response.json

What Broke During the Build

Debugging Story 1 — The Lambda That Would Not Package

When I first ran terraform apply, it failed with this error:

Error: error creating Lambda Function: InvalidParameterValueException:
Could not unzip uploaded file. Please check your file, then try again.

The archive_file data source had created the ZIP file, but Terraform was trying to upload it before the file was fully written to disk in some runs. The fix was adding a depends_on to the Lambda resource pointing to the archive data source, forcing Terraform to wait for the ZIP to be created and its hash computed before attempting the upload.

The lesson is that archive_file data sources can have timing issues with Lambda resources in the same apply. Explicit depends_on resolves this reliably.

Debugging Story 2 — The Budget Alert That Never Arrived

After terraform apply completed, I invoked the Lambda, checked the CloudWatch dashboard, and waited for the budget notification. Nothing arrived.

Fifteen minutes later I realized I had not confirmed the SNS email subscription. AWS had sent a confirmation email immediately after the SNS subscription resource was created. I had missed it in my inbox. Budget alerts and Lambda reports are silently discarded for unconfirmed email subscriptions. There is no error. No retry. No warning. The notification just disappears.

The fix was finding the confirmation email, clicking the link, and waiting sixty seconds for the confirmed status to propagate. Every subsequent notification arrived correctly.\

The lesson is that SNS email subscription confirmation is not optional and is not visible anywhere in the AWS console if you miss it. Check your inbox immediately after terraform apply.

Debugging Story 3 — The IAM Policy Condition That Did Nothing

My first attempt at the IAM tagging Deny policy used StringLike instead of Null for the condition operator.

"Condition": {
  "StringLike": {
    "aws:RequestTag/Environment": ""
  }
}

This condition checks whether the tag value matches an empty string pattern. It does not trigger when the tag key is absent entirely. An EC2 instance could be created without any tags at all and this condition would not fire because StringLike requires the key to be present to evaluate the value.

The correct operator is Null with value "true", which evaluates to true specifically when the tag key is absent from the request. Once I made that change, the policy correctly blocked resource creation when any required tag was missing.

Metrics and Improvements

Running this system for one month on a typical engineering account produces measurable results across all five control areas.

Idle instance detection catches instances that engineers forget to stop after testing, experimentation, or abandoned features. In an account with twenty running instances, an idle detection run typically finds two to four instances that have had near-zero CPU activity for more than 24 hours.

S3 lifecycle policies reduce storage costs progressively over time. In the first month the savings are small because objects need time to age into cheaper tiers. By month three, objects that have not been accessed begin transitioning to Glacier at a storage cost of approximately \(0.004 per GB per month compared to \)0.023 per GB in Standard tier.

Budget alerts at 50% give the engineering team two weeks of reaction time in a typical monthly spend cycle. By the time the alert fires, there is enough of the month remaining to investigate anomalies, stop unnecessary resources, and avoid breaching the full budget.

IAM tagging enforcement eliminates the tagging compliance problem at the source. After the policy is attached to the relevant IAM groups or roles, untagged resource creation simply fails. No remediation workflow. No audit cycle. No Config rule evaluation delay.

Production Considerations

This platform covers the essential automated cost controls. A production deployment at engineering team scale would additionally require the following.

Multi-account cost aggregation via AWS Organizations and Cost Explorer at the management account level gives visibility across all accounts in the organization in a single view. Single-account Budget alerts are a starting point, not a complete solution for organizations with more than one account.

AWS Compute Optimizer analyses EC2 instance utilization patterns and recommends rightsizing from overprovisioned instance types to appropriately sized ones. This addresses a different category of waste than idle detection — instances that are always running but at 20% of their capacity because they were provisioned for a peak that never materialised.

AWS Cost Anomaly Detection uses machine learning to identify spend patterns that deviate from the historical baseline. This catches cost spikes that do not breach a fixed budget threshold but are anomalous relative to normal spending patterns. It is more sensitive than threshold-based alerting for detecting unexpected cost drivers.

Slack notifications via SNS to Lambda to webhook give the engineering team cost alerts in the channel they are already monitoring rather than requiring them to check a separate email inbox. Real-time alerts in Slack have a significantly faster response rate than email notifications for operational issues.

Terraform remote state with S3 backend and DynamoDB locking replaces local state management. This allows multiple engineers to run Terraform operations without state conflicts and stores state history for auditing and recovery.

Key Lessons Learned

The archive_file data source requires explicit dependency management

Terraform does not always resolve the timing between data source evaluation and resource creation correctly for Lambda packaging. Use depends_on to make the dependency explicit and prevent upload failures caused by incomplete ZIP file creation.

SNS email confirmation is invisible if missed

There is no AWS console indicator that a subscription is unconfirmed and silently discarding messages. Check the inbox immediately after apply and confirm before testing any notification path.

IAM Null condition versus StringLike is a meaningful difference

Null evaluates key presence. StringLike evaluates key value. For tagging enforcement where the goal is to prevent keyless resource creation, Null is the correct operator. StringLike will not catch requests where the tag key is entirely absent.

Terraform default_tags is the most reliable tagging mechanism

Per-resource tag blocks are skipped. default_tags at the provider level is never skipped. For any Terraform configuration where consistent tagging is a compliance requirement, default_tags is the correct implementation.

EventBridge cron expressions are more operationally useful than rate expressions

A rate(24 hours) expression runs relative to the function creation time. A cron expression runs at a predictable time each day. For cost optimization, receiving the idle instance report at 6am UTC before the team starts work is more useful than receiving it at a random time determined by when the Lambda was first deployed.

How This Helps Companies

The startup that received a $34,000 bill in January 2023 was not unusual. The pattern of uncontrolled cloud spend driven by forgotten instances, untiered storage, and missing tags repeats across engineering organizations at every scale.

The five controls in this platform directly address the specific failure modes that caused that bill. Idle detection would have stopped the fourteen forgotten load test instances within 24 hours. S3 lifecycle policies would have tiered the 2.1 terabytes of build artifacts automatically. IAM tagging enforcement would have prevented the creation of the unattributed instances entirely. Budget alerts at 50% would have surfaced the anomaly weeks before the invoice arrived.

The skills demonstrated here apply directly to any engineering team that is running AWS infrastructure and wants to move from reactive cost management to proactive cost control.

I am actively looking for remote DevOps and Cloud Engineering roles globally. If your team is building the kind of automated infrastructure controls described in this article, I want to contribute to that work.

Full code is at: https://github.com/Eaglewings966/aws-cost-optimization

Destroy your environment when done:

cd terraform && terraform destroy --auto-approve

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

LinkedIn: linkedin.com/in/ubaniemmanuel

GitHub: github.com/Eaglewings966

Hashnode Blog: emmanuelubani.hashnode.dev

Medium Article: medium.com/@emmaubani966

Portfolio: ops-run.lovable.app

In 2021, a well-funded fintech startup in London had a deployment process that looked perfectly reasonable on paper. Developers pushed code to GitHub. A CI pipeline ran tests. A senior engineer reviewed the build. Someone ran kubectl apply against the production cluster.

It worked most of the time.

On a Tuesday afternoon in November, it did not.

A configuration change for a new payment processor integration was deployed to production. The change passed all automated tests. It passed human review. It was applied to the cluster at 2:47pm.

By 3:04pm, the payment processing service had crashed across all replicas. The database connection pool had been exhausted by a misconfigured retry loop in the new integration. Every transaction request was timing out. Seventeen minutes after the deployment, the engineering team discovered the problem — not from their monitoring system, which had no alerting for connection pool exhaustion but from customer support receiving 340 failure reports in their inbox.

The rollback took 38 minutes. Not because the fix was complex. Because nobody could agree on which state the cluster should return to. The only deployment history was the engineer's memory of what kubectl apply command had been run. There was no Git reference point for the previous cluster state because the deployment had never been tracked in Git in the first place.

Estimated revenue loss for that 55-minute incident: $280,000.

The engineering post-mortem identified four root causes.

No Git-based deployment tracking meant no clean rollback target. No progressive delivery meant 100% of users hit the bad release simultaneously. No self-healing meant configuration drift from previous manual changes had gone undetected for weeks and contributed to the failure mode. No audit trail meant the post-mortem could not definitively identify what state the cluster was in before the deployment.

All four of those root causes have one solution.

"GitOps".

That is what this project builds.

What GitOps Actually Solves

Before I describe what I built, I want to be precise about what GitOps is solving and what it is not.

GitOps is not a deployment automation tool. It is an operational model that moves cluster authority from the people operating the cluster to the Git repository. In a traditional model, the cluster reflects whatever engineers have done to it recently. kubectl apply, manual edits, configuration patches during incidents. The cluster is the source of truth.

In a GitOps model, Git is the source of truth. The cluster is an execution environment that continuously tries to match what Git describes. Any divergence is automatically corrected. Every change to the cluster must come through Git, which means every change is reviewed, version-controlled, timestamped, and attributed.

This eliminates configuration drift because drift cannot persist beyond the next reconciliation cycle. It provides a clean rollback target because the previous cluster state is the previous Git commit. It makes progressive delivery the natural default because the delivery controller reads from Git on every sync.

The fintech incident I described could not have unfolded the same way in a GitOps environment. The deployment would have been a Git push. The rollback would have been a Git revert. The root cause investigation would have started with a git diff. And if progressive delivery had been in place, the bad configuration would have hit 20% of users before promotion, not 100%.

Who I Am and Why I Built This

I am Emmanuel Ubani, a Cloud and DevOps Engineer based in Lagos, Nigeria. Before I wrote a single line of Terraform, I was a zoo volunteer tourist guard, an automotive technician apprentice, and a beauty store manager. I transitioned into tech in January 2024.

I tell you this not as decoration but as context for how I approach learning. I did not come from a background where infrastructure was familiar. Every concept I understand, I understand because I broke it first and then fixed it. This article is written from that place. I will show you what I built, how I built it, what broke during the build, and what I would do differently for a production deployment.

I am going to give both the engineer who already runs Kubernetes in production and the engineer who is building their first real cluster something worth reading.

What I Built

I built a production-grade GitOps platform on AWS EKS Fargate with the following components working together as a coherent system.

An EKS Fargate cluster provisioned with eksctl. Zero EC2 instances. Zero node groups. AWS provisions the exact compute each pod needs, invisibly, per pod.

Terraform with the Helm provider managing Argo CD and Argo Rollouts installation as version-controlled infrastructure. The platform tooling itself is infrastructure as code.

Argo CD implementing the App of Apps pattern. One root application watches a folder in my GitHub repository. Every manifest file in that folder is automatically registered as a child application.

Argo Rollouts handling canary progressive delivery. New versions receive 20% of traffic first, then 50%, then 100%. Automatic rollback if health checks fail at any step.

GitHub Actions closing the GitOps delivery loop. Every push to main triggers an immediate Argo CD sync. The entire cycle from code change to cluster update is automated.

Architecture Decisions — The Why Behind Each Choice

Why App of Apps

If you register applications individually in Argo CD, you create a tight coupling between your platform team and the Argo CD server. Every time a new service needs to be onboarded, someone has to touch Argo CD directly. At three services this is manageable. At thirty services it is a bottleneck.

The App of Apps pattern removes this coupling completely.

One root application watches a specific path in the Git repository. Every YAML file at that path is automatically treated as a child application and registered in Argo CD without any manual action. To add a new service to the platform, an engineer creates one YAML file, pushes it to Git, and waits three minutes. The platform does the rest.

The trade-off is one additional sync layer and a dependency on sync-wave annotations when services have startup ordering requirements. For most platforms, that trade-off is worth it.

Why Fargate Over EC2 Node Groups

Fargate removes an entire category of operational work from the platform team. No AMI version management. No cluster autoscaler configuration. No node drain sequences during Kubernetes upgrades. No capacity planning for the worker node pool. No node-level security patching.

The trade-offs are real. Fargate pods start slower than EC2 pods because AWS provisions compute per pod rather than using pre-warmed node capacity. Per-pod cost is higher than equivalent EC2 capacity at sustained utilisation. All Fargate pods run in private subnets, which constrains external access patterns and requires additional networking components for production exposure.

For a platform running Argo CD and supporting workloads, those trade-offs are acceptable. The operational simplification is significant enough to justify them.

There was also a practical dimension to this decision. My AWS account had hard limits that blocked standard EKS node group provisioning. My vCPU quota was 1 and my EC2 Fleet Request quota was effectively 0. I tried four different approaches before discovering that Fargate bypasses both constraints entirely. What started as a workaround turned out to be a genuinely better architecture for this use case.

Why Argo Rollouts Over Kubernetes Rolling Updates

A standard Kubernetes Deployment rolling update gives you zero traffic control during the transition. Old and new pod versions coexist temporarily but the distribution of traffic between them is not configurable. There is no mechanism to validate behavior at 20% of users before continuing.

Argo Rollouts introduces two separate Kubernetes services, one for stable traffic and one for canary traffic, and a controller that manages the weighted distribution between them at each step. At 20%, one pod revision receives the canary service traffic and the other receives the stable service traffic. The controller advances the weight only when the configured pause duration passes and health checks confirm the canary pods are healthy.

If health checks fail at any step, the entire rollout stops. Traffic returns to 100% stable. No manual intervention required. This is what prevents the fintech incident I described from unfolding the way it did.

Why Terraform Helm Provider Over Manual helm install

A manual helm install command is not reproducible across environments. It produces no infrastructure state that tooling can inspect or modify. It cannot be included in a terraform plan output. It cannot be rolled back to a previous version via a single command.

The Terraform Helm provider makes Argo CD a first-class infrastructure resource. It appears in plan output. It can be updated via apply. It is destroyed cleanly via destroy. And critically, it allows per-component resource requests to be set declaratively, which is a hard requirement on Fargate where every pod must have explicit CPU and memory requests meeting the minimum Fargate thresholds.

Implementation — The Full Build

Step 1 — Check Your AWS Account Constraints First

Before provisioning anything, identify your account-level restrictions. Two quotas matter for EKS on a new AWS account.

# Check running On-Demand vCPU quota
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-1216C47A \
  --region us-east-1

# Check EC2 Fleet Request quota
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-34B43A08 \
  --region us-east-1

If your vCPU quota is below 4 or your Fleet Request quota is 0, standard EKS managed node groups will fail. Use Fargate. It bypasses both constraints entirely and is a legitimate production architecture, not a workaround.

Step 2 — Create the EKS Fargate Cluster

eksctl create cluster \
  --name emmanuel-gitops \
  --region us-east-1 \
  --fargate

aws eks update-kubeconfig \
  --region us-east-1 \
  --name emmanuel-gitops

kubectl get nodes

eksctl provisions the VPC, public and private subnets, NAT Gateway, EKS control plane, and a default Fargate profile in a single operation. The entire process takes 15 to 20 minutes.

When kubectl get nodes returns nothing, this is correct behaviour on a Fargate cluster. Fargate has no persistent node pool to list. Compute is provisioned per pod on demand and is not visible as a node object in Kubernetes.

Step 3 — Create Fargate Profiles for Every Namespace

This is the most operationally important step in the entire Fargate setup and the one most tutorials understate.

On EKS Fargate, the scheduler cannot place a pod into a namespace unless a matching Fargate profile exists for that namespace. The default profile created by eksctl covers the default and kube-system namespaces only. Every additional namespace requires an explicit profile.

If you skip this and deploy a pod into an uncovered namespace, the pod will remain in Pending state indefinitely. There is no error message. No scheduler event that clearly identifies the cause. The pod just waits. This cost me significant debugging time before I understood the mechanism.

for ns in argocd argo-rollouts devops-demo; do
  eksctl create fargateprofile \
    --cluster emmanuel-gitops \
    --region us-east-1 \
    --name fp-${ns} \
    --namespace ${ns}
done

# Verify all profiles are Active before continuing
eksctl get fargateprofile \
  --cluster emmanuel-gitops \
  --region us-east-1

Wait until all three profiles show Active status before proceeding to Terraform. Profiles in a Creating state will not accept pod scheduling.

Step 4 — Install Argo CD and Argo Rollouts via Terraform

The Terraform configuration reads the existing EKS cluster via data sources, configures the Kubernetes and Helm providers against it, creates the three namespaces, and installs Argo CD and Argo Rollouts via helm_release resources.

The most important detail in this Terraform configuration is the resource request values. Argo CD installs six separate components as independent deployments. Every single one must have explicit CPU and memory requests that meet Fargate minimums of 250m CPU and 512Mi memory. Missing a single component means that component stays in Pending state and Argo CD operates in a degraded mode that fails in non-obvious ways.

Here is the Argo CD helm_release resource with all required resource requests:

resource "helm_release" "argocd" {
  name       = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = var.argocd_chart_version
  namespace  = kubernetes_namespace.argocd.metadata[0].name
  timeout    = 600

  set {
    name  = "configs.params.server\\.insecure"
    value = "true"
  }

  set { name = "server.resources.requests.cpu"           value = "250m"  }
  set { name = "server.resources.requests.memory"        value = "512Mi" }
  set { name = "controller.resources.requests.cpu"       value = "250m"  }
  set { name = "controller.resources.requests.memory"    value = "512Mi" }
  set { name = "repoServer.resources.requests.cpu"       value = "250m"  }
  set { name = "repoServer.resources.requests.memory"    value = "512Mi" }
  set { name = "redis.resources.requests.cpu"            value = "250m"  }
  set { name = "redis.resources.requests.memory"         value = "512Mi" }
  set { name = "applicationSet.resources.requests.cpu"   value = "250m"  }
  set { name = "applicationSet.resources.requests.memory" value = "512Mi" }
  set { name = "notifications.resources.requests.cpu"    value = "250m"  }
  set { name = "notifications.resources.requests.memory" value = "512Mi" }

  depends_on = [kubernetes_namespace.argocd]
}

cd terraform
terraform init
terraform apply --auto-approve

# Monitor pod startup — allow 5 to 8 minutes on Fargate
kubectl get pods -n argocd --watch
kubectl get pods -n argo-rollouts --watch

Step 5 — Access Argo CD and Deploy the Root Application

# Retrieve the initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d && echo

# Open port-forward — note this will drop intermittently on Fargate
kubectl port-forward svc/argocd-server -n argocd 8080:80

# Open http://localhost:8080 in browser
# Username: admin
# Password: retrieved above

Before applying the root application, push all manifests to GitHub. Argo CD reads from the remote Git repository, not the local filesystem. Applying the root app before pushing produces a ComparisonError state where Argo CD cannot find the child application manifests it was told to watch.

git add .
git commit -m "feat: production GitOps platform with canary delivery"
git push origin main

# Deploy the root application
kubectl apply -f argocd/apps/root-app.yaml

# Verify child apps are registered and syncing
argocd app list --insecure --server localhost:8080

Here is the root application manifest that implements App of Apps:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/Eaglewings966/argocd-gitops-platform
    targetRevision: HEAD
    path: argocd/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true

The prune: true flag removes cluster resources that no longer have a corresponding Git manifest. The selfHeal: true flag reverts manual cluster changes that diverge from Git state. Together they enforce Git as the authoritative source of truth.

Step 6 — The Canary Rollout

Here is the rollout configuration that implements progressive delivery with 20/50/100 traffic weight steps:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: devops-demo-rollout
  namespace: devops-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: devops-demo
  template:
    metadata:
      labels:
        app: devops-demo
    spec:
      containers:
        - name: devops-demo
          image: eaglewings6/devops-demo-app:latest
          ports:
            - containerPort: 3000
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
  strategy:
    canary:
      stableService: devops-demo-stable
      canaryService: devops-demo-canary
      steps:
        - setWeight: 20
        - pause:
            duration: 60s
        - setWeight: 50
        - pause:
            duration: 60s
        - setWeight: 100
      maxSurge: 1
      maxUnavailable: 0

Watch the rollout progress in real time:

kubectl argo rollouts get rollout devops-demo-rollout \
  -n devops-demo --watch

You will see the traffic weight change in your terminal as each step completes. 20% to 50% to 100%. Every step is gated by a health check. If the readiness probe fails at 20%, the rollout stops. Traffic returns to 100% stable. Nobody gets paged.

What Broke During This Build — The Real Debugging Stories

Debugging Story 1 — The Silent Pending State

After creating the EKS cluster and running terraform apply, I deployed the Argo CD manifests and waited for the pods to start. Five minutes passed. Eight minutes passed. The pods were still in Pending state.

I ran kubectl describe pod on one of the Argo CD pods.

Events:
  Warning  FailedScheduling  default-scheduler
  0/0 nodes are available: 0 Fargate nodes available.

Zero Fargate nodes available. I had created the cluster with Fargate enabled but had not created a Fargate profile for the argocd namespace. The default profile only covered kube-system and default. The scheduler could not place the pods anywhere. The fix was creating the Fargate profile for the argocd namespace before reapplying. The pods started within four minutes of the profile reaching Active status. The lesson is that Fargate profiles are not optional configuration. They are the scheduling permission system. No profile means no compute assignment for that namespace. Create them before deploying.

Debugging Story 2 — The Argo CD CLI Refusing Connections

After Argo CD was running and I had opened the port-forward session, I tried to login via the CLI.

argocd login localhost:8080 
--username admin 
--password [PASSWORD] 
--insecure

The response was FATA[0000] dial tcp [::1]:8080: connect: connection refused

The UI was loading fine in the browser. The CLI was getting a connection refused error. They were hitting the same port on the same localhost address.

The cause was that the port-forward session had dropped between when I opened the browser tab and when I ran the CLI command. The browser had retried the connection transparently. The CLI had not. By the time the CLI command ran, the port-forward process had terminated. The fix was to restart the port-forward session in a separate terminal window and keep it running as a background process.

kubectl port-forward svc/argocd-server -n argocd 8080:80

The deeper lesson is that port-forward on Fargate is inherently unstable for anything requiring persistent connectivity. A browser handles reconnection silently. A CLI tool and a CI/CD pipeline do not. For automated operations, you need a stable network endpoint.

Debugging Story 3 — The Terraform Plan That Would Not Run

After creating the cluster with eksctl, I ran terraform init and terraform plan inside the terraform directory. The plan failed immediately.

Error: Invalid provider configuration
Provider "registry.terraform.io/hashicorp/kubernetes"
requires explicit configuration.

The Kubernetes and Helm providers were configured using data sources that read the cluster endpoint and certificate from the EKS API. But the plan was failing before reaching those data sources.

The cause was that I had not yet run aws eks update-kubeconfig after creating the cluster. The Terraform Kubernetes provider was falling back to the local kubeconfig, which had no entry for the new cluster yet.

The fix was running aws eks update-kubeconfig before terraform plan. Once the local kubeconfig had the cluster entry, the data sources resolved correctly and the plan succeeded.

Debugging Story 4 — AWS Blocking ELB Provisioning

When I tried to expose the Argo CD server via a LoadBalancer service type, the service remained in Pending external IP state indefinitely.

kubectl get svc argocd-server -n argocd NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) argocd-server LoadBalancer 172.20.84.123 80:30080/TCP

The underlying cause was an account-level AWS restriction on ELB provisioning. The restriction was independent of VPC configuration or subnet settings. AWS was denying the ELB creation at the account level.

The workaround for this project was to switch to port-forward for UI access and document that the full production solution requires the AWS Load Balancer Controller, which provisions ALBs through Ingress resources rather than LoadBalancer services and is not subject to the same account restriction.

Production Gap Analysis

This platform is a functional demonstration of the GitOps delivery model. A production deployment at engineering team scale would require these additional components:

ALB Controller for stable Argo CD exposure

The AWS Load Balancer Controller provisions ALBs through Kubernetes Ingress resources rather than Service type LoadBalancer. This provides a stable HTTPS endpoint with ACM certificate termination and Route 53 DNS. It enables browser access, CLI access, and GitHub Actions pipeline connectivity without port-forward sessions.

IRSA for pod-level AWS identity

Pods that interact with AWS services should operate under IAM Roles for Service Accounts. IRSA binds an IAM role to a Kubernetes service account via an OIDC trust relationship, giving each workload a scoped identity with minimum required permissions rather than relying on the broader Fargate execution role.

External Secrets Operator for secret management

Kubernetes Secrets store values as base64-encoded strings without encryption at rest by default. External Secrets Operator with AWS Secrets Manager provides automatic secret rotation, encryption at rest, and eliminates the pattern of managing secrets manually in the cluster.

Argo CD SSO for team access

The default admin credential is a single shared password with no user-level audit trail. Argo CD's Dex component supports OIDC integration with GitHub, Okta, and Google Workspace, providing named user authentication and per-user activity logs for every sync and rollback operation.

Container image security scanning

The GitHub Actions workflow should include a container vulnerability scanning step using Trivy or Grype before the Argo CD sync is triggered. This prevents images with known critical CVEs from reaching the cluster.

Full GitHub Actions GitOps loop

The gitops-sync.yml workflow is committed and correct. It requires a network-reachable Argo CD endpoint to execute fully. That endpoint exists once the ALB Controller is deployed. The workflow is ready. The infrastructure is not yet complete for full automation in this environment.

Metrics and Impact

Key Technical Lessons

Fargate networking is a different compute model Fargate is not EC2 with node management removed. It is a fundamentally different compute model with different networking semantics. Port-forward stability, load balancer provisioning, and pod scheduling all behave differently on Fargate. Understand the networking model before choosing Fargate for a production platform.

GitOps self-healing is a governance mechanism Teams that adopt GitOps as a deployment tool get deployment automation. Teams that adopt it as a governance model get configuration drift prevention, audit trails, and the guarantee that the cluster always reflects an approved state. The difference is in how you frame it internally and what policies you enforce around it.

Partial automation is documented architecture The GitHub Actions GitOps loop in this project requires an ALB endpoint to execute fully. That endpoint is not available in this environment. The workflow is committed, documented, and ready. Documenting what exists, what is blocked, and what the path to close the gap is demonstrates the same engineering depth as a fully operational system.

Resource request completeness on Fargate is binary On EC2, missing resource requests result in best-effort scheduling. On Fargate, missing requests at minimum thresholds result in pods never scheduling. This is not a degraded mode. It is a hard failure. Treat resource requests as correctness requirements, not optimisation suggestions.

How This Helps Companies

The four failure modes that caused the fintech incident are not unusual. They are the default state of Kubernetes operations without intentional architecture to address them.

Configuration drift accumulates in every cluster where engineers have direct kubectl access and no automated reconciliation. Deployment blast radius is 100% in every environment without progressive delivery. Rollback is manual and slow in every system without a clear Git-based deployment history. And audit trails are absent in every cluster where deployments are not tracked in version control.

This platform addresses all four. The skills demonstrated here apply directly to any engineering team running Kubernetes in production and looking to improve their delivery reliability, reduce deployment risk, and introduce progressive delivery without replacing their existing toolchain.

I am actively looking for DevOps and Cloud Engineering roles globally, including remote positions. If your team is working on problems like this, I would like to be part of solving them.

Full code is at: [github.com/Eaglewings966/argocd-gitops-platform](http:// https://github.com/Eaglewings966/argocd-gitops-platform)

Destroy your environment when you are done:

kubectl delete application --all -n argocd kubectl delete namespace devops-demo argo-rollouts argocd 
--ignore-not-found=true cd terraform && terraform destroy --auto-approve eksctl delete cluster --name emmanuel-gitops --region us-east-1

Total spend for this project was approximately $5. Project 5 in this series is AWS Cost Optimization — building the automated system that finds and eliminates wasted cloud spend across a multi-service AWS environment.

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

LinkedIn: linkedin.com/in/ubaniemmanuel

GitHub: github.com/Eaglewings966

Hashnode: emmanuelubani.hashnode.dev

Medium: medium.com/@emmaubani966

Portfolio: ops-run.lovable.app

And nothing tests your infrastructure thinking like EKS.

I expected this project to take a few hours. It took days. Not because my code was wrong, but because AWS account restrictions blocked every path I tried. This is the story of what actually happened, what I learned, and how I finally got a Node.js application running on Kubernetes on AWS EKS using Fargate.

What I Set Out to Build

• AWS EKS cluster provisioned with Terraform

• Custom VPC with two public subnets across two availability zones

• IAM roles for the cluster and node group with least-privilege policies

• Kubernetes Deployment with liveness and readiness probes

• Kubernetes LoadBalancer Service for external access

• Full infrastructure teardown with a single command

The First Wall — EC2 vCPU Quota of 1

Before I even ran terraform apply, I should have checked this:

aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-1216C47A \
  --region us-east-1

My account had a vCPU limit of 1. Not 5 like AWS documentation suggests for new accounts. Just 1.

A single t3.medium instance needs 2 vCPUs. So my node group was never going to launch — not even one node.

I submitted a quota increase request to AWS Support. The first request for 10 vCPUs was rejected. The appeal that worked asked for 4 vCPUs and explicitly pointed out that 4 vCPUs is still below AWS's own standard default of 5 for new accounts.

The Second Wall — EC2 Fleet Request Quota

Even after addressing the vCPU issue, every node group attempt failed with this error:

You've reached your quota for maximum Fleet Requests for this account.
Launching EC2 instance failed.

This quota is completely separate from the vCPU quota. EKS managed node groups use EC2 Fleet internally to provision instances — and my account had a Fleet Request limit so low it blocked every launch attempt.

The worst part — this quota is not visible in the Service Quotas console and cannot be queried via AWS CLI. It is an invisible wall.

I tried three approaches to bypass it:

Attempt 1 — EKS managed node group with On-Demand capacity Failed. EC2 Fleet error every time.

Attempt 2 — EKS managed node group with Spot capacity Failed. Spot instances also route through EC2 Fleet internally.

Attempt 3 — Launch template with managed node group Failed. The launch template changes how the instance is configured but not how it is requested from AWS capacity.

The Third Attempt — Self-Managed Node Group

Amazon Q suggested using self-managed node groups, which use AWS Auto Scaling Groups directly and bypass EC2 Fleet entirely.

I updated my Terraform to use aws_autoscaling_group instead of aws_eks_node_group:

resource "aws_autoscaling_group" "eks_nodes" {
  name                = "${var.project_name}-nodes-asg"
  desired_capacity    = var.desired_nodes
  min_size            = var.min_nodes
  max_size            = var.max_nodes
  vpc_zone_identifier = [aws_subnet.public_a.id, aws_subnet.public_b.id]

  launch_template {
    id      = aws_launch_template.eks_nodes.id
    version = "$Latest"
  }
}

This bypassed the Fleet Request quota. But then hit the vCPU wall again:

You have requested more vCPU capacity than your current vCPU limit of 1

The self-managed approach proved the Fleet Request issue was solved — but the vCPU limit of 1 was still the final blocker.

The Solution — AWS Fargate

Fargate is serverless compute for containers. No EC2 instances. No vCPU quota. No Fleet Requests. Nothing.

One command creates the entire cluster:

eksctl create cluster --name emmanuel-eks --region us-east-1 --fargate

Then add a Fargate profile for your application namespace:

eksctl create fargateprofile \
  --cluster emmanuel-eks \
  --region us-east-1 \
  --name fp-devops-demo \
  --namespace devops-demo

Deploying the Application

With the cluster ready, deploying the app took less than 2 minutes:

kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

Watch the pods come up:

kubectl get pods -n devops-demo -w

Both pods went from Pending to ContainerCreating to Running in under 60 seconds.

Verifying the App is Live

kubectl port-forward -n devops-demo deployment/devops-demo-app 8080:3000

Then open your browser at http://localhost:8080

The app responded with:

{
  "status": "healthy",
  "message": "CI/CD pipeline is working",
  "author": "Emmanuel Ubani"
}

One Important Fargate Gotcha

Fargate pods run in private subnets. Classic Load Balancers cannot register pods in private subnets as targets — so kubectl get svc will show a LoadBalancer hostname but requests to it will return empty responses.

For local access, port-forward works perfectly. For production external access, you need the AWS Load Balancer Controller which creates an Application Load Balancer instead of a Classic LB.

For portfolio purposes, port-forward proves the application is running correctly.

Cleanup — Always Destroy When Done

eksctl delete cluster --name emmanuel-eks --region us-east-1

This removes everything — cluster, Fargate profiles, VPC, subnets, IAM roles. Zero orphaned resources. Zero ongoing charges.

Key Lessons From This Build

Always check your vCPU quota first. Run the service-quotas command before writing a single line of Terraform. It takes 10 seconds and saves days.

EC2 Fleet Request quota is invisible. It does not appear in the Service Quotas console and cannot be queried via CLI. If you hit Fleet Request errors on a new account, Fargate is your fastest path forward.

Fargate bypasses every EC2 restriction. No vCPU quota. No Fleet Requests. No instance type selection. AWS manages all the compute underneath.

Fargate needs private subnets. Always use eksctl to create the cluster — it handles the VPC networking correctly with both public and private subnets.

Never store projects in C:/Windows/System32. Git cannot write to protected Windows directories. Use C:/Projects or your user home folder.

Never commit terraform.tfstate to GitHub. Check your .gitignore before every first push.

Debugging is the job. I tried four different approaches before finding the one that worked. That is not failure — that is engineering.

Full Code

Complete Terraform and Kubernetes manifests are on my GitHub:

👉 eks-kubernetes-deployment

What Is Next

This is Project 3 in my Cloud and DevOps portfolio series. Next up is a production-grade GitOps platform on AWS EKS using Argo CD — implementing the App of Apps pattern used by companies like Spotify, Airbnb, and Goldman Sachs, with progressive delivery using Argo Rollouts and automated canary deployments.

Previous projects in this series:

• Project 1 — AWS IAM Multi-Account Setup

• Project 2 — GitHub Actions CI/CD Pipeline

Emmanuel Ubani is a Cloud and DevOps Engineer based in Lagos, Nigeria. He writes about real infrastructure builds, career transitions into tech, and lessons learned the hard way.

LinkedIn · GitHub · Hashnode · Medium

They test nothing. They push unscanned images. They have no gates between stages.

Here is what a proper one looks like — and exactly how I built it.

What I Built

A full CI pipeline using GitHub Actions that does the following on every push to main:

→ Runs unit tests automatically

→ Lints the Dockerfile with Hadolint

→ Only if both pass — builds a Docker image and pushes it to Docker Hub

→ Tags the image with the commit SHA for full traceability

If tests fail, the image never gets built. If the Dockerfile has issues, the image never gets built. The pipeline has opinions and it enforces them.

Project Structure

github-actions-ci-pipeline/
├── app/
│   ├── index.js
│   ├── index.test.js
│   └── package.json
├── Dockerfile
├── .dockerignore
├── .github/
│   └── workflows/
│       └── ci.yml
└── README.md

The Pipeline — ci.yml Explained

The pipeline has three jobs:

Job 1 — test

Runs unit tests using Node.js before anything else happens. If this fails, the entire pipeline stops.

Job 2 — lint-dockerfile

Uses Hadolint to check the Dockerfile for bad practices. Things like running as root, using latest tags, or missing HEALTHCHECK instructions get flagged here.

Job 3 — build-and-push

This job only runs if both test and lint-dockerfile pass. It uses Docker Buildx for multi-platform builds and pushes the image to Docker Hub with three tags:

• latest (for main branch only)

• branch name

• commit SHA for full traceability

Two Things I Paid Extra Attention To

Multi-stage Docker build — separates the build environment from the production image. Smaller, cleaner, more secure.

GitHub Actions cache — caches Docker layers between runs so builds are faster after the first one.

These are not advanced techniques. They are just good engineering habits that a lot of teams skip.

The Lesson

Most deployment issues are not deployment issues. They are integration issues that a good CI pipeline would have caught much earlier.

Gate your pipeline. Test before you build. Build before you deploy.

Full Code

Complete pipeline code is on my GitHub: https://github.com/Eaglewings966/github-actions-ci-pipeline

The console is great for learning. But it doesn't scale.

You can't review a console click in a pull request. You can't reproduce it in staging with one command. You can't audit it when something breaks at 2am.

Infrastructure as Code fixes all three. Here's exactly how I built a production-style AWS VPC using Terraform — step by step.

What I Built

• Custom VPC with a 10.0.0.0/16 CIDR block

• Public subnet with an Internet Gateway and route table

• Private subnet for internal workloads

• Security group with least-privilege rules

• Every resource tagged with environment and ownership metadata

Project Structure

terraform-aws-vpc/
├── main.tf
├── variables.tf
├── outputs.tf
├── terraform.tfvars
├── .gitignore
└── README.md

Step 1 — The VPC

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name        = "${var.project_name}-vpc"
    Environment = "dev"
    ManagedBy   = "terraform"
  }
}

Two things worth noting here:

• enable_dns_support and enable_dns_hostnames are both set to true. This is required if you ever want to resolve AWS service endpoints inside the VPC.

• Every resource is tagged with ManagedBy = "terraform" so anyone looking at the AWS console immediately knows this is IaC-managed and should not be edited manually.

Step 2 — Public vs Private Subnet

The public subnet has map_public_ip_on_launch = true — any EC2 instance launched here gets a public IP automatically.

The private subnet has no such setting. Databases and internal services live here and are never directly reachable from the internet. That is intentional.

Step 3 — The Route Table

The public route table has one route: 0.0.0.0/0 pointing to the Internet Gateway. This is what makes the public subnet actually public.

Without this route, even if your subnet is labeled public, traffic has nowhere to go.

The Lesson That Stuck With Me

If you can't tear it down cleanly, you don't fully understand what you built.

terraform destroy is the most honest test of your IaC. Run it. If it errors, you have a dependency problem worth fixing now, not in production.

Full Code

The complete Terraform code is available on my GitHub: https://github.com/Eaglewings966/terraform-aws-vpc