End-to-End HTTPS for Kubernetes: Overcoming Let’s Encrypt Validation Issues

When configuring a secure web application on Kubernetes, ensuring end-to-end encryption is often a critical requirement. This means the traffic is encrypted from the client all the way to the backend services, without terminating SSL at an intermediary like CloudFront. However, this setup can introduce challenges when using Let’s Encrypt to manage certificates.

The Challenge: ACME HTTP Validation Behind CloudFront

Let’s Encrypt uses the ACME HTTP challenge to validate ownership of your domain. This involves serving a validation file under the /.well-known/acme-challenge/ path over HTTP. But when you configure CloudFront with redirect-to-https, all HTTP traffic is redirected to HTTPS. This breaks the ACME HTTP challenge because Let’s Encrypt expects the response over HTTP.

Allowing all HTTP traffic as a workaround would compromise your HTTPS enforcement, which isn’t ideal. A more secure approach involves creating an exception for the ACME challenge path while enforcing HTTPS for the rest of the application.

The Solution: Path-Based Exceptions in CloudFront

You can configure CloudFront to allow HTTP access specifically for the /.well-known/acme-challenge/ path, while maintaining HTTPS redirection for all other traffic. Below is a Terraform configuration that demonstrates how to achieve this.

data "aws_cloudfront_cache_policy" "CachingDisabled" {
  name = "Managed-CachingDisabled"
}

data "aws_cloudfront_origin_request_policy" "AllViewer" {
  name = "Managed-AllViewer"
}

data "aws_cloudfront_response_headers_policy" "SimpleCors" {
  name = "Managed-SimpleCORS"
}

resource "aws_cloudfront_distribution" "cf" {
  origin {
    domain_name = data.aws_lb.nlb.dns_name
    origin_id   = "k8s-nlb-origin"
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "match-viewer"
      origin_ssl_protocols   = ["TLSv1.2"]
      origin_read_timeout    = 60
    }
  }

  enabled         = true
  is_ipv6_enabled = true
  http_version    = "http2"
  price_class     = "PriceClass_100"
  comment         = "Kubernetes CloudFront Distribution"
  aliases         = ["example.com"]
  web_acl_id      = aws_wafv2_web_acl.main.arn

  # Default behavior: enforce HTTPS
  default_cache_behavior {
    cache_policy_id            = data.aws_cloudfront_cache_policy.CachingDisabled.id
    origin_request_policy_id   = data.aws_cloudfront_origin_request_policy.AllViewer.id
    response_headers_policy_id = data.aws_cloudfront_response_headers_policy.SimpleCors.id
    allowed_methods            = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods             = ["HEAD", "GET"]
    target_origin_id           = "k8s-nlb-origin"
    viewer_protocol_policy     = "redirect-to-https"
  }

  # Special behavior for ACME HTTP challenge
  ordered_cache_behavior {
    path_pattern               = "/.well-known/acme-challenge/*"
    target_origin_id           = "k8s-nlb-origin"
    viewer_protocol_policy     = "allow-all" # Allow HTTP and HTTPS

    allowed_methods            = ["GET", "HEAD"]
    cached_methods             = ["GET", "HEAD"]
    compress                   = true
    cache_policy_id            = aws_cloudfront_cache_policy.acme.id
    origin_request_policy_id   = aws_cloudfront_origin_request_policy.acme.id
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn            = data.aws_acm_certificate.main.arn
    ssl_support_method             = "sni-only"
    cloudfront_default_certificate = false
    minimum_protocol_version       = "TLSv1.2_2021"
  }

  tags = {
    Name = "K8S-CloudFront"
  }
}

resource "aws_cloudfront_cache_policy" "acme" {
  name        = "acme-cache-policy-${var.env}"
  comment     = "Cache policy for ACME validation"
  default_ttl = 86400
  min_ttl     = 3600
  max_ttl     = 86400

  parameters_in_cache_key_and_forwarded_to_origin {
    cookies_config {
      cookie_behavior = "none"
    }
    headers_config {
      header_behavior = "whitelist"
      headers         = ["Host"]
    }
    query_strings_config {
      query_string_behavior = "none"
    }
  }
}

resource "aws_cloudfront_origin_request_policy" "acme" {
  name = "acme-origin-request-policy-${var.env}"

  headers_config {
    header_behavior = "whitelist"
    headers         = ["Host"]
  }

  cookies_config {
    cookie_behavior = "none"
  }

  query_strings_config {
    query_string_behavior = "none"
  }
}

Explanation

1. Default Cache Behavior:
   • Enforces HTTPS (viewer_protocol_policy = "redirect-to-https") for all traffic except paths explicitly overridden by ordered_cache_behavior.
2. Ordered Cache Behavior:
   • Matches requests to /.well-known/acme-challenge/* and allows both HTTP and HTTPS (viewer_protocol_policy = "allow-all").
   • Uses specific cache and origin request policies for minimal caching and forwarding.
3. Custom Policies:
   • Cache policy (aws_cloudfront_cache_policy.acme) ensures minimal caching for ACME challenges.
   • Origin request policy (aws_cloudfront_origin_request_policy.acme) forwards only essential headers to the origin.
4. Security:
   • All other paths remain fully HTTPS-enforced, preserving end-to-end encryption.
   • The ACME challenge path is narrowly scoped to minimize potential abuse.

Conclusion

By leveraging CloudFront’s path-based cache behaviors and Terraform, you can enable Let’s Encrypt to perform HTTP validation without compromising HTTPS enforcement. This approach keeps your application secure while meeting the requirements for certificate management.

Feel free to adapt the Terraform configuration to suit your specific environment. If you have additional considerations, like stricter geo-restrictions or advanced cache policies, you can layer those on top of this foundation.

Sources

https://github.com/odileon-net/examples/blob/main/terraform/aws/cloudfront/acme-challenge-example.tf

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudfront_distribution