Dual gateway on AWS — AZ failover

Downloads

• 📦 Download the deployment artifacts — deployment-key (.zip) · Marketplace-AMI (.zip)

A basic, non-production high-availability pattern for the Enforza NVA. Two firewalls run across two Availability Zones in one VPC: firewall A is the primary, firewall B a warm passive secondary. Private-subnet workloads egress through the primary, which SNATs to the internet and applies L7 egress filtering. If firewall A fails its EC2 status checks, an automation flips the private default route to firewall B so egress is restored automatically. Deploy with Terraform or CloudFormation — identical topology and failover behaviour.

Deployment-key variant

The firewalls boot from a stock Amazon Linux 2023 base AMI and bootstrap the engine from the public CDN using deployment keys — no Marketplace subscription. A Marketplace-AMI flavour of the same topology (self-registering AMI) is in the Marketplace-AMI bundle; only the firewall image and bootstrap differ — the networking and the entire failover mechanism are identical.

Non-production pattern

This is a teaching/reference HA pattern. Failover keys off coarse EC2 status checks (~2-minute detection), manages a single route, and does not preserve connection state across the flip. See Limitations before using anywhere near production.

Architecture

                              Internet
                                 │
                         Internet Gateway   (public-rt: 0.0.0.0/0 → IGW)
                                 │
   VPC · 10.0.0.0/16             │
   ┌──────────── AZ 0 ───────────┴───────────── AZ 1 ─────────────┐
   │  public-a 10.0.0.0/24          public-b 10.0.10.0/24         │
   │    Firewall A (PRIMARY)          Firewall B (SECONDARY)      │
   │    EIP · ENI-A · src/dst off     EIP · ENI-B · src/dst off   │
   │                                                             │
   │  private-a 10.0.1.0/24         private-b 10.0.11.0/24        │
   │        └──────── shared private route table ────────┘       │
   │        0.0.0.0/0 → ENI-A  (normally)  ⇢ flips to ENI-B       │
   └─────────────────────────────────────────────────────────────┘

   Failover loop:  CloudWatch alarm (FW-A StatusCheckFailed, 2× 1-min)
                     → SNS topic → Lambda (boto3)
                       → ReplaceRoute: 0.0.0.0/0  ENI-A → ENI-B
                   On recovery (alarm OK): Lambda flips back ENI-B → ENI-A

What it deploys

• 1 VPC 10.0.0.0/16, 1 Internet Gateway.
• 2 public subnets — 10.0.0.0/24 (AZ 0, firewall A), 10.0.10.0/24 (AZ 1, firewall B), both auto-assign public IP on launch. Public route table 0.0.0.0/0 → IGW.
• 2 private workload subnets — 10.0.1.0/24 (AZ 0), 10.0.11.0/24 (AZ 1).
• 1 shared private route table with 0.0.0.0/0 → firewall A's ENI, associated with both private subnets.
• 2 Enforza EC2 firewalls (default c6i.large), each with an Elastic IP and an ENI with source/dest check disabled.
• Failover: 1 CloudWatch alarm (firewall A status checks) → 1 SNS topic → 1 Lambda (route flipper) with a least-privilege IAM role.

Outputs: VPC id, firewall A/B instance ids and EIPs, the private route table id, the failover Lambda name, the SNS topic ARN, and the alarm name.

Prerequisites

• TWO Enforza deployment keys — one per firewall, each one-time-use (Onboard Firewall → Deployment Keys). Firewall A registers with key A, firewall B with key B — reusing one key across both fails the second registration.
• AWS credentials able to create VPC, EC2, IAM, Lambda, SNS, and CloudWatch resources.
• Terraform ≥ 1.5 or the AWS CLI.
• Two Availability Zones in your region.

How the bootstrap works

Each firewall boots from a stock AL2023 AMI; its user-data installs the engine straight from the Enforza CDN:

curl -fsSL https://dl.neon.efz.io/install.sh | bash -s -- --regkey="<deployment-key>" --name="enforza-fw-a"

The installer auto-installs prerequisites (nftables, ulogd2, libnetfilter-queue, the nfqueue kernel module) and registers the engine. One key per firewall. Both public subnets auto-assign a public IP so each instance reaches the CDN at first boot; the EIPs then provide a stable SNAT source.

Option A — Deploy with Terraform

cd terraform
terraform init
terraform apply \
  -var "deployment_key_a=<KEY-FOR-FIREWALL-A>" \
  -var "deployment_key_b=<KEY-FOR-FIREWALL-B>"
# optional overrides:
#   -var 'base_ami_id=ami-xxxxxxxx'          # else latest AL2023
#   -var 'instance_type=c6i.large'
#   -var 'ssh_key_name=my-key'
#   -var 'availability_zones=["eu-west-2a","eu-west-2b"]'

The two keys must be distinct. The Lambda source (failover_lambda.py) is zipped automatically by the archive_file data source — no manual packaging.

Option B — Deploy with CloudFormation

The stack creates a named IAM role, so pass --capabilities CAPABILITY_NAMED_IAM:

cd cloudformation
aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name enforza-02-dual-gateway-deployment-key \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      DeploymentKeyA=<KEY-FOR-FIREWALL-A> \
      DeploymentKeyB=<KEY-FOR-FIREWALL-B> \
      AvailabilityZoneA=eu-west-2a \
      AvailabilityZoneB=eu-west-2b

View outputs with aws cloudformation describe-stacks --stack-name enforza-02-dual-gateway-deployment-key --query "Stacks[0].Outputs".

What happens during a failover

1. Firewall A becomes unhealthy (hang, kernel panic, or dead host) — its EC2 StatusCheckFailed metric goes to 1.
2. After 2 consecutive 1-minute periods, the CloudWatch alarm transitions to ALARM.
3. The alarm publishes to the SNS topic enforza-02-failover.
4. SNS invokes the Lambda, which reads NewStateValue = ALARM.
5. The Lambda calls ec2:ReplaceRoute on the shared private route table, repointing 0.0.0.0/0 from firewall A’s ENI to firewall B’s ENI (idempotent — it skips if already correct).
6. Workloads now egress through firewall B. Recovery ≈ the alarm window (~2 min) plus a few seconds.
7. Flip-back (automatic): when firewall A recovers, the alarm returns to OK, and the Lambda flips 0.0.0.0/0 back to firewall A’s ENI.

How to test it

# Stop the primary to trigger failover
aws ec2 stop-instances --instance-ids <FirewallAInstanceId>

# ~2 min later, confirm the route now points at firewall B's ENI
aws ec2 describe-route-tables --route-table-ids <PrivateRouteTableId> \
  --query "RouteTables[0].Routes[?DestinationCidrBlock=='0.0.0.0/0']"

# Watch the Lambda flip
aws logs tail /aws/lambda/enforza-02-failover --follow

# Recover — the alarm returns to OK and the route flips back to firewall A
aws ec2 start-instances --instance-ids <FirewallAInstanceId>

Limitations

Read before using anywhere near production:

• Single flip / single route. Only 0.0.0.0/0 in one shared private route table is managed — deliberate for simplicity.
• Coarse health signal. Failover keys off EC2 status checks, which detect a dead/hung instance but not an in-appliance problem (engine degraded while the OS stays reachable). Production should use a deeper probe.
• Detection latency ~2 minutes — not sub-second HA.
• Flip-back caveat. Auto flip-back is idempotent but can cause a second brief disruption and flap if the primary is unstable. For production, disable the OK action and flip back manually.
• Terraform drift. After a failover the live route points at firewall B; the route resource uses ignore_changes = [network_interface_id] so a later apply won’t fight the Lambda.
• No stateful session sync across the flip — in-flight connections reset.
• One-time-use keys — a re-created firewall needs a fresh key.

Teardown

# Terraform
cd terraform
terraform destroy \
  -var "deployment_key_a=<KEY-FOR-FIREWALL-A>" \
  -var "deployment_key_b=<KEY-FOR-FIREWALL-B>"

# CloudFormation
aws cloudformation delete-stack --stack-name enforza-02-dual-gateway-deployment-key

If a failover occurred, the live route may point at firewall B — both destroy paths remove the route table entirely, so no manual cleanup is needed. Then delete the firewalls from the Enforza console.