Introduction

Welcome

The purpose of this site is to provide documentation on how FortiOS FGCP HA works in AWS post deployment, during a failover event, and best practice for common use cases.

For other documentation needs such as FortiOS administration, please reference docs.fortinet.com.

Overview

FortiOS supports using FGCP (FortiGate Clustering Protocol) in unicast form to provide an active-passive clustering solution for deployments in AWS. This feature shares a majority of the functionality that FGCP on FortiGate hardware provides with key changes to support AWS SDN (Software Defined Networking).

This solution works with two FortiGate instances configured as a master and slave pair and that the instances are deployed in different subnets and different availability zones within a single VPC. These FortiGate instances act as a single logical instance and do not share interface IP addressing as they are in different subnets. You can deploy these in the same subnets and availability zone if desired and share secondary interface IP addressing, however this is a less common design.

The main benefits of this solution are:

• Fast and stateful failover of FortiOS and AWS SDN without external automation\services
• Automatic AWS SDN updates to EIPs and route targets (and ENI secondary IPs, for single AZ)
• Native FortiOS session sync of firewall, IPsec\SSL VPN, and VOIP sessions
• Native FortiOS configuration sync
• Ease of use as the cluster is treated as single logical FortiGate

For further information on FGCP reference the High Availability chapter in the FortiOS Handbook on the Fortinet Documentation site.

Note: Other Fortinet solutions for AWS such as FGCP HA (Single AZ), AutoScaling, and Transit Gateway are available. Please visit www.fortinet.com/aws for further information.

Solution Components

FGCP HA provides AWS networks with enhanced reliability through device fail-over protection, link fail-over protection, and remote link fail-over protection. In addition, reliability is further enhanced with session fail-over protection for most IPv4 and IPv6 sessions including TCP, UDP, ICMP, IPsec\SSL VPN, and NAT sessions.

A FortiGate FGCP cluster appears as a single logical FortiGate instance and configuration synchronization allows you to configure a cluster in the same way as a standalone FortiGate unit. If a fail-over occurs, the cluster recovers quickly and automatically and can also send notifications to administrator so that the problem that caused the failure can be corrected and any failed resources restored.

The FortiGate instances will use multiple interfaces for data plane and control plane traffic to achieve FGCP clustering in an AWS VPC. The FortiGate instances require four ENIs for this solution to work as designed so make sure to use an AWS EC2 instance type that supports this. Reference AWS Documentation for further information on this.

For data plane functions the FortiGates will use two dedicated ENIs, one for a public interface (ie ENI0\port1) and another for a private interface (ie ENI1\port2). These ENIs will utilize primary IP addressing and FortiOS should not sync the interface configuration (config system interface) or static routes (config router static) as these FGTs are in separate subnets. This is controlled via the CLI (config system vdom-exception). Thus when configuring these items, you should do so individually on both FortiGates.

A cluster EIP will be associated to the primary IP of the public interface (ie ENI0\port1) of the current master FortiGate instance and will be reassociated to a new master FortiGate instance as well.

For control plane functions, the FortiGates will use a dedicated ENI (ie ENI2\port3) for HA management access to each instance and also allow each instance to independently and directly communicate with the public AWS EC2 API. This dedicated interface is critical to failing over AWS SDN properly when a new FGCP HA master is elected and is the only method of access available to the current slave FortiGate instance.

Depending on the FortiOS version, the FortiGates will either reuse the HA management interface (ie ENI2\port3) or use another dedicated ENI (ie ENI3\port4) for FGCP HA communication to perform tasks such as heartbeat checks, configurationport4 sync, and session sync. In FortiOS 7.0.x and newer versions, one ENI will be used for both HA management and communication channels.

The FortiGates are configured to use the unicast version of FGCP by applying the configuration below on both the master and slave FortiGate instances. This configuration is automatically configured and bootstrapped to the instances when deployed by the provided CloudFormation or Terraform Templates.

Example Master FGCP Configuration:

config system vdom-exception
edit 1
set object sytem.interface
next
edit 2
set object router.static
next
end
config system ha
set group-name "group1"
set mode a-p
set hbdev "port3" 50
set session-pickup enable
set ha-mgmt-status enable
config ha-mgmt-interface
edit 1
set interface port3
set gateway 10.0.3.1
next
end
set override disable
set priority 255
set unicast-hb enable
set unicast-hb-peerip 10.0.30.10
end

Example Slave FGCP Configuration:

config system vdom-exception
edit 1
set object sytem.interface
next
edit 2
set object router.static
next
end
config system ha
set group-name "group1"
set mode a-p
set hbdev "port3" 50
set session-pickup enable
set ha-mgmt-status enable
config ha-mgmt-interface
edit 1
set interface port3
set gateway 10.0.30.1
next
end
set override disable
set priority 1
set unicast-hb enable
set unicast-hb-peerip 10.0.3.10
end

The FortiGate instances will make calls to the public AWS EC2 API to update AWS SDN to failover both inbound and outbound traffic flows to the new master FortiGate instance. There are a few components that make this possible.

FortiOS will assume IAM permissions to access the AWS EC2 API by using the IAM instance role attached to the FortiGate instances. The instance role is what grants the required permissions for FortiOS to:

• Reassign cluster EIPs assigned to primary IPs assigned to the data plane ENIs
• Update existing routes to target the new master instance ENIs

The FortiGate instances will utilize their independent and direct internet access available through the FGCP HA management interface (ie ENI2\port3) to access the public AWS EC2 API. It is critical that this ENI is in a public subnet with an EIP assigned so that each instance has independent and direct access to the internet or the AWS SDN will not reference the current master FortiGate instance which will break data plane traffic.

config system dns
set primary 169.254.169.253
end

For further details on FGCP and its components, reference the High Availability chapter in the FortiOS Handbook on the Fortinet Documentation site.

Failover Process

The following network diagram will be used to illustrate a failover event from the current master FortiGate (FortiGate 1), to the current slave FortiGate (FortiGate 2).

Inbound failover is provided by reassigning the EIPs associated to the primary IP address of ENI0\port1 from FortiGate 1’s public interface to FortiGate 2’s public interface.

Outbound failover is provided updating any route targets referencing FortiGate 1’s private interface reference FortiGate 2’s private interface.

The AWS SDN updates are performed by FortiGate 2 initiating API calls from the dedicated HA management interface (ie ENI2\port3) through the AWS Internet Gateway.

diag debug enable
diag test app awsd 4

diag debug enable
diag debug app awsd -1

diag debug reset
diag debug disable

Here is an example of running test and debug commands before a failover event to get more verbose output. Notice the EIP and VPC route replacements to reference the new master FortiGate 2.

Templates

It is best practice to use Infrastructure as Code (IaC) templates to deploy FGCP in AWS as there are quite a bit of components that make up the entire solution. These can be used to deploy with a new VPC or to use an existing VPC. You can also integrate with a new or existing Transit Gateway as well.

Reference the CloudFormation and Terraform templates in the Github repos below and reference the quick start guides for how to use these for a deployment.

fortigate-aws-ha-dualaz-cloudformation

• Repo
• Quick Start

fortigate-aws-ha-dualaz-terraform

• Repo
• Quick Start

Post Deployment

1. When using CloudFormation, your stack will have outputs you can use to login to the FortiGates via the cluster or dedicated EIPs. If you used Terraform, these outputs will be listed in your terminal session.

2. Using the login information in the stack outputs, login to the master FortiGate instance with the ClusterLoginURL. This should put you on FortiGate 1. You will also be prompted to change the initial password for the admin account.

3. Navigate to the HA status page on the master FortiGate by going to System > HA. Now you should see both FortiGate 1 and FortiGate 2 in the cluster with FortiGate 2 as the current slave.

4. Give the HA cluster time to finish synchronizing their configuration and update files. You can confirm that both the master and slave FortiGates are in sync by looking at the Status column and confirming there is a green check next to both FortiGates and the status is Synchronized.

*** Note: Due to browser caching issues, the icon for Synchronization status may not update properly after the cluster is in-sync. So either close your browser and log back into the cluster or alternatively verify the HA config sync status with the CLI command ‘get system ha status’. ***

diag debug cloudinit show

5. Navigate to the AWS EC2 console and reference the instance Detail tab for FortiGate 1. Notice the primary IPs assigned to the instance ENIs as well as the 2 EIPs associated to the instance, the Cluster EIP and the HAmgmt EIP.

6. Now reference the instance Detail tab for FortiGate 2. Notice the primary IPs assigned to the instance ENIs and only one EIP is the HAmgmt EIP.

7. Navigate to the AWS VPC console notice the default route in the VPC route table with a next hop targeting ENI1\port2 of FortiGate 1 which is the current master. This VPC route table is applied to the TGW attach subnets to direct traffic from Transit Gateway to the current master FortiGate private interface.

8. Navigate back to the AWS EC2 console and reference the instance Detail tab for FortiGate 1. Now shutdown FortiGate 1 via the EC2 console and refresh the page after a few seconds. Notice that the Cluster EIP is no longer assigned to FortiGate 1.

9. Now reference the instance Detail tab for FortiGate 2. Notice that the Cluster EIP is now associated to FortiGate 2.

10. Navigate back to the AWS VPC console and look at the routes for the default route table. The default route target is now pointing to ENI1\port2 of FortiGate 2.

11. Now log back into the cluster_login_url and you will be placed on the current master FortiGate, which should now be FortiGate 2.

12. Now power on FortiGate 1 and confirm that it joins the cluster successfully as the slave and FortiGate 2 continues to be the master FortiGate.

13. This concludes the post deployment example.

Use Cases

FAQs

• Does FGCP support having multiple Cluster EIPs and secondary IPs on ENI0\port1?

Yes. FGCP will move over any secondary IPs associated to ENI0\port1 and EIPs associated to those secondary IPs to the new master FortiGate instance. You will need to configure secondary IPs on the ENI via the AWS EC2 Console and in FortiOS for port1. The private IPs configured on the ENI and FortiOS must match.

• Does FGCP support having multiple routes for ENI1\port2?

Yes. FGCP will move any routes (regardless of the network CIDR) found in AWS route tables that are referencing any of the current master FortiGate instance’s data plane ENIs (ENI0\port1, ENI1\port2, etc) to the new master on a failover event.

• What VPC configuration is required when deploying either of the existing VPC CloudFormation templates?

The existing VPC CloudFormation template is expecting the same VPC configuration that is provisioned in the new Base VPC template. The existing customer VPC would need to have 4 subnets in each of the two availability zones to cover the required Public, Private, HAsync, and HAmgmt subnets. Also ensure that an S3 gateway endpoint deployed and assigned to both of the PublicSubnet’s AWS route table. Another critical point is that all of the Public and HAmgmt subnets need to be configured as public subnets. This means that an IGW needs to be attached to the VPC and a route table, with a default route using the IGW, needs to be associated to the Public and HAmgmt subnets.

• During a failover test we see successful failover to a new master FortiGate instance, but then when the original master is online, it becomes master again.

The master selection process of FGCP will ignore HA uptime differences unless they are larger than 5 minutes. The HA uptime is visible in the GUI under System > HA. This is expected and the default behavior of FortiOS but can be changed in the CLI under the ‘config system ha’ table. For further details on FGCP master selection and how to influence the process, reference primary unit selection section of the High Availability chapter in the FortiOS Handbook on the Fortinet Documentation site.

• During a failover test we see FGCP select a new master but AWS SDN is not updated to point to the new master FortiGate instance.

Confirm the FortiGates configuration are in-sync and are properly selecting a new master by seeing the HA role change as expected in the GUI under System > HA or CLI with ‘get sys ha status’. However during a failover the routes, and Cluster EIPs are not updated, then your issue is likely to do with direct internet access via HAmgmt interface (ENI2\port3) of the FortiGates, failed DNS resolution, or IAM instance role permissions issues.

For details on the IAM instance profile configuration that should be used, reference the policy statement attached to the ‘iam-role-policy’ resource in any of the CloudFormation templates.

For the HAmgmt interface, confirm this is configured properly in FortiOS under the ‘config system ha’ section of the CLI. Reference the example master\slave CLI HA configuration in the Solutions Components section of this document.

Also confirm that subnet the HAmgmt interface is associated to, is a subnet with public internet access and that this interface has an EIP associated to it. This means that an IGW needs to be attached to the VPC, and a route table with a default route to the IGW needs to be associated to the HAmgmt subnet.

Finally, the AWS API calls can be debugged on the FortiGate instance that is becoming master with the following CLI commands:

diag deb app awsd -1
diag deb enable

This can be disabled with the following CLI commands:

diag deb app awsd 0
diag deb disable

• Is it possible to validate that both FortiGate instances are able to assume the IAM instance role and successfully and access AWS EC2 API endpoints without a failover event?.

Yes. You can run the following command on either FortiGate to confirm availability. Remember that all DNS queries and HTTPS calls to AWS EC2 API endpoints will be sent out of the HAmgmt interfaces. To get more verbose output, you can use the diag debug commands shown above.

diag test app awsd 4

• Is it possible to remove direct internet access from the HAmgmt subnet and provide private AWS EC2 API access via a VPC interface endpoint?

Yes. However, there are a few caveats to consider.

First, a dedicated method of access to the FortiGate instances needs to be setup to allow dedicated access to the HAmgmt interfaces. This method of access should not use the master FortiGate instance so that either instance can be accessed regardless of the cluster status. Examples of dedicated access are Direct Connect or IPsec VPN connections to an attached AWS VPN Gateway. Reference AWS Documentation for further information.

Second, the FortiGates should be configured to use the ‘169.254.169.253’ IP address for the AWS intrinsic DNS server as the primary DNS server to allow proper resolution of AWS API hostnames during failover to a new master FortiGate. Here is an example of how to configure this with CLI commands:

config system dns
set primary 169.254.169.253
end

Finally, the VPC interface endpoint needs to be deployed into both of the HAmgmt subnets and must also have ‘Private DNS’ enabled to allow DNS resolution of the default AWS EC2 API public hostname to the private IP address of the VPC endpoint. This means that the VPC also needs to have both DNS resolution and hostname options enabled as well. Reference AWS Documentation for further information.

• Is it possible to further restrict the IAM policy used by the FortiGates to specific resources?

Yes. You can use resource definitions like shown below to restrict the actions of ’ec2:ReplaceRoute’ and ’ec2:AssociateAddress’ to specific VPC route table IDs and a set of interfaces, EIP allocation IDs, and instances. Reference AWS Documentation for further information.

{
	"Version": "2012-10-17",
	"Statement": [{
			"Sid": "BootStrapFromS3",
			"Effect": "Allow",
			"Action": [
				"s3:GetObject"
			],
			"Resource": "*"
		},
		{
			"Sid": "SDNConnectorFortiView",
			"Effect": "Allow",
			"Action": [
				"ec2:DescribeRegions",
				"eks:DescribeCluster",
				"eks:ListClusters",
				"inspector:DescribeFindings",
				"inspector:ListFindings"
			],
			"Resource": "*"
		},
		{
			"Sid": "HAGatherInfo",
			"Effect": "Allow",
			"Action": [
				"ec2:DescribeAddresses"
				"ec2:DescribeInstances",
				"ec2:DescribeRouteTables",
				"ec2:DescribeVpcEndpoints"
			],
			"Resource": "*"
		},
		{
			"Sid": "FailoverEIPs",
			"Effect": "Allow",
			"Action": "ec2:AssociateAddress",
			"Resource": [
				"arn:aws:ec2:us-east-2:123456789012:elastic-ip/eipalloc-0dfae290176fc15d9",
				"arn:aws:ec2:us-east-2:123456789012:network-interface/*",
				"arn:aws:ec2:us-east-2:123456789012:instance/*"
			]
		},
		{
			"Sid": "FailoverVPCroutes",
			"Effect": "Allow",
			"Action": "ec2:ReplaceRoute",
			"Resource": "arn:aws:ec2:us-east-2:123456789012:route-table/rtb-0d5d1757917c71c6e"
		}
	]
}

• Is it possible to restrict the instance metadata service (IMDS) version to v2 only?

Yes. Starting in FortiOS 6.4.3 GA, you can modify this via the AWS CLI command ‘aws ec2 modify-instance-metadata-options’ to set ‘–http tokens’ to required. This disables the v1 version of the IMDS for that instance. Reference AWS Documentation for further information.

aws ec2 modify-instance-metadata-options --instance-id i-01234567890123456 --http-tokens required --region us-west-2