Fortinet XPerts 2024

You will learn how FortiGates can address the following standard concerns and requirements when migrating applications and workloads to public cloud.

Common Concerns:

  • Exposure to inbound internet attacks
  • Environment and application segmentation to reduce exploit blast radius
  • Next generation firewall (NGFW) protection and URL filtering for outbound web traffic
  • Simple security policy across corporate cyber infrastructure

Common Requirements:

  • Regional, highly available architecture (multiple availability zones (AZs))
  • NGFW protection featuring FortiGate FortiGuard advanced protection
  • Logging of all traffic

Workshop Goals

You will learn how to use FortiGate NGFW deployed as AWS EC2 instances to protect traffic flows in Advanced AWS architecture patterns, as well as some fundamental AWS networking concepts.

The intent is to help clarify the following:

  • Advanced AWS networking concepts such as VPC Peering, Transit Gateway (TGW), and Gateway Load Balancer (GWLB)
  • Use of FortiGate instances in AWS to secure inbound, outbound, and East/West traffic flows
  • AWS architectures with Transit Gateway (TGW) and Gateway Load Balancer (GWLB), and the FortiGate security VPC concept
Version:
Last updated: Mon, Jun 23, 2025 18:30:30 UTC
Copyright© 2025 Fortinet, Inc. All rights reserved. Fortinet®, FortiGate®, FortiCare® and FortiGuard®, and certain other marks are registered trademarks of Fortinet, Inc., and other Fortinet names herein may also be registered and/or common law trademarks of Fortinet. All other product or company names may be trademarks of their respective owners. Performance and other metrics contained herein were attained in internal lab tests under ideal conditions, and actual performance and other results may vary. Network variables, different network environments and other conditions may affect performance results. Nothing herein represents any binding commitment by Fortinet, and Fortinet disclaims all warranties, whether express or implied, except to the extent Fortinet enters a binding written contract, signed by Fortinet’s General Counsel, with a purchaser that expressly warrants that the identified product will perform according to certain expressly-identified performance metrics and, in such event, only the specific performance metrics expressly identified in such binding written contract shall be binding on Fortinet. For absolute clarity, any such warranty will be limited to performance in the same ideal conditions as in Fortinet’s internal lab tests. Fortinet disclaims in full any covenants, representations, and guarantees pursuant hereto, whether express or implied. Fortinet reserves the right to change, modify, transfer, or otherwise revise this publication without notice, and the most current version of the publication shall be applicable.

Subsections of Fortinet XPerts 2024

Introduction

FortiGate: Protecting AWS Traffic Flows in Advanced Architectures

Learning Objectives

Upon completion of this workshop, you will gain an understanding of:

  • AWS Advanced networking architectures
  • FortiGate deployments to secure traffic in each of these architectures
  • AWS VPC Peering
  • AWS Transit Gateway (TGW)
  • AWS Gateway Load Balancer (GWLB)
  • AWS Cloud WAN (CWAN)
  • Creating and applying firewall policies with security profiles and objects to control traffic flows
  • Testing traffic flows to validate the implemented networking and security controls

Workshop Components

The following Fortinet & AWS components will be used during this workshop:

  • AWS EC2 instances (Amazon Linux OS, FortiGate NGFW)
  • AWS networking components:
    • VPCs
    • Subnets
    • VPC Peering
    • Route Tables (RTBs)
    • Transit Gateway (TGW)
    • Gateway Load Balancer (GWLB)
    • Global & Core Networks
    • Core Network Edge (CNE)
    • Core Network Policy
    • Network Segment
  • FortiGate instances running FortiOS (Amazon Machine Images (AMI) on EC2)

AWS Architecture Patterns

AWS networking offers multiple ways to organize your AWS architecture to take advantage of FortiGate traffic inspection. The most important part of designing your network is to ensure traffic follows a symmetrical routing path (for forward and reverse flows). As long as flows are symmetrical, the architecture will work and traffic will flow through the FortiGate NGFW for inspection.

We will investigate the configuration of different architecture patterns, including:

  • VPC Peering
  • FortiGate Centralized Architecture for Selective inspection of East/West & Egress traffic
  • FortiGate Centralized Architecture with Dynamic Routing for inspection of East/West & Egress Traffic with multiple VPCs
  • FortiGate Centralized & Distributed Inspection architectures featuring AWS GWLB

QuickStart & FAQ

This section contains a few quick tips and tricks to help you get started on this workshop. See the quick links below to jump straight into it, otherwise navigate to the workshop logistics section to begin.

Subsections of QuickStart & FAQ

Workshop Logistics

Accessing Qwiklabs & AWS environment

For Fortinet cloud workshops, we will use Qwiklabs to provide a pre-staged AWS account with the workshop environment ready to use.

  • After logging into Qwiklabs, click Dashboard (top of page).
  • Find and select FortiGate: AWS Networking 201 Workshop.
  • Click the green Start Lab button to begin.
Warning

Provisioning the lab is fairly quick once started. This environment will then run for 4 hours and then automatically shut down and clean up all resources. If you restart the lab, you will start with a clean environment where you can restart the last task from the beginning, that was not 100% completed, or move on to the next task.

Once the environment has finished provisioning, check the Qwiklabs Console left menu (see screenshot below) to find your AWS login URL and credentials, along with other useful information you will need throughout your lab. You may now proceed to the hands-on section of the lab.

AWS Networking Concepts

AWS Network Routing

  • AWS networking is Software Defined Networking where VPCs have a built-in or implicit router
  • VPC Route Tables (RT) are similar to static routes in traditional routing
    • Generally they are attached to a subnet impacting the destination routing decisions for that subnet only
    • Putting the pieces together, AWS routing decisions happen at every hop along the traffic path
Tip

Symmetrical paths are the most important rule of AWS routing

  • Traffic must follow the same path in the outbound and inbound direction
  • Sometimes RT can be associated with services (like IGW) allowing special routing decisions in specific scenarios
  • Every VPC has a default RT which allows communication between all subnets in the VPC
    • Every newly created subnet is associated with this default RT
    • You cannot change the VPC CIDR route entry in the default RT
    • You can add more specific subnet routes to a RT to do things like FortiGate NGFW inspection of traffic BETWEEN subnets in a VPC

AWS Networking Services

Before diving into the reference architecture for this workshop, let’s review core AWS networking concepts. Click the hyperlinks for AWS documentation.

Service NameDefined byLimitationsAffinity
VPCCIDR block (can add multiple CIDR blocks)CIDR block between /16 and /28lives in 1 region
RegionLarge geographical areaisolated from other regions (which can be connected with TGW)can have many VPCs
Availability Zonegenerally 1 physical datacenter/MANpower, cooling, and networking are shared in an AZeach region has multiple AZ’s
SubnetIP subnet of the VPC’s CIDR definitionCIDR block between /16 and /28belongs to 1 VPC and 1 AZ only
Internet GatewayVPC ConfigurationVPC Route Tables (RTBs) can be associated to IGW to influence routing beyond subnet (advanced scenario)belongs to 1 VPC
NAT Gateway1 (or more) EIPsoutbound traffic onlybelongs to 1 subnet(AZ)
Elastic IPsingle Public IP addressbelongs to a regionattachment to EC2, NLB/ALB, NATGW
VPC Peeringpeering connection specifying 2 VPCs onlyno transitive routingrouting path between 2 VPCs only (legacy)
TGWTGW routing tables, VPC, VPN, DXC and other attachments5,000 attachments per transit gatewaybelongs to 1 region

AWS Networking Base Services

  • Virtual Private Cloud (VPC) is a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways.

  • Region is a collection of multiple Availability Zones in a geographic location. The collection of AZs in the same region are all interconnected via redundant, ultra-low-latency networks.

  • Availability Zones (AZ) are multiple, isolated datacenter locations within each Region that have independent power, cooling, physical security, etc. A VPC spans all of the AZs in the Region.

  • All subnets within a VPC are able to reach each other with the default or intrinsic router within the VPC. All resources in a subnet use the intrinsic router (1st host IP in each subnet) as the default gateway. Each subnet must be associated with a VPC route table, which specifies the allowed routes for outbound traffic leaving the subnet. Subnets are tied to a single AZ.

  • Internet Gateway (IGW) is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet. It therefore imposes no availability risks or bandwidth constraints on your network traffic.

  • Elastic IP (EIP) is a static IPv4 address designed for dynamic cloud computing within a region. Simply allocate one to a region in your account and associate this to an EC2 instance to access it over the public internet.

  • NAT Gateway (NAT GW) is a Network Address Translation (NAT) service. You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances.

AWS Network Connectivity Services

  • VPC Peering is a private and point to point connection between two VPCs. While you can connect many VPCs together in the same region, VPC peering does not support transitive routing. In the diagram below, VPC A has peering connections to both VPC B & C, so A can talk to B or C. However, VPC B can’t talk to VPC C, and the reverse as well, unless you setup a VPC peering connection directly between them. As VPCs scale, it is better to use Transit Gateway instead.
  • Transit Gateway (TGW) is a highly scalable cloud router that connects your VPCs in the same region to each other, to on-premise networks, and even to the internet through one hub. With the use of multiple route tables for a single TGW, you can design hub and spoke routing for traffic inspection and enforcement of security policy across multiple VPCs. As the name implies, Transit Gateway supports transitive routing.
  • AWS Gateway Load Balancer (GWLB) is a transparent network gateway that distributes traffic (in a 3/5 tuple flow aware manner) to a fleet of virtual appliances for inspection. This is a regional load balancer that uses GWLB endpoints (GWLBe) to securely intercept data plane traffic within consumer VPCs in the same region.
  • AWS Cloud WAN (CWAN) is an intent-driven managed wide area network (WAN), described by a policy you define that unifies your data center, branch, and AWS networks. While you can create your own global network by interconnecting multiple Transit Gateways across Regions, Cloud WAN provides built-in automation, segmentation, and configuration management features designed specifically for building and operating global networks, based on your core network policy. Cloud WAN has added features such as automated VPC attachments, integrated performance monitoring, and centralized configuration.

AWS Data transfer cost considerations

Below is a simplified description of the most basic AWS networking & data transfer charges. The listed costs are examples only and change frequently based on the region in use. Please consult AWS documentation for costs of your specific scenario

Note

Any specific pricing listed below is for estimation purposes only. Please reference the AWS Cost Calculator for the most up-to-date pricing for your infrastructure.

Tip

Workshop participants running this workshop during a Fortinet event will not incur any AWS charges.

Transfer descriptioncost ($)
Inbound (from Internet)$0.00 (Free)
Outbound (to Internet)$0.09/GB (volume discounts apply)
Within an AZ$0.00 (Free)
Across AZs (within Region)$0.01/GB
Across Regions$0.01-0.02/GB
AWS Service costscost ($)
VPC Peering$0.00 (Free)
CWAN$0.02/GB + $0.50/hr (per CNE) + $0.065-0.09 (per attachment) See AWS CWAN Pricing
TGW$0.02/GB + $0.05/hr (per attachment) See AWS TGW Pricing
IGW$0.00 (Free) See AWS VPC Pricing
NATGW$0.045/GB + $0.045/hr See AWS VPC Pricing
EIP$0.005/hr See AWS VPC Pricing

In this workshop we will use these components to highlight insertion of FortiGate NGFW into an enterprise architecture.

AWS Tips and tricks

Helpful tips when working with AWS console

  • Upon login the Console Home page is displayed.
  • Use the Search Box at the top to search for services or features such as EC2, VPC, Network Manager, etc.
  • When results appear, right-click the service name to open the desired console in a new tab.
    • Multiple tabs simplify navigation when using multiple AWS services.

General

  • It’s often helpful to open each new service in its own tab. That way, you can refer back to each service independently without losing the info on your screen.
    • This trick is also useful when working in multiple regions. Just be careful you make a change in the appropriate region!
  • Search/filter within a service
    • Often, when working with AWS services, it can be confusing matching a particular item to its “owner”
      • e.g. Every Subnet and RT belongs to a single VPC.
      • Identifying a specific RT from a long list of all RT in a region can be difficult
      • You can use the filters in left-nav, or top-middle search bar to filter and narrow down the viewable items
      • In the VPC Dashboard, you can filter subnets or RT to a specific VPC to help more easily identify the one you need
  • Working in different AWS regions
    • The AWS console displays services for 1 Region only.
    • If you need to work with resources in different regions, you can open multiple browser tabs to view Service consoles in different regions
    • The example below shows VPCs in US-WEST-1(N. California) & US-East-1(N. Virginia)
    • Be careful when working between regions, ensuring you make changes to the appropriate region’s services

Workshop Hands-On

Tasks in this workshop

In this workshop, we will explore AWS common networking architectures and FortiGate Deployments to secure traffic flows in them. These architectures will be used to demonstrate how to secure traffic flows for the following services and network scenarios:

  1. VPC Peering
  2. Transit Gateway with VPC attachments
  3. Transit Gateway with Connect and VPN attachments
  4. Gateway Load Balancer w/ distributed and centralized networking
  5. Cloud WAN with VPC and Connect attachments
Note

EC2 Instance Connect

In each of these scenarios, you’ll need to connect to EC2 instances via Instance Connect.

From the AWS console, follow these directions to connect to the specific instance for the given task instruction:

  • In the EC2 Console go to the Instances page select the TASK_SPECIFIC_INSTANCE.
  • Click Connect > EC2 serial console.
    • Copy the instance ID as this will be the username and click Connect.
  • Login to the EC2 instance:
    • You may need to hit ENTER to get a login prompt
      • username: «copied Instance ID from above»
      • Password: FORTInet123!

Subsections of Workshop Hands-On

VPC Peering

GoalEstablish point to point access from VPC A to VPC B
TaskCreate a peering connection and VPC routes so EC2 Instance-A can ping Instance-B
ValidationConfirm point to point VPC connectivity with ping from EC2 Instance-A to B and Instance-B to C

Introduction

In this task, there are multiple VPCs in the same region that have one instance each. VPC peering and the appropriate VPC routes are already configured for VPC-B to VPC-C. VPC peering and VPC routes will need to be configured between VPC-A and VPC-B. Then traffic will be tested to confirm what traffic routing is and is not supported with VPC peering.

Warning

There are no security controls in this example. Instance-B can freely communicate with the Internet.

Summarized Steps (click to expand each for details)

0) Lab environment setup
Detailed Steps…
  • 0.1: Login to your AWS account, and click this Launch Stack button to Launch the CloudFormation Stack for Task 1

  • 0.2: You must:

    • select an IAM role in the Permissions section
    • check the boxes to acknowledge the warnings in the Capabilities section
    • then scroll down and click Create stack
Warning

If you do not select a the IAM role and continue with stack creation, this will fail! If this occurred, simply create another stack with a different name and follow the steps closely for this section.

  • 0.3: The CloudFormation stack will take ~5 minutes to finish deploying. Once the main/root CloudFormation stack shows as Create_Complete, proceed with the steps below.

1) Find EC2 Instance-A and verify it can’t access Instance-B or C
Detailed Steps…
  • 1.1: In your AWS account, navigate to the EC2 Console and go to the Instances page (menu on the left).

  • 1.2: Find Instance-A and connect to it using the Serial Console directions

    • Password: FORTInet123!
  • 1.3: Run the following commands to test connectivity and make sure the results match expectations

    SRC / DSTVPC BVPC CInternet
    Instance Aping 10.2.2.10 FAIL !!!ping 10.3.2.10 FAIL !!!
    Instance Acurl ipinfo.io FAIL !!!
  • 1.4: Run the command ifconfig ens5 and take note of the instance IPv4 address.

  • 1.5: Run the command route -n and take note of the default route and the gateway IP for that route.

Info

The instance has a default route going to the private IP of 10.1.2.1. This is the AWS VPC router (aka intrinsic router), which is the first host IP in any subnet within a VPC. Reference this AWS document to learn what other IPs in a subnet are always used by AWS.

2) Create a VPC peering connection and VPC routes to allow Instance-A in VPC-A to reach Instance-B in VPC-B
Detailed Steps…
  • 2.1: Navigate to the VPC Console and go to the Peering connections page (menu on the left) and click Create peering connection.
  • 2.2: Provide a name then select VPC-A as the requester and VPC-B as the Accepter and click Create peering connection at the bottom of the page.
  • 2.3: On the next page, click Actions and select Accept Request, and again on the pop-up window.
  • 2.4: Go to the VPC Route tables page (menu on the left) and find VPC-A-PrivateRouteTable , select the Routes tab and click Edit Routes.
  • 2.5: Create a route for 0.0.0.0/0 with the peering connection you just created as your target.
  • 2.6: Repeat the same steps above to create a route for 10.1.0.0/16 in VPC-B-PrivateRouteTable to allow reply traffic.
3) Verify communication over the VPC Peering connection
Detailed Steps…
  • 3.1: Navigate to the EC2 Console and connect to Instance-A using the Serial Console directions

    • Password: FORTInet123!
  • 3.2: Run the following commands to verify connectivity and make sure results match expectations

    SRC / DSTVPC BVPC CInternet
    Instance Aping 10.2.2.10 SUCCESS !!!ping 10.3.2.10 FAIL !!!
    Instance Acurl ipinfo.io FAIL !!!
    • Now you can successfully ping Instance-B over the peering connection.
    • Instance-A SHOULD NOT be able to ping Instance-C or access the Internet through VPC-B.
4) Let’s dig deeper to understand how VPC Peering works
Detailed Steps…
Info
  • VPC Peering permits point to point connectivity between resources in 2 directly peered VPC’s and nothing else
  • Transitive peering or peering from VPC A through VPC B to VPC C is not permitted
  • Accessing AWS Services (like NAT GW or Internet GW) via a peering connection does not work
  • 4.1: In the EC2 Console go to the Instances page connect to Instance-B using the Serial Console directions

    • Password: FORTInet123!
  • 4.2: Run the following commands to verify connectivity and make sure results match expectations

    SRC / DSTInstance AInstance CInternet
    Instance Bping 10.1.2.10 SUCCESS !!!ping 10.3.2.10 SUCCESS !!!
    Instance Bcurl ipinfo.io SUCCESS !!!
  • 4.3: In the EC2 Console go to the Instances page connect to Instance-C using the Serial Console directions

    • Password: FORTInet123!
  • 4.4: Run the following commands to verify connectivity and make sure results match expectations

    SRC / DSTInstance AInstance BInternet
    Instance Cping 10.1.2.10 FAIL !!!ping 10.2.2.10 SUCCESS !!!
    Instance Ccurl ipinfo.io FAIL !!!
    • Instance-C should ping Instance-B but not be able to ping Instance-A or access the internet through VPC-B.
Info

The VPC peering connection is at the VPC level. This means the VPC peering connection is not directly tied to any VPC subnet or route table explicitly. AWS routing for VPC peering connections will only deliver traffic to an IP address that is within the destination VPC CIDR. The routes you created in both VPC-A and B’s private route tables only direct traffic out of the local VPC to the target destination VPC. Reference this AWS documentation to learn more about the limitations of VPC Peering.

  • 4.5 Below is a step by step of the packet handling for the traffic from Instance-A to Instance-B.
HopComponentDescriptionPacket
1Instance-A -> 0.0.0.0/0 PCXOutbound traffic destined to Instance-B is sent to the VPC router (its default gw) which routes traffic to the VPC peering connection as configured in the VPC-A-PrivateRouteTable.10.1.2.10:src-port -> 10.2.2.10:dst-port
2VPC Router -> Instance-BInbound traffic leaves the VPC peering connection and is sent to the VPC router which delivers traffic directly to Instance-B.10.1.2.10:src-port -> 10.2.2.10:dst-port
3Instance-B -> 10.1.0.0/16 PCXInstance-B receives the traffic, seeing the original private source IP, and replies. This traffic is sent to the VPC router (its default gw) which routes the traffic to the VPC peering connection as configured in the VPC-B-PrivateRouteTable.10.2.2.10:src-port -> 10.1.2.10:dst-port
4VPC Router -> Instance-AResponse traffic leaves the VPC peering connection and is sent to the VPC router which delivers traffic directly to Instance-A.10.2.2.10:src-port -> 10.1.2.10:dst-port
Warning

When Instance-A attempts to ping Instance-C or access the Internet through VPC-B (using the default route), the VPC router at step 2 above would drop the traffic as the destination does not match the CIDR of VPC-B.

  • 4.6 Below is a step by step of the packet handling for the traffic from Instance-B to the internet.
HopComponentDescriptionPacket
1Instance-B -> 0.0.0.0/0 NAT GWInstance-B sends outbound traffic to the VPC router (it’s default gw) which routes the traffic to NAT GW as configured in the VPC-B-PrivateRouteTable.10.2.2.10:src-port -> x.x.x.x:80
2NAT GW -> 0.0.0.0/0 IGWNAT GW changes the source IP to its own private IP and sends the traffic to VPC router. The VPC router routes traffic to IGW as configured in the VPC-B-PublicRouteTable.y.y.y.y:src-port -> x.x.x.x:80
3IGW -> InternetIGW changes the source IP to the associated EIP of NAT GW and routes the traffic to the internet.z.z.z.z:src-port -> x.x.x.x:80
4Internet -> IGWIGW receives reply traffic, changes the DEST IP to the private IP of NAT GW, and sends the traffic to VPC router. The VPC router routes traffic to the NAT GW.x.x.x.x:80 -> y.y.y.y:dst-port
5NAT GW -> Instance-BNAT GW changes the DEST IP back to the private IP of Instance-B and routes the traffic to the VPC router which delivers the traffic to Instance-B.x.x.x.x:80 -> 10.2.2.10:dst-port

5) Lab environment teardown
Detailed Steps…
  • 5.1: Before deleting the main CloudFormation Stack, we must remove the VPC routes referencing the VPC peering connection, and the VPC peering connection itself.
  • 5.2: Navigate to the VPC Console and go to the Peering connections page (menu on the left), select the peering for VPC-A to B and click Actions, then select Delete peering connection. This will prompt you to delete the related route table entries. Select Delete related route table entries, then to confirm, type delete in the field and click Delete.
  • 5.3: Navigate to the CloudFormation Console, select the main stack you created and click Delete.
  • 5.4: The CloudFormation stack will take ~5 minutes to clean up. Once the stack is deleted, proceed to the next task.

Discussion Points

  • VPC peering is a point-to-point connection only (no transitive routing)
  • It is best used when a few trusted VPCs are transferring large amounts of data between each other (ie DB replication)
  • Full mesh is required to connect all VPCs together
    • For example connecting 16 VPCs in full mesh would require (16*15)/2 = 120 connections
    • The max supported VPC peering connections per VPC is 125
  • VPC peering supports connections between VPCs:
    • In the same or different AWS Accounts
    • In the same (intra) or across (inter) regions
  • When using inter-region peering, AEAD encryption is used
  • No Internet Gateway is required for VPC Peering (even when inter-region)
  • Jumbo frames (9001 bytes) are only supported for intra-region connections, inter-region is limited to 1500 bytes

This concludes this task

Transit Gateway w/ VPCs

GoalEstablish open, secured east/west (Inter-VPC), and outbound flows through the provisioned NGFW VPC and Transit gateway architecture.
TaskCreate attachment associations + propagations and configure FortiGate routes and firewall policy to allow secured traffic.
ValidationConfirm outbound and east/west connectivity from EC2 Instance-A via Ping, HTTP, HTTPS.

Introduction

In this task, there are multiple VPCs in the same region that have one instance each. Transit Gateway (TGW) is configured with multiple Transit Gateway Route Tables. You will need to create the appropriate VPC attachment associations and propagations to the correct TGW Route Tables, FW policy and update a static route on the FortiGate Active-Passive cluster.

In this scenario you will allow traffic between the workload and shared services VPC to communicate directly (without going through the FortiGate A-P cluster), while inspecting outbound and east/west connectivity for the workload VPCs.

Summarized Steps (click to expand each for details)

0) Lab environment setup
Detailed Steps…
  • 0.1: Login to your AWS account, and click the Launch Stack button below to launch the CloudFormation stack for Task 2

  • 0.2: You must:
    • select an IAM role in the Permissions section
    • check the boxes to acknowledge the warnings in the Capabilities section
    • then scroll down and click Create stack
Warning

If you do not select a the IAM role and continue with stack creation, this will fail! If this occurred, simply create another stack with a different name and follow the steps closely for this section.

  • 0.3: The CloudFormation stack will take ~10 minutes to finish deploying. Once the main/root CloudFormation stack shows as Create_Complete, proceed with the steps below.
1) Inspect the Transit Gateway VPC attachments
Detailed Steps…
  • 1.1: In the VPC Console go to the Transit gateway attachments page (menu on the left), then find and select the VPC-A-spoke-vpc-attachment attachment.
  • 1.2: On the Details tab, click 2 Subnets under Subnet IDs.
  • 1.3: Copy the Transit gateway attachment ID and navigate to the EC2 Console.
  • 1.4: On the Network Interfaces page, paste the attachment ID into the search field and find the two interfaces.
Info

Notice that the VPC attachment is associated or attached to two subnets in VPC-A. This means VPC route tables can be used to direct traffic that comes through the attachment. This is what allows transitive routing to occur.

2) Create associations and propagations for VPC attachments to Transit Gateway Route Tables for east/west to shared services
Detailed Steps…
  • 2.1: Navigate to the VPC Console and go to the Transit gateway route tables page (menu on the left).
  • 2.2: Each tgw-rtb has an associations and propagations tab.
  • 2.3: Use the table below to create the associations and propagations to setup the environment for general east/west between VPC-A and VPC-B to VPC-C.
TGW-RTB NameAssociationsPropagations
NGFW-security-tgw-rtbNGFW-security-vpc-attachmentVPC-A-spoke-vpc-attachment and
VPC-B-spoke-vpc-attachment
NGFW-spoke-tgw-rtbVPC-A-spoke-vpc-attachment and
VPC-B-spoke-vpc-attachment
VPC-C-spoke-vpc-attachment
NGFW-sharedservices-tgw-rtbVPC-C-spoke-vpc-attachmentVPC-A-spoke-vpc-attachment and
VPC-B-spoke-vpc-attachment
3) Create a default route for centralized east/west and outbound through an inspection VPC
Detailed Steps…
  • 3.1: In the VPC Console and go to the Transit gateway route tables page (menu on the left).
  • 3.2: Select NGFW-spoke-tgw-rtb, select the routes tab and click Create static route.
  • 3.3: Create a route for 0.0.0.0/0 and the NGFW-security-vpc-attachment attachment.
4) Login to the FortiGate GUI, modify a route and add a Firewall Policy allowing east/west
Detailed Steps…
  • 4.1: Navigate to the CloudFormation Console and toggle View Nested to off.
  • 4.2: Select the main template and select the Outputs tab.
  • 4.3: Login to the FortiGate GUI, using the outputs ClusterLoginURL, Username, and Password.
  • 4.4: Upon login, navigate to Network > Static Routes.
  • 4.5: Edit the existing route to 10.0.0.0/16.
    • This route needs to be updated to allow access to both the Spoke VPCs that are part of this centralized design. Change the destination to 10.0.0.0/8 and click OK to save the change.
Tip

Since these FortiGates are in separate availability zones and subnets, the gateway IP address will be different for each FortiGate. Thus, there are exceptions for what configuration can be synchronized such as interface settings and static routes. In production you would need to log into the secondary FortiGate and repeat this step to update the static route so traffic handling is the same after a failover.

  • 4.6: Navigate to Policy & Objects > Firewall Policy and click Use new layout when prompted, then click Create new.
Tip

If you accidentally did not use the new layout, you can change it by clicking on Classic Layout in the top right and switch to Use new layout.

  • 4.7: Create a new policy with the settings shown below and click OK to allow east/west ICMP.

5) Test open east/west connectivity from Instance-A to Instance-C in shared services
Detailed Steps…
  • 5.1: Navigate to the EC2 Console and connect to Instance-A and/or Instance-B using the Serial Console directions
    • Password: FORTInet123!
  • 5.2: Run the following commands to test connectivity and make sure the results match expectations
    SRC / DSTVPC C (Shared Services)
    Instance A or Bping 10.3.2.10 SUCCESS !!!
    Instance A or Bcurl 10.3.2.10 SUCCESS !!!
Tip

Due to the configuration of the Transit gateway route tables, the east/west traffic between VPC-A and VPC-B to VPC-C is not being routed through the inspection VPC. That is why you are able to allow HTTP, SSH, and other traffic between these VPCs. While this may be acceptable for trusted and low security risk environments, it is best practice to have clear visibility and control on what communication is allowed by routing this through a security stack.

6) Test secured east/west and outbound connectivity from Instance-A to Instance-B
Detailed Steps…
  • 6.1: While still in the console session for Instance-A, Run the following commands to test connectivity and make sure the results match expectations

    SRC / DSTVPC BInternet
    Instance Aping 10.2.2.10 SUCCESS !!!ping 8.8.8.8 SUCCESS !!!
    Instance Acurl 10.2.2.10 FAIL !!!curl ipinfo.io FAIL !!!
    Instance Acurl -k https://ipinfo.io SUCCESS !!!
    Instance Acurl -k --retry 2 --retry-all-errors https://secure.eicar.org/eicar.com.txt -o file.txt FAIL !!!
    • A –> B succeeded for ping but not HTTP due to FortiGate Firewall Policy
    • A –> Internet was successful for PING, Web Filtered for HTTPS, and blocked for HTTP due to FortiGate Firewall Policy
  • 6.2: To check the content of the file from the last command, run the command cat file.txt | grep -A 15 'High Security Alert'.

    • You will see you were blocked, and a block page was returned due to the Web Filtering profile
Tip

Due to the configuration of the Transit gateway route tables, the east/west traffic from VPC-A to VPC-B and both VPCs to the internet is being routed through the inspection VPC. This is allowing us to have better visibility and control from both a layer 4 and 7 perspective. This allows us to trust but verify what is allowed and occurring in our environment.

7) Let’s dig deeper to understand how all of this works
Detailed Steps…
Info

Transit Gateway (TGW) is a highly scalable cloud router that connects your VPCs in the same region to each other, to on-premise networks, and even to the internet through one hub. With the use of multiple route tables for a single TGW, you can design hub and spoke routing for traffic inspection and enforcement of security policy across multiple VPCs. As the name implies, Transit Gateway supports transitive routing.

In this section, we used Transit gateway to provide open and direct east/west between certain VPCs A and B, to C while providing controlled east/west and centralized egress for other traffic flows (VPC-A to VPC-B). This allows broad routing decisions to be made on how traffic is handled for anything connecting to Transit gateway within the same region.

Key points to understand is that:

  • a single Transit gateway can have multiple route tables
  • multiple attachment types exist (VPC, VPN, Direct Connect (dedicated circuit), TGW Connect (GRE))
  • each attachment can only be associated to one route table
  • each attachment can be propagated to multiple route tables (ie VPC A and B)
  • additionally static and dynamic routes (covered in next section) can be added to route tables
  • 7.1: In the FortiGate GUI navigate to Log & Report > Forward Traffic. You should see logs for the traffic you generated.
  • 7.2: Double click a log entry to view the Log Details.
Info

The instance has the private IP 10.1.2.10/24, but is seen as coming from a public IP when running ‘curl -k https://ipinfo.io’, etc. This is because the primary FortiGate is providing secured outbound access to the internet for this private EC2 instance. This is because of the VPC routes in all the VPCs (VPC-A and Inspection) are working together with the Transit Gateway (TGW) and Transit Gateway route tables to route the in/outbound traffic through the primary FortiGate. This is a centralized design that is also commonly called an appliance, inspection, or security VPC.

Navigate to Policy & Objects > Firewall Policy and look at the security profiles being applied to the secured-outbound-traffic-with-nat policy. This pre-configured policy is applying source NAT to act as a NAT Gateway but is also applying advanced NGFW protection such as SSL MitM, Application Control, Intrusion Prevention, and Anti-Virus features.

  • 7.3 Below is a step by step of the packet handling for the outbound web traffic from Instance-A.
HopComponentDescriptionPacket
1Instance-A -> 0.0.0.0/0 TGWInstance-A sends outbound traffic to the VPC router (its default gw) which routes traffic to TGW as configured in the Spoke1 VPC RTB.10.1.2.10:src-port -> x.x.x.x:80
2Spoke1-TGW-Attachment -> 0.0.0.0/0 Sec-TGW-AttachmentSpoke1-TGW-attachment is associated to the Spoke VPC TGW RTB. This TGW RTB has a default route to Sec-TGW-Attachment, so traffic is forwarded there.10.1.2.10:src-port -> x.x.x.x:80
3Sec-TGW-Attachment -> 0.0.0.0/0 FGT1-Port2Sec-TGW-Attachment is attached to the Security VPC TGW Attach Subnets which have a default route to primary FortiGate1 Port2, private interface.10.1.2.10:src-port -> x.x.x.x:80
4FGT1-Port1 -> 0.0.0.0/0 IGWFGT1 changes the source IP to the private IP of Port1 as this has an EIP associated. FGT1 sends inspected and allowed traffic to the VPC router via Port1 (its default gw), which routes traffic to the IGW as configured in the Public Subnet VPC RTB.10.0.1.10:src-port -> x.x.x.x:80
5IGW -> InternetIGW changes the source IP to the associated EIP of FortiGate1 Port1 and routes traffic to the internet.z.z.z.z:src-port -> x.x.x.x:80
6Internet -> IGWIGW receives reply traffic and changes the source IP to the private IP of FortiGate1 Port1. The VPC router routes traffic to FortiGate1 Port1.x.x.x.x:80 -> 10.0.1.10:dst-port
7FGT1-Port2 -> 0.0.0.0/0 TGWFGT1 receives traffic on Port1, changes the source IP to the private IP of Instance-A, and routes inspected and allowed traffic to the VPC router (10.0.0.0/8 static route out port2) via Port2. The VPC router sends traffic to TGW as configured in the Private Subnet VPC RTB.x.x.x.x:80 -> 10.1.2.10:dst-port
8Sec-TGW-Attachment -> 10.1.0.0/16 Spoke1-TGW-AttachmentSec-TGW-Attachment is associated to the Sec VPC TGW RTB. This TGW RTB has a route for Spoke1 VPC via Spoke1-TGW-Attachment, so traffic is forwarded there.x.x.x.x:80 -> 10.1.2.10:dst-port
9Spoke1-TGW-Attachment -> Instance-ASpoke1-TGW-Attachment is attached to subnets in Spoke1 VPC which have a local VPC route to reach Instance-A.x.x.x.x:80 -> 10.1.2.10:dst-port

8) Lab environment teardown
Detailed Steps…
  • 8.1: Navigate to the CloudFormation Console, select the main stack you created and click Delete.
  • 8.2: The CloudFormation stack will take ~10 minutes to clean up. Once the stack is deleted, proceed to the next task.

Discussion Points

  • TGW is essentially a regional router
  • TGW supports transitive routing and has many use cases
  • TGW supports the following attachments in the same region:
    • VPC (static propagation of VPC CIDR)
    • VPN (static or dynamic routing)
    • Direct Connect Gateway (static or dynamic routing)
    • TGW Connect (GRE over VPC or Direct Connect attachment, supports static or dynamic routing)
  • Jumbo frames (8500 bytes) are supported for all attachments except VPN (1500 bytes)
  • TGW supports peering directly between TGWs in the same or different regions (supports static routing only)
    • FGTs can be attached to the TGWs to provide dynamic routing between them with an overlay tunnel
  • TGW supports default route table propagation and association settings which can be used to automate connecting spoke VPCs to a simple centralized design
  • Centralized Inspection VPC handles FortiGate NGFW inspection for any traffic flow (Inbound, Outbound, East/West)
    • Advanced architectures for all of these scenarios can be found here

This concludes this task

Transit Gateway w/ BGP

GoalUtilize dynamic routing with Transit Gateway and FortiGates.
TaskCreate attachment associations + propagations and configure FortiGate routes and firewall policies to allow secured traffic to pass.
ValidationConfirm outbound and east/west connectivity from EC2 Instance-A via Ping, HTTP, HTTPS.

Introduction

In this task, there are multiple VPCs in the same region that have one instance each. Transit Gateway is configured with multiple Transit Gateway Route Tables. You will need to create the appropriate VPC attachment associations and propagations to the correct TGW Route Tables, FW policy, and update BPG configuration on the independent FortiGates.

In this scenario the FortiGates are completely independent of each other (not clustered, nor sharing config/sessions, etc.) and are showing different connectivity options to attach remote locations to Transit Gateway. VPN attachments can be used to connect to any IPsec capable device located anywhere. TGW Connect attachments require a private path to reach a VM deployed in a VPC or HW/VM deployed on premise and must be reachable over Direct Connect (a dedicated, private circuit).

Summarized Steps (click to expand each for details)

0) Lab environment setup
Detailed Steps…
  • 0.1: Login to your AWS account, and click the Launch Stack button below to launch the CloudFormation stack for Task 3

  • 0.2: When creating this stack, ensure the following options are configured (See screenshots below for additional guidance):
    • An IAM role in the Permissions section
    • Acknowledge the warnings in the Capabilities section
    • then scroll down and click Create stack
Warning

If you do not select a the IAM role and continue with stack creation, this will fail! If this occurred, simply create another stack with a different name and follow the steps closely for this section.

  • 0.3: The CloudFormation stack will take ~10 minutes to finish deploying. Once the main/root CloudFormation stack shows as Create_Complete, proceed with the steps below.
1) Check the Transit Gateway Route Tables and confirm east/west is not working
Detailed Steps…
  • 1.1: In the VPC Console go to the Transit gateway route tables page (menu on the left) and check out all the associations, propagations, and routes for each Transit gateway route table. You can quickly compare the attachment IDs to the assigned names by opening another tab for the Transit gateway attachments page. You should see the following:
Transit Gateway Route Table NameAssociations
TGW-Connect-security-tgw-rtbTGW-Connect-security-vpc-attachment
TGW-Connect-security-connect-attachment
Unnamed-vpn-attachment
TGW-Connect-spoke-tgw-rtbVPC-A-spoke-vpc-attachment
VPC-B-spoke-vpc-attachment
TGW-Connect-sharedservices-tgw-rtbVPC-C-spoke-vpc-attachment
Transit Gateway Route Table NamePropagations
TGW-Connect-security-tgw-rtbVPC-A-spoke-vpc-attachment
VPC-B-spoke-vpc-attachment
VPC-C-spoke-vpc-attachment
TGW-Connect-spoke-tgw-rtbVPC-C-spoke-vpc-attachment
TGW-Connect-security-vpc-attachment
TGW-Connect-security-connect-attachment
Unnamed-vpn-attachment
TGW-Connect-sharedservices-tgw-rtbVPC-A-spoke-vpc-attachment
VPC-B-spoke-vpc-attachment
Transit Gateway Route Table NameRoutes
TGW-Connect-security-tgw-rtb10.1.0.0/16
10.2.0.0/16
10.3.0.0/16
TGW-Connect-spoke-tgw-rtb10.0.0.0/16
10.3.0.0/16
TGW-Connect-sharedservices-tgw-rtb10.1.0.0/16
10.2.0.0/16
  • 1.2: Navigate to the EC2 Console and connect to Instance-A using the Serial Console directions
    • Password: FORTInet123!
  • 1.3: Run the following commands to test connectivity and make sure the results match expectations
    SRC / DSTVPC BInternet
    Instance Aping 10.2.2.10 FAIL !!!
    Instance Acurl ipinfo.io FAIL !!!
2) Review FortiGate1’s GRE + BGP config and advertise a summary route to Transit Gateway
Detailed Steps…
  • 2.1: Navigate to the CloudFormation Console and toggle View Nested to off.
  • 2.2: Select the main template and select the Outputs tab.
  • 2.3: Login to FortiGate1, using the outputs FGT1LoginURL, Username, and Password.
  • 2.4: Upon login in the upper right-hand corner click on the >_ icon to open a CLI session.
  • 2.5: Run the command show system gre and notice the interface and IPs that are used are private.
  • 2.6: Run the command show system interface tgw-conn-peer and notice the IPs assigned to the inside interfaces of the GRE tunnel.
  • 2.7: Run the command show router bgp and notice the BGP peers fall into the remote-ip CIDR in the output of the command above.
  • 2.8: Run the command get router info routing-table all and notice the routes received match the routes tab of TGW-Connect-security-tgw-rtb.
  • 2.9: Copy and paste the commands below to configure a 10.0.0.0/8 summary route to be advertised back to Transit gateway:
    config router bgp
    config aggregate-address
    edit 1
    set prefix 10.0.0.0 255.0.0.0
    set summary-only enable
    next
    end
    end
  • 2.10: Run the commands below to confirm that the 10.0.0.0/8 route is now being advertised to our BGP neighbors.
    get router info bgp summary
    get router info bgp neighbors 169.254.6.2 advertised-routes
    get router info bgp neighbors 169.254.6.3 advertised-routes
3) Test east/west connectivity from Instance-A to Instance-B and validate there is no internet connectivity
Detailed Steps…
  • 3.1: While still in the console session for Instance-A, run the following commands to test connectivity and make sure the results match expectations
  • SRC / DSTVPC BInternet
    Instance Aping 10.2.2.10 SUCCESS !!!ping 8.8.8.8 FAIL !!!
    Instance Acurl 10.2.2.10 SUCCESS !!!curl ipinfo.io FAIL !!!
4) Let’s dig deeper to understand how all of this works
Detailed Steps…
  • 4.1: High level diagram showing how the Connect attachment goes over a VPC attachment which allows FortiGate1 to have a GRE tunnel over a private path which has BGP peering configured as well. This provides an overlay tunnel where dynamic routes and data-plane traffic can be routed without adding additional routes to the VPC router.

Info

FortiGate1 is getting data-plane traffic over a GRE tunnel between it’s port2 private IP (10.0.3.x/24) and an IP out of Transit Gateway CIDR block (100.64.0.x/24). This GRE tunnel is going over the TGW-Connect-security-connect-attachment, so this is all over a private path. Also Transit Gateway can support jumbo frames up to 8500 bytes for traffic between VPCs, AWS Direct Connect, Transit Gateway Connect, and peering attachments. However, traffic over VPN connections can have an MTU of 1500 bytes. Find out more in AWS Documentation.

BGP peering can be either iBGP or eBGP but the IP addressing will always use the inside tunnel IPs from a specific selection of CIDRs from 169.254.0.0/16. To find out which ones can or can’t be used, please reference AWS Documentation.

Regardless which type of BGP is used, each connect peer is only required to create one GRE tunnel to peer to the redundant BGP peers on the Transit Gateway side. For more information, reference AWS Documentation.

5) Review FortiGate2’s VPN and BGP configurations
Detailed Steps…
  • 5.1: Navigate to the CloudFormation Console and toggle View Nested to off.
  • 5.2: Select the main template and select the Outputs tab.
  • 5.3: Login to FortiGate2, using the outputs FGT2LoginURL, Username, and Password.
  • 5.4: Upon login in the upper right-hand corner click on the >_ icon to open a CLI session.
  • 5.5: Run the command show vpn ipsec phase1-interface and notice there are two tunnels where the remote-gw values are different public IPs and the interfaces are port1.
  • 5.6: Run the command show router route-map rmap-aspath1 and notice the as-path is set to 65000.
  • 5.7: Copy and paste the commands below to configure default-route-originate with the route-map to advertise 0.0.0.0/0 with an as-path of 65000:
    config router bgp
    config neighbor
    edit 169.254.10.1
    set capability-default-originate enable
    set default-originate-routemap rmap-aspath1
    next
    edit 169.254.11.1
    set capability-default-originate enable
    set default-originate-routemap rmap-aspath1
    next
    end
    end
  • 5.8: Run the commands below to confirm that the 0.0.0.0/0 route is now being advertised to our BGP neighbors with an as-path of 65000.
    get router info bgp summary
    get router info bgp neighbors 169.254.10.1 advertised-routes
    get router info bgp neighbors 169.254.11.1 advertised-routes
6) Test secured egress connectivity from Instance-A through FortiGate2
Detailed Steps…
  • 6.1: While still in the console session for Instance-A, run the following commands to test connectivity and make sure the results match expectations
    SRC / DSTVPC BInternet
    Instance Aping 10.2.2.10 SUCCESS !!! via FGT1ping 8.8.8.8 SUCCESS !!! via FGT2
    Instance Acurl 10.2.2.10 SUCCESS !!! via FGT1curl ipinfo.io SUCCESS !!! via FGT2
    • Ping and curl to public resources should connect successfully through FGT2 which is connected to Transit Gateway over a VPC attachment (IPsec + BGP over the internet).
    • Notice that the public IP returned from curl ipinfo.io should match the public IP of FGT2
  • 6.2: In the VPC Console go to the Transit gateway route tables page.
  • 6.3: Find TGW-Connect-spoke-tgw-rtb go to the Routes tab and notice there are ECMP routes for 0.0.0.0/0 since there are two VPN tunnels configured and a single route for 10.0.0.0/8 for the TGW connect peer.
Info

FortiGate2 is getting data-plane traffic over two IPsec tunnels between its port1 private ip (10.0.2.x/24) that has an associated Elastic IP and two public IPs managed by AWS, so this is all over a public path but encrypted. This means that jumbo frames are not supported for VPN based attachments.

Since there are two tunnels with a BGP peer configured for each, FortiGate2 is able to advertise ECMP paths which Transit Gateway can use. TGW supports ECMP for different attachment types.

7) Configure FortiGate1 with default-route-originate and a route-map as well
Detailed Steps…
  • 7.1: Login to FortiGate1, using the outputs FGT1LoginURL, Username, and Password.
  • 7.2: Upon login in the upper-right-hand corner click on the >_ icon to open a CLI session.
  • 7.3: Run the command show router route-map rmap-aspath1 and notice the as-path is set to 65000 just like FortiGate2.
  • 7.4: Copy and paste the commands below to configure default-route-originate with the route-map to advertise 0.0.0.0/0 with an as-path of 65000:
    config router bgp
    config neighbor
    edit 169.254.6.2
    set capability-default-originate enable
    set default-originate-routemap rmap-aspath1
    next
    edit 169.254.6.3
    set capability-default-originate enable
    set default-originate-routemap rmap-aspath1
    next
    end
    end
  • 7.5: Run the commands below to confirm that the 0.0.0.0/0 route is now being advertised to our BGP neighbors with an as-path of 65000.
    get router info bgp summary
    get router info bgp neighbors 169.254.6.2 advertised-routes
    get router info bgp neighbors 169.254.6.3 advertised-routes
8) Test secured egress connectivity from Instance-A through FortiGate1
Detailed Steps…
  • 8.1: While still in the console session for Instance-A, run the following commands to test connectivity and make sure the results match expectations
    SRC / DSTVPC BInternet
    Instance Aping 10.2.2.10 SUCCESS !!! via FGT1ping 8.8.8.8 SUCCESS !!! via FGT1
    Instance Acurl 10.2.2.10 SUCCESS !!! via FGT1curl ipinfo.io SUCCESS !!! via FGT1
    • Ping and curl should connect successfully through FGT1 which is connected to Transit Gateway over a Connect attachment (GRE + BGP over a VPC attachment).
    • Notice that the public IP returned from curl ipinfo.io should match the public IP of FGT1
  • 8.2: In the VPC Console go to the Transit gateway route tables page.
  • 8.3: Find TGW-Connect-spoke-tgw-rtb go to the Routes tab and notice there is only one route for 0.0.0.0/0 since Transit Gateway only supports ECMP routing from the same attachment types.
Info

While Transit Gateway does support ECMP routing, it only does so for the same attachment types. There is a route evaluation order that takes place. The reason behind this is that different attachment types go over different paths (ie connect over private vpc-attachment vs vpc over public internet) which can have different latency, RTTs, etc. and even different MTUs supported.

9) Lab environment teardown
Detailed Steps…
  • 9.1: Navigate to the CloudFormation Console, select the main stack you created and click Delete.
  • 9.2: The CloudFormation stack will take ~10 minutes to clean up. Once the stack is deleted, proceed to the next task.

Discussion Points

  • TGW Connect & VPN attachments allow a simple means to connecting remote resources to TGW
  • These attachment types are also helpful when dynamic routing is needed for a design
    • Connect uses GRE + BGP to privately connect to an appliance reachable via Direct Connect or within a VPC
    • VPN uses IPsec + BGP to publicly connect to an appliance reachable over the Internet
  • TGW has a route evaluation priority to select the best path when multiple routes have the same CIDR
  • TGW supports ECMP routing with routes from the same attachment type
    • TGW is a stateless router which will result in asymmetric routing of traffic
    • SNAT is required for flow symmetry to the correct FortiGate in Active-Active design
    • Simple & scalable Active-Active for Ingress/Egress inspection
    • Active-Active for East/West inspection possible with caveats
  • Each TGW VPN connection (2x IPsec tunnels per connection) supports up to 2.5 Gbps
  • Each TGW Connect peer supports up to 5 Gbps
  • TGW supports multiple peers per TGW Connect attachment and multiple attachments to a single VPC
  • TGW supports multiple VPN attachments to the same or different customer gateway (remote IPsec appliance)
  • Jumbo frames (8500 bytes) are supported for all attachments except VPN (1500 bytes)

This concludes this task

Gateway Load Balancer

GoalUtilize the provisioned Gateway Load Balancer architecture to provide security for distributed ingress and centralized egress flows with FortiGate NGFW.
TaskCreate VPC routes and FortiGate Policy objects allowing both flows of traffic.
ValidationConfirm connectivity to Public NLB1 and from Instance-B.

Introduction

In this scenario, there are multiple VPCs in the same region that have one instance each. Transit Gateway is configured with multiple Transit Gateway Route Tables and Gateway Load Balancer and endpoints are already configured as well. You will need to create the appropriate VPC routes to redirect traffic to Gateway Load Balancer via the deployed endpoints so the Active-Active FortiGates can inspect the traffic.

In this scenario these FortiGate NGFWs are working together in an Active-Active design to provide more capacity for bump in the wire inspection. This design is usable with workload VPCs that have a direct path to/from the Internet via an attached Internet Gateway (IGW), with or without a NAT Gateway (commonly referred to a distributed design). This design can also work in conjunction with Transit Gateway to offer centralized egress, ingress, and east/west inspection (commonly referred to a centralized design).

Summarized Steps (click to expand each for details)

0) Lab environment setup
Detailed Steps…
  • 0.1: Login to your AWS account, and click the Launch Stack button below to launch the CloudFormation stack for Task 4

  • 0.2: When creating this stack, ensure the following options are configured (See screenshots below for additional guidance):
    • An IAM role in the Permissions section
    • Acknowledge the warnings in the Capabilities section
    • then scroll down and click Create stack
Warning

If you do not select a the IAM role and continue with stack creation, this will fail! If this occurred, simply create another stack with a different name and follow the steps closely for this section.

  • 0.3: The CloudFormation stack will take ~12 minutes to finish deploying. Once the main/root CloudFormation stack shows as Create_Complete, proceed with the steps below.
1) Create VPC routes for ingress routing
Detailed Steps…
  • 1.1: In the VPC Console go to the Endpoints page (menu on the left) and look at the VPC Endpoints for VPC A. You should see four endpoints, one endpoint for each Availability Zone in both VPC-A and the NGFW VPC. Note the Endpoint IDs as these will be used in the next step.

  • 1.2: Go to the Route tables page (menu on the left) and find VPC-A-IgwRouteTable, select the Routes tab and click Edit routes. Create routes for the two public subnets to go to the VPC endpoint in the same Availability Zone.

Route TableCIDRGWLB/VPC Endpoint
VPC-A-IgwRouteTable10.1.1.0/24VPC-A-GWLB-VPCE-AZ1
VPC-A-IgwRouteTable10.1.10.0/24VPC-A-GWLB-VPCE-AZ2
  • 1.3: On the Route tables page (menu on the left) and find both VPC-A-Public1RouteTable and VPC-A-Public2RouteTable, select the Routes tab and click Edit routes. Create a default route to go to the VPC endpoint in the same Availability Zone.
Route TableCIDRGWLB/VPC Endpoint
VPC-A-Public1RouteTable0.0.0.0/0VPC-A-GWLB-VPCE-AZ1
VPC-A-Public2RouteTable0.0.0.0/0VPC-A-GWLB-VPCE-AZ2
2) Create an ingress FW policy using a dynamic address object for Public NLB1 in VPC-A on both FortiGates
Detailed Steps…
  • 2.1: Navigate to the CloudFormation Console and toggle View Nested to off.
  • 2.2: Select the main template and select the Outputs tab.
  • 2.3: Login to FortiGate1, using the outputs FGT1LoginURL, Username, and Password.
  • 2.4: In the FortiGate GUI, navigate to Policy & Objects > Addresses, and click Create new.
  • 2.5: Create a dynamic address object with the settings shown below, including searching for the NLB with PublicNLB1 to find the full filter, then click OK.
Tip

Dynamic address objects allow creating address objects based on resource metadata such as VPC ID, Auto Scale Group, EKS Cluster or Pod, and even Tag Name + Value pairs applied to the resource. FortiOS is using AWS API calls behind the scenes such as ec2:DescribeInstances, ec2:DescribeNetworkInterfaces, eks:ListClusters, eks:DescribeCluster, etc. to find running resources to match based on metadata (AWS, Kubernetes, etc) and pull their IP address information. This is done on a frequent basis to keep the dynamic address object up to date automatically. To learn more about all the public and private clouds this feature supports, check out our documentation.

  • 2.6: Navigate to Policy & Objects > Firewall Policy and click Create new.
  • 2.7: Create a new policy with the settings shown below and click OK to allow ingress traffic to the Public NLB in VPC-A.

  • 2.8: Validate that the dynamic address objects are automatically populated by hovering over both objects. In the popup menu, click View Matched Addresses button.

  • 2.9: Repeat the above steps on FortiGate2 as this is an Active-Active design.
Info

You can use FortiManager to manage a single policy set (FW policies, address & security objects, etc) and apply this across multiple FortiGates in the same or different deployments. Alternatively, if you are using FortiGate Auto Scale, then there is configuration sync that occurs from the config-primary to all config-secondaries in the Auto Scale Group.

3) Test distributed ingress to Public NLB1 (and Instance-A) through GWLB and the FortiGates
Detailed Steps…
  • 3.1: Navigate to the CloudFormation Console and toggle View Nested to off.

  • 3.2: Select the main template and select the Outputs tab.

  • 3.3: Copy the URL from Application1URL and navigate to this in your browser.

  • 3.4: You should see the web page with Instance-A and your connection details.

4) Let’s dig deeper to understand how all of this works
Detailed Steps…
  • 4.1 Here is an example of how GWLB routing works at a low level, including what information is tracked, and included in the GENEVE tunnel headers.

  • 4.2 To see this in your own environment, you can use the steps below. You will need Wireshark installed or another tool to handle .pcap files to do this.

  • 4.3 In the FortiGate GUI (either one), navigate to Network > Diagnostics > Packet Capture.

  • 4.4 Create a new packet capture with the settings shown below and click Start capture to see GENEVE traffic. Keep in mind that this is an active-active setup, so sessions are being sent to both FortiGates. You may need to generate multiple sessions before traffic is captured on your current FortiGate.

  • 4.5 After enough packets have been seen, click Stop capture, then click Save as .pcap.

  • 4.6 Open the .pcap file and drill into the GENEVE header like shown at the beginning of this section to find the GWLBVPC Endpoint and match this to the endpoints seen in your VPC console. Keep in mind that leading zeros of the endpoint ID are not listed in Wireshark.

Info

Gateway Load Balancer provides a very scalable Active-Active design for bump in the wire inspection with FortiGates applying NGFW, SSL MitM, etc. This is a regional setup that can support VPCs regardless if there is centralized networking (ie Transit Gateway used) or more of a distributed setup (each VPC is an island).

Gateway Load Balancer receives all traffic through Gateway Load Balancer Endpoints (ie GWLBE also called VPC Endpoints or VPCE for short). GWLB does not have a route table to track source CIDRs but rather relies on the GWLBE/VPCE ID and will return inspected traffic directly back to the endpoint for VPC routing to then take over.

Gateway Load Balancer is both a gateway in that it can be (more specifically the GWLBE/VPCE) a route target, and it is a load balancer in that it is flow aware (tracks 3/5 tuple flows) and keeps flows sticky to each FortiGate.

The GWLBE/VPCE ID, a flow cookie, and other information are stored and passed to the FortiGates via the GENEVE tunnel headers. Geneve is similar to VXLAN with optional TLVs (type length value options in the headers) to provide more context on the encapsulated data-plane traffic. Reference AWS Documentation to find out more.

On a side note, since GWLB does not track source CIDRs for routing, this means that both GWLB and the FortiGates can support environments where there are overlapping CIDR blocks. You can inspect this traffic in the same regional deployment, but this means you would be applying the same FW policies to the traffic. It is best practice to either use separate deployments or to use VDOMs and map your traffic from each GWLBE/VPCE to different VDOMs so that you can apply unique FW policies.

  • 4.7 Below is a step by step of the packet handling for the ingress web traffic to Public NLB1 in VPC-A.
HopComponentDescriptionPacket
1Internet -> 10.1.x.x/24 GWLBE(VPCE)An inbound connection starts with DNS resolution of the Public NLB to a public IP for one of the Availability Zones. The first packet (TCP sync) will be seen at the IGW attached to the VPC where the NLB is deployed. The IGW will perform destination NAT to the private IP of the public NLB and forward this to the GWLBE/VPCE as configured in VPC-A-IgwRouteTable.x.x.x.x:src-port -> 10.1.x.x:80
2GWLBE(VPCE) -> GWLBGWLBE/VPCE will route the traffic to the associated GWLB network interface in the same Availability Zone. This is done behind the scenes using AWS Private Link.x.x.x.x:src-port -> 10.1.x.x:80
3GWLB -> FGTs-Port1GWLB receives the traffic, encapsulates this in a GENEVE tunnel and forwards this flow to one of the FGTs. Post inspection the FGTs return the traffic to GWLB, which then hairpins the traffic back to the original GWLBE/VPCEx.x.x.x:src-port -> 10.1.x.x:80
4GLBE(VPCE) -> 10.1.0.0/16 LocalThe GWLBe endpoint will then route the inspected traffic to the intrinsic router. The intrinsic router will route traffic directly to the NLB’s ENI as specified in the VPC route table assigned to the subnet.x.x.x.x:src-port -> 10.1.x.x:80
5NLB -> Instance-AThe NLB will send traffic to a healthy target, in either AZ since cross zone load balancing is enabled. Instance-A is the only target so it receives the trafficx.x.x.x:src-port -> 10.1.2.10:80
6Instance-A -> NLBInstance-A will receive the traffic and respond. The return traffic will follow these steps in reverse.10.1.2.10:80 -> x.x.x.x:src-port

5) Test centralized egress from Instance-B through GWLB & the FortiGates
Detailed Steps…
  • 5.1: Navigate to the EC2 Console and connect to Instance-B using the Serial Console directions
    • Password: FORTInet123!
  • 5.2: Run the following commands to test connectivity and make sure the results match expectations
    SRC / DSTInternet
    Instance Bping 8.8.8.8 SUCCESS !!!
    Instance Bcurl ipinfo.io SUCCESS !!!
6) Let’s dig deeper to understand how all of this works
Detailed Steps…
  • 6.1: Note that the output of curl ipinfo.io returned the public IP of one of the FortiGates connected to GWLB. If you continue to run the command multiple times, you will see that your outbound traffic is flowing through both FortiGates and being Source NAT’d to look like traffic is coming from their Elastic IPs.
Info

One-Arm Model

GWLB supports two different models of firewall deployments, one-arm and two-arm where a firewall appliance can also perform NAT.

In the one-arm model, the FortiGates will inspect traffic and forward this back to GWLB where Internet bound traffic has NAT applied by a NAT GW. Typically, the NAT GW will be in a workload VPC in a distributed design. Distributed designs have GWLB endpoints in each workload VPC requiring an attached Internet Gateway (IGW) and public load balancer or NAT GW. A centralized design can use NAT GW in an inspection VPC for centralized egress and have GWLB endpoints only deployed in the inspection VPC (no need for GWLB endpoints in each workload VPC).

We can use static and policy routes like below to support this setup. In a 2 AZ deployment there are two static routes using priority setting to bypass the reverse path filtering check when receiving data plane traffic over the GENEVE tunnels. The static routes are default routes to simplify the config, but you could also specify a route for each spoke VPC for each GENEVE tunnel. Also, there are two policy routes to hairpin traffic received over each GENEVE tunnel, back to the same tunnel.

config router static
edit 1
set distance 5
set priority 100
set device gwlb1-az1
next
edit 2
set distance 5
set priority 100
set device gwlb1-az2
next
end
config router policy
edit 1
set input-device gwlb1-az1
set output-device gwlb1-az1
next
edit 2
set input-device gwlb1-az2
set output-device gwlb1-az2
next

Two-Arm Model

In the two-arm model, the FortiGates will inspect traffic, forward, & SNAT traffic out port1 (public interface) to act as a NAT GW. This removes the need for deploying NAT GWs in each AZ of each workload VPC. This is typically used in a centralized design where the data plane traffic used TGW to reach the GWLB endpoints in the inspection/security VPC and be inspected by the FortiGates. In summary centralized vs distributed designs is in reference to the GWLB endpoint placement which impacts how traffic is routed to these for inspection of traffic for different directions (ingress, egress, and east/west).

We can use static and policy routes like below to support this setup. In a 2 AZ deployment there are two static routes using priority setting to bypass the reverse path filtering check when receiving data plane traffic over the GENEVE tunnels. The static routes are default routes to simplify the config, but you could also specify a route for each spoke VPC for each GENEVE tunnel. Also, there are two policy routes to hairpin traffic received over each GENEVE tunnel, back to the same one. These policy routes will only hairpin traffic destined RFC1918 addresses and will route internet bound traffic out port1.

config router static
edit 1
set distance 5
set priority 100
set device gwlb1-az1
next
edit 2
set distance 5
set priority 100
set device gwlb1-az2
next
end
config router policy
edit 1
set input-device gwlb1-az1
set dst "10.0.0.0/255.0.0.0" "172.16.0.0/255.240.0.0" "192.168.0.0/255.255.0.0"
set output-device gwlb1-az1
next
edit 2
set input-device gwlb1-az2
set dst "10.0.0.0/255.0.0.0" "172.16.0.0/255.240.0.0" "192.168.0.0/255.255.0.0"
set output-device gwlb1-az2
next

Supporting Both Models

In a single region, you can have one deployment of FGTs & GWLB support both distributed and centralized designs. This all comes down to implementing the appropriate routing at the VPC & TGW route tables and FortiGates. For examples on the VPC & TGW routes for different designs, reference common architecture patterns.

Here is an example of the static & policy routes to support a distributed spoke1 VPC (CIDR 10.1.0.0/16) and centralized spoke2 VPC.

config router static
edit 1
set distance 5
set priority 100
set device gwlb1-az1
next
edit 2
set distance 5
set priority 100
set device gwlb1-az2
next
end
config router policy
edit 1
set input-device gwlb1-az1
set src "10.1.0.0/16"
set output-device gwlb1-az1
next
edit 2
set input-device gwlb1-az2
set src "10.1.0.0/16"
set output-device gwlb1-az2
next
edit 3
set input-device gwlb1-az1
set dst "10.0.0.0/255.0.0.0" "172.16.0.0/255.240.0.0" "192.168.0.0/255.255.0.0"
set output-device gwlb1-az1
next
edit 4
set input-device gwlb1-az2
set dst "10.0.0.0/255.0.0.0" "172.16.0.0/255.240.0.0" "192.168.0.0/255.255.0.0"
set output-device gwlb1-az2
next
  • 6.2 Below is a step by step of the packet handling for the centralized egress from Instance-B.
HopComponentDescriptionPacket
1Instance-B -> 0.0.0.0/0 TGWInstance-B sends egress traffic to the VPC router (its default gw) which routes traffic to TGW as configured in the VPC-B-Private1RouteTable.10.2.2.10:src-port -> y.y.y.y:80
2VPC-B-TGW-Attachment -> 0.0.0.0/0 NGFW-security-vpc-attachmentVPC-B-TGW-Attachment is associated to the Spoke VPC TGW RTB. This TGW RTB has a default route to NGFW-security-vpc-attachment, so traffic is forwarded there.10.2.2.10:src-port -> y.y.y.y:80
3NGFW-security-vpc-attachment -> 0.0.0.0/0 VPCE-…NGFW-security-vpc-attachment is attached to the NGFW VPC TGW Attach Subnets which have a default route to the GWLBE/VPCE in the same Availability Zone.10.2.2.10:src-port -> y.y.y.y:80
4GWLBE/VPCE -> GWLBGWLBE/VPCE will route traffic to the associated GWLB network interface in the same Availability Zone. This is done behind the scenes using AWS Private Link10.2.2.10:src-port -> y.y.y.y:80
5GWLB -> FGTs-Port1GWLB receives the traffic, encapsulates this in a GENEVE tunnel and forwards this flow to one of the FGTs.10.2.2.10:src-port -> y.y.y.y:80
6FGTs-Port1 -> 0.0.0.0/0 IGWFGTs changes the source IP to the private IP of Port1 as this has an EIP associated. FGTs sends inspected & allowed traffic to the VPC router via Port1 (its default gw), which routes traffic to the IGW as configured in the NGFW Public Subnet VPC Route Table.10.0.x.x:src-port -> y.y.y.y:80
7IGW -> InternetIGW changes the source IP to the associated EIP of FortiGates Port1 and routes traffic to the internet.z.z.z.z:src-port -> y.y.y.y:80
8Internet -> IGWIGW receives reply traffic and changes the source IP to the private IP of FortiGates Port1. The VPC router routes traffic to FortiGates Port1. The return traffic will follow these steps in reverse.y.y.y.y:80 -> 10.0.x.x:src-port

7. Lab environment teardown
Detailed Steps…
  • 7.1: Before deleting the main CloudFormation Stack, we must remove the VPC routes referencing the GWLBE/VPCEs.
  • 7.2: Navigate to the VPC Console and go to the Route tables page (menu on the left), and delete the following routes from the route tables shown below:
Route TableCIDRVPC Endpoint
VPC-A-IgwRouteTable10.1.1.0/24VPC-A-GWLB-VPCE-AZ1
VPC-A-IgwRouteTable10.1.10.0/24VPC-A-GWLB-VPCE-AZ2
VPC-A-Public1RouteTable0.0.0.0/0VPC-A-GWLB-VPCE-AZ1
VPC-A-Public2RouteTable0.0.0.0/0VPC-A-GWLB-VPCE-AZ2
  • 7.3: Navigate to the CloudFormation Console, select the main stack you created and click Delete.
  • 7.4: The CloudFormation stack will take ~12 minutes to clean up. Once the stack is deleted, proceed to the next task.

Discussion Points

  • GWLB is a regional service that is both a gateway (VPC route target) and a flow aware load balancer
    • This allows very scalable active-active inspection for all directions of traffic
    • No SNAT requirement to achieve flow symmetry to the correct FortiGate as GWLB is flow aware
  • GWLB and FortiGates supports one and two arm mode (distributed vs centralized egress access & NAT GW replacement)
  • Jumbo frames (8500 bytes) are supported
  • Inspection VPC handles FortiGate NGFW inspection for any traffic flow (Inbound, Outbound, East/West) and for any network design (distributed vs centralized)
    • Appliance Mode is required for this design to keep flows sticky to the correct availability zone which in turn means the correct GWLB endpoint.
    • Advanced architectures for all of these scenarios can be found here

This concludes this task

Cloud WAN

GoalUtilize Cloud WAN components and Core Network Policy to provide a secured & orchestrated network.
TaskUpdate Core Networking Policy with logic to automate connecting resources to segments and propagating routes to allow secured traffic flow.
ValidationConfirm east/west connectivity from EC2 Instance-A via Ping, HTTP.

Introduction

In this design, there are multiple VPCs in the same region that have one instance each. Cloud WAN is configured with multiple segments and attachments (both VPC and Tunnel-less Connect). You will need to create the appropriate Cloud WAN Core Network Policy to automatically enforce segment attachment rules and propagation of routes between segments to direct traffic to the independent FortiGates for traffic inspection.

In this setup these FortiGate are independent (not clustered, nor sharing config) but are working together with FGSP in an Active-Active design to provide more capacity and synchronize session tables. This design specifically uses Tunnel-less Connect attachments to allow dynamic routing between EC2 instances and a Cloud WAN Core Network Edge (CNE) without needing IPsec or GRE based overlay tunnels. This removes the overhead and bottlenecks that come with overlay tunnel protocols while still providing dynamic routing.

Summarized Steps (click to expand each for details)

0) Lab environment setup
Detailed Steps…
  • 0.1: Login to your AWS account, and click the Launch Stack button below to launch the CloudFormation stack for Task 4

  • 0.2: When creating this stack, ensure the following options are configured (See screenshots below for additional guidance):
    • An IAM role in the Permissions section
    • Acknowledge the warnings in the Capabilities section
    • then scroll down and click Create stack
Warning

If you do not select a the IAM role and continue with stack creation, this will fail! If this occurred, simply create another stack with a different name and follow the steps closely for this section.

  • 0.3: The CloudFormation stack will take ~20 minutes to finish deploying. Once the main/root CloudFormation stack shows as Create_Complete, proceed with the steps below.
1) Inspect the Cloud WAN attachments and segments
Detailed Steps…
  • 1.1: In the Network Manager Console go to the Global Networks page (menu on the left), then find and select your Global Network ID.
  • 1.2: Once the CWAN Global Network has loaded, go to the Core network page (menu on the left), then notice that there are three segments and one edge location.
  • 1.3: Scroll down the page and notice the attachments widget shows a total of 5 attachments (4 VPC & 1 Connect).
  • 1.4: Navigate to the attachments page under your core network. Select the CWAN-security-connect-attachment and view the Details tab in the pane below, notice the segment and attachment policy rule number are empty.
  • 1.5: On the same pane, switch to the Connect peers tab and notice the Peer and Core network BGP 1/2 addresses. The Peer addresses are the FGT private IPs and Core network are the BGP router endpoints for the Core Network Edge (CNE, ie managed TGW). The BGP Status may show as down due to a delay for the console to update the latest values.
  • 1.6: In the same pane, switch to the Tags tab and notice the segment key and the configured value. These should match the table below. Select the other attachments to check the tag key value as well.
    Attachment NameTag (Key, Value)
    CWAN-security-connect-attachmentsegment = firewall
    CWAN-security-vpc-attachmentsegment = firewall
    VPC-A-spoke-vpc-attachmentsegment = production
    VPC-B-spoke-vpc-attachmentsegment = production
    VPC-C-spoke-vpc-attachmentsegment = development
2) Review FortiGate1’s BGP config and current routes advertised/received
Detailed Steps…
  • 2.1: Navigate to the CloudFormation Console and toggle View Nested to off.
  • 2.2: Select the main template and select the Outputs tab.
  • 2.3: Login to FortiGate1, using the outputs FGT1LoginURL, Username, and Password.
  • 2.4: Upon login in the upper right-hand corner click on the >_ icon to open a CLI session.
  • 2.5: Run the command get sys int physical port2 and notice the interface and IP is private.
  • 2.7: Run the command show router bgp and notice the router-id and BGP peers match the Connect peers information from the previous section.
  • 2.8: Run the command get router info bgp summary and notice the State/PfxRcd values are zero.
  • 2.9: Run the command get router info bgp neighbors <peer-ip> advertised-routes for each BGP neighbor and notice a default route is advertised.
  • 2.10: Run the command get router info bgp neighbors <peer-ip> routes for each BGP neighbor and notice no routes are received from Cloud WAN.
3) Update & Apply Core Network Policy
Detailed Steps…
  • 3.1: In the Network Manager Console navigate to the Policy versions page for your Core Network and select the only policy version and click Edit.

  • 3.2: Select the Segments tab and notice the existing segments. You should see three segments (firewall, production, and development).

  • 3.3: Select the Segments actions tab then find the Sharing section and click Create. Use the table below to create the sharing rules need.

    Segment fromSegment toAllow segment list
    productionallow selectedfirewall
    developmentallow selectedfirewall
  • 3.4: Next, select the Attachment policies tab then find the Attachment policies section and click Create. Use the table below to create the sharing rules need. Here is an example of the first rule.

    rule numberAttach to SegmentConditions Values (Tag Key, Tag Value)
    100firewallType=Tag Value, Operator=Equals, Condition Values=segment, firewall
    200productionType=Tag Value, Operator=Equals, Condition Values=segment, production
    300developmentType=Tag Value, Operator=Equals, Condition Values=segment, development
  • 3.5: Once completed, you should see these attachment policies. Next, click Create policy.

  • 3.6: You should be back on the Policy versions page with a new policy version showing. Once Policy version - 2 shows Ready to execute, select the version and click View or apply change set.

  • 3.7: On the next page click Apply change set. You will be returned to the Policy version page and see the new policy version is executing. In a few moments this will show as Execution succeeded.

4) Test traffic and Validate Results
Detailed Steps…
  • 4.1: Navigate to the attachments page under your Core Network. Select the CWAN-security-connect-attachment and view the Details tab. Notice the segment and attachment policy rule number are now populated. The table below should match what your environment looks like after applying the correct Core Network Policy. Select the other attachments to verify the results.

    AttachmentSegmentRule
    CWAN-security-connect-attachmentfirewall100
    CWAN-security-vpc-attachmentfirewall100
    VPC-A-spoke-vpc-attachmentproduction200
    VPC-B-spoke-vpc-attachmentproduction200
    VPC-C-spoke-vpc-attachmentdevelopment300

  • 4.2: Navigate to the main Core network page for your Core Network. Select the Routes tab and in the route filter, select a segment and edge location and click Search routes. You should eventually see routes matching the table below. Notice, the default route is an ECMP route received from both FGTs through the connect attachment.

    SegmentCIDRs
    firewall10.0.0.0/16 (Sec-VPC), 10.1.0.0/16 (VPC-A), 10.2.0.0/16 (VPC-B), 10.3.0.0/16 (VPC-C), 0.0.0.0/0 (Sec-Connect)
    production10.0.0.0/16 (Sec-VPC), 10.1.0.0/16 (VPC-A), 10.2.0.0/16 (VPC-B), 0.0.0.0/0 (Sec-Connect)
    development10.0.0.0/16 (Sec-VPC), 10.3.0.0/16 (VPC-C), 0.0.0.0/0 (Sec-Connect)

  • 4.2: Login to FortiGate1, using the outputs FGT1LoginURL, Username, and Password.

  • 4.4: Upon login in the upper right-hand corner click on the >_ icon to open a CLI session.

  • 4.8: Run the command get router info bgp summary and notice State/PfxRcd is now showing four routes received.

  • 4.9: Run the command get router info bgp neighbors <peer-ip> routes for each BGP neighbor and notice four routes are received from Cloud WAN. Notice that the Next Hop address is the IP of the Core Network attachment CWAN-security-vpc-attachment in the same subnet as port2 of FortiGate1.

  • 4.10: Run the command get router info routing-table all and notice there is a static route for 100.64.0.0/24 & 10.0.0.0/16 out port2. This allows the FGTs to BGP directly with the Core Network Edge (CNE) and use FGSP between each FGT.

  • 4.11: Navigate to the EC2 Console and connect to Instance-A using the Serial Console directions

    • Password: FORTInet123!
  • 4.12: Run the following commands to test connectivity and make sure the results match expectations

    SRC / DSTVPC BVPC C
    Instance Acurl 10.2.2.10 SUCCESS !!!curl 10.3.2.10 FAIL !!!
    Instance Aping 10.2.2.10 SUCCESS !!!ping 10.3.2.10 SUCCESS !!!
5) Let’s dig deeper to understand how all of this works
Detailed Steps…
  • 5.1 Notice that Instance-A can access Instance-B over HTTP but could not access Instance-C, however pings were successful. This is because Instance-A and Instance-B are in VPCs attached to the production segment which is configured as a shared routing domain by default. This allows anything attached to the same segment to communicate bidirectionally. This means anything in VPC A can reach VPC B without being sent through the FGTs in the inspection VPC which is in the firewall segment.

  • 5.2 VPC C is in the development segment so when VPC A reaches out to this destination, the routes for the production segment first forwards traffic to the FGTs in the inspection VPC (via 0.0.0.0/0 to Connect attachments). This allowed the FGTs to enforce FW policy that blocked HTTP access from VPC A to VPC C but allowed pings between the different segments.

  • 5.3 Segments can be configured to be isolated so that resources attached to the same segment can’t communicate directly. Through the Core Network Policy you can still allow access to specific routes or other segments explicitly.

  • 5.4 In the Network Manager Console navigate to the Policy versions page select ‘Policy version - 2’ and click Edit.

  • 5.5 Select the Segments tab, select the production segment and click Edit.

  • 5.6 On the Edit segment page, check the box for Isolated attachments and click Edit Segment, then on the next page click Create Policy.

  • 5.7 You should be back on the Policy versions page with a new policy version showing. Once Policy version - 3 shows Ready to execute, select the version and click View or apply change set.

  • 5.8 On the next page click Apply change set. You will be returned to the Policy version page and see the new policy version is executing. In a few moments this will show as Execution succeeded.

  • 5.9: Navigate back to the EC2 Console and connect to Instance-A using the Serial Console directions

    • Password: FORTInet123!
  • 5.10: Run the following commands to test connectivity again and make sure the results match expectations

    SRC / DSTVPC BVPC C
    Instance Acurl 10.2.2.10 FAIL !!!curl 10.3.2.10 FAIL !!!
    Instance Aping 10.2.2.10 SUCCESS !!!ping 10.3.2.10 SUCCESS !!!
    • HTTP should now be block by the FW policy on the FGTs for VPC B and C but Pings allowed
  • 5.11 Navigate back to the main Core network page for your Core Network. Select the Routes tab and in the route filter, select the production segment and edge location and click Search routes. You should eventually see routes matching the table below. The production segment now does not automatically share routes for attachments.

    SegmentCIDRs
    firewall10.0.0.0/16 (Sec-VPC), 10.1.0.0/16 (VPC-A), 10.2.0.0/16 (VPC-B), 10.3.0.0/16 (VPC-C), 0.0.0.0/0 (Sec-Connect)
    production10.0.0.0/16 (Sec-VPC), 0.0.0.0/0 (Sec-Connect)
    development10.0.0.0/16 (Sec-VPC), 10.3.0.0/16 (VPC-C), 0.0.0.0/0 (Sec-Connect)
Info

At this point, all East/West and egress traffic is being sent through the FGTs in the Inspection VPC, even for VPC A to VPC B traffic which are in the same segment. The FGTs in this design are independent but use FGSP (FortiGate Session Life Support Protocol) to synchronize sessions between each other. In this design each FGT is actively handling traffic from Cloud WAN which means that at some point there will be asymmetric traffic flows since TGWs, CNEs, etc are stateless routers. Thus, FGSP is used to keep each FGT aware of each other’s sessions.

To view synchronized sessions, generate more PING traffic from Instance A and run the commands below on both FGTs. When a session entry is created on the current FGT the synced session state flag is set. When a session entry is received from another FGT the syn_ses session state flag is set. Notice this when running the commands below on each FGT.

diag sys session filter proto 1
diag sys session list | grep -c 'syn_ses'
diag sys session list | grep -c 'synced'
diag sys session list
get sys int physical port2
show system standalone-cluster
diag sys ha standalone-peers
diag sys session sync

While FGSP is great, there are caveats to keep in mind such as: inspecting asymmetric traffic with NGFW L7 features, increased packets per second (PPS) rates due to FGSP can trigger throttling from cloud providers, etc. To find out more about FGSP reference this documentation.

Appliance Mode is not required but recommended if FGSP is to be used as it limits the amount of asymmetric traffic that will be handled in an ECMP Active-Active design. Appliance mode will use a flow hash algorithm to send traffic, including reply traffic, for the life of the flow to the same availability zone and network interface of the attachment within the appliance or inspection VPC.

6. Lab environment teardown
Detailed Steps…
  • 6.1: Navigate to the CloudFormation Console, select the main stack you created and click Delete.
  • 6.2: The CloudFormation stack will take ~20 minutes to clean up. Once the stack is deleted, take a long break 😄, you deserve it!

Discussion Points

  • Cloud WAN (CWAN) is a global service
    • Network Manager Console, Global Network, and Core Network Policy are Global
    • Segments can be regional or global, based on CNE locations and attached resources
    • Core Network Edge (CNEs), and attachments (VPC, Connect, VPN, Direct Connect, etc) are regional
  • Segments are dedicated routing domains that can be isolated or allow direct communication between attached resources
  • Core Network Edges (CNEs) are essentially managed TGWs which are peered together with BGP
  • Core Network Policy allows granular automation of attachment association, propagation, and sharing of other routes between segments
  • CWAN supports ECMP routing with routes from the same attachment type
    • CWAN is a stateless router which will result in asymmetric routing of traffic
    • SNAT is required for flow symmetry to the correct FortiGate in Active-Active design
    • FGSP can be used instead for Active-Active East/West inspection with caveats
    • Appliance Mode is not required but recommended as it limits the amount of asymmetric traffic
  • Connect (tunnel-less) attachments use BGP directly to privately connect to an appliance within a VPC only
  • Each CWAN Connect (tunnel-less) peer supports up to 100 Gbps, (actual limit is based on instance type BW)
  • Jumbo frames (8500 bytes) are supported for all attachments except VPN (1500 bytes)

This concludes this task