Hybrid multicloud

One colocation site connected to AWS, Azure and Google Cloud through a Fabric Cloud Router.

Scenario#

A company runs its core systems in a cage in Ashburn (IBX DC6) and uses AWS us-east-1, Azure East US and Google Cloud us-east4. Each cloud is currently reached through its own VPN over the internet. The target design connects all three clouds privately under one routing policy and routes cloud-to-cloud traffic without passing through the cage.

Architecture#

A Fabric port in the cage connects to a Cloud Router in the same metro. Each cloud attaches to the router with an IP_VC connection and a BGP session. Cloud-to-cloud traffic is routed on the Cloud Router.

Element	Count	Notes
Fabric port	1, existing	A second port on another Fabric device is the first resilience step, see Variations
Cloud Router	1	`STANDARD` package
Connections	4	Port to router, router to each cloud
Routing protocols	8	DIRECT and BGP per connection

The alternative design uses three EVPL connections from the port to the clouds and no Cloud Router. It saves the router fee and keeps all routing on the cage routers. In that design cloud-to-cloud traffic crosses the cage, and each added cloud needs a VLAN and a BGP session on customer hardware.

Step 1: provider-side prerequisites#

The SDK creates the Equinix resources. Before that, an object has to be created at each provider with the provider's own tooling, and that object supplies the authentication key for the Fabric connection.

Provider	Object	Authentication key for Fabric
AWS	Nothing in advance	The 12-digit account ID. The hosted connection appears in the AWS account and is accepted there.
Azure	ExpressRoute circuit with provider "Equinix"	The circuit's service key
Google Cloud	Partner Interconnect VLAN attachment on a Cloud Router	The pairing key

Step 2: discover profiles and inputs#

Java

import com.eqixiac.equinix.fabric.model.Port;
import com.eqixiac.equinix.fabric.model.ServiceProfile;

Port port = fabric.ports().list().stream()
    .filter(p -> "DC".equals(p.getLocation().getMetroCode()))
    .findFirst().orElseThrow();

Map<String, String> profileByName = new HashMap<>();
for (ServiceProfile profile : fabric.serviceProfiles().search().loadAll()) {
    profileByName.put(profile.getName(), profile.getUuid());
}
profileByName.keySet().stream().filter(n -> n.contains("AWS") || n.contains("Azure") || n.contains("Google"))
    .forEach(System.out::println);

The providers set the service profile names and change them. Store the chosen UUIDs in production code and do not match on names.

Step 3: Cloud Router#

Java

import com.eqixiac.equinix.fabric.enums.GatewayPackageCode;
import com.eqixiac.equinix.fabric.enums.NotificationType;
import com.eqixiac.equinix.fabric.model.CloudRouter;

CloudRouter router = fabric.cloudRouters().define()
    .name("fcr-dc-hybrid")
    .inMetro("DC")
    .withPackage(GatewayPackageCode.STANDARD)
    .accountNumber(123456L)
    .projectId(projectId)
    .notification(NotificationType.ALL, List.of("noc@example.com"))
    .create();

The same builder with .dryRun() validates the request. The router is billed from the time it is provisioned.

Step 4: connections, dry run first#

Java

import com.eqixiac.equinix.fabric.enums.ConnectionType;
import com.eqixiac.equinix.fabric.enums.PeeringType;
import com.eqixiac.equinix.fabric.model.Connection;
import com.eqixiac.equinix.fabric.model.implementation.LinkProtocol;
import com.eqixiac.equinix.fabric.model.implementation.cloud.AwsDirectConnectAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.AzureExpressRouteAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.CloudProviderConnectionAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.GoogleCloudInterconnectAdapter;

CloudRouter fcr = fabric.cloudRouters().getByUuid(routerUuid);

// Cage to router
Connection toCage = fabric.connections().define(ConnectionType.IP_VC)
    .name("fcr-dc-to-cage")
    .bandwidth(2000)
    .aSideAccessPointCloudRouter(fcr)
    .zSideAccessPointPort(portUuid, LinkProtocol.dot1q().vlanTag(1100).create())
    .notification("noc@example.com")
    .create();

// Router to each cloud
Map<String, CloudProviderConnectionAdapter<?>> clouds = new LinkedHashMap<>();
clouds.put("aws", AwsDirectConnectAdapter.of("123456789012", "us-east-1", awsProfileUuid, "DC"));
clouds.put("azure", AzureExpressRouteAdapter.of("azure-service-key", azureProfileUuid, PeeringType.PRIVATE, "DC"));
clouds.put("gcp", GoogleCloudInterconnectAdapter.of("gcp-pairing-key", "us-east4", gcpProfileUuid, "DC"));

Map<String, Connection> created = new LinkedHashMap<>();
for (var entry : clouds.entrySet()) {
    var builder = fabric.connections().define(ConnectionType.IP_VC)
        .name("fcr-dc-to-" + entry.getKey())
        .bandwidth("gcp".equals(entry.getKey()) ? 500 : 1000)
        .aSideAccessPointCloudRouter(fcr)
        .zSideCloudProvider(entry.getValue())
        .notification("noc@example.com");
    builder.dryRun().create();                       // throws if Fabric rejects the payload
}

The loop validates the three cloud connections and creates none of them. The same request without dryRun() creates a connection and returns the Connection object that the later steps need. Validating all three before the first create reduces the chance that the build stops with only some of the connections in place.

Step 5: routing#

Each connection needs a DIRECT protocol for the /30 and a BGP protocol. The cloud side of each session is configured at the provider with the matching addresses.

Java

record Peering(String connectionUuid, String subnetBase, long peerAsn) {}

Java

import com.eqixiac.equinix.fabric.enums.RoutingProtocolType;

List<Peering> peerings = List.of(
    new Peering("uuid-of-fcr-dc-to-cage",  "10.100.0.", 65100L),
    new Peering("uuid-of-fcr-dc-to-aws",   "10.100.1.", 64512L),
    new Peering("uuid-of-fcr-dc-to-azure", "10.100.2.", 12076L),
    new Peering("uuid-of-fcr-dc-to-gcp",   "10.100.3.", 16550L));

for (Peering p : peerings) {
    fabric.routingProtocols().define()
        .ofType(RoutingProtocolType.DIRECT)
        .withDirectIpv4(p.subnetBase() + "1/30")
        .create(p.connectionUuid());

    fabric.routingProtocols().define()
        .ofType(RoutingProtocolType.BGP)
        .withCustomerAsn(p.peerAsn())
        .withBGPIpv4(p.subnetBase() + "2", p.subnetBase() + "1", true)
        .withBFD(true, 300)
        .create(p.connectionUuid());
}

The ASNs are the providers' well-known defaults for these services and your own for the cage. The provider objects in your account can have other values. Azure and Google Cloud assign parts of the link addressing on their side, so the addresses of those two sessions have to be read from the provider object and used in place of the scheme above.

Step 6: verify#

Java

import com.eqixiac.equinix.fabric.model.RouteTableEntry;

for (RouteTableEntry route : fabric.cloudRouters().searchRoutes(routerUuid).loadAll()) {
    System.out.println(route.getPrefix() + " via " + route.getNextHop() + " " + route.getProtocolType() + " " + route.getState());
}
fabric.connections().searchReceivedRoutes("uuid-of-fcr-dc-to-aws")
    .forEach(r -> System.out.println("from AWS: " + r.getPrefix()));

The route table should contain the VPC, VNet and VPC network prefixes as BGP routes from the three clouds, and the cage prefixes from AS 65100. When a cloud sends no routes, the usual cause is an unfinished step at the provider: the hosted connection is not accepted, the circuit has no peering, or the attachment is not activated.

Step 7: constrain what is exchanged#

By default the router advertises the routes it learns to every neighbour, so the clouds can reach each other and the cage. Route filters attached to a connection restrict the prefixes exchanged on it.

Deployment Wizard alternative#

The Deployment Wizard plans steps 3 to 5 from an optimization result, validates the plan in layers, and either executes it or exports Terraform. The wizard suits a design whose metros are not yet decided or that spans several metros. For one known metro the direct calls on this page need less code.

Variations#

Requirement	Change	Cost
Survive a port or device failure	Second port on a different Fabric device, second cage connection in the same redundancy group	One more port and connection
Survive a metro failure	Second Cloud Router in another metro with its own cloud connections, backbone connection between the routers	Roughly doubles the design, see Disaster recovery
Stateful inspection between clouds	Network Edge firewall chained to the router	See Network functions
Very high sustained cloud-to-cloud volume	Compare with a native link	See Cloud-to-cloud

Cost check#

Before the build, run the savings calculator once per cloud with the real monthly egress, and the TCO comparison with the router included. The private path costs less than the internet path only above the break-even egress volume that the estimate reports.