AI and ML workloads

Scenario#

A team trains models on GPU capacity in a colocation cage in Ashburn and at a GPU cloud provider reachable on Fabric. The training data lives in AWS S3 and Google Cloud Storage. Trained models serve inference from two regions, with a latency budget of 50 ms round trip from the main user populations.

Training and inference are sized separately.

Architecture#

Step 1: size the training path#

The transfer time for a dataset is its size divided by the usable rate.

Each figure is size x 8 / rate, at 100% utilization: 300 TB is 2,400,000 Gb, and 2,400,000 Gb / 10 Gbps is 240,000 s, or 2.8 days. A real transfer reaches less than the nominal rate, because object-store request limits and the number of parallel streams bound the throughput. A measured sample transfer gives the rate to plan with.

Where the provider side allows it, a connection can be enlarged for the run and reduced afterwards.

import com.eqixiac.equinix.fabric.model.Connection;

Connection toS3 = fabric.connections().getByUuid(connectionUuid);
toS3.update().bandwidth(10_000).dryRun().save();    // validate the resize
toS3.update().bandwidth(10_000).save();             // before the training run
// ... run ...
toS3.update().bandwidth(1_000).save();              // afterwards

The capacity of an AWS hosted connection is fixed, and a different capacity requires a new hosted connection. A dedicated Direct Connect port with a transit virtual interface avoids that limit at a higher fixed cost.

Step 2: check the egress economics#

AWS bills data that leaves S3 per GB. The private path has a lower rate per GB and adds a fixed monthly cost.

import com.eqixiac.equinix.core.enums.MetroCode;
import com.eqixiac.equinix.design.value.savings.DataUnit;
import com.eqixiac.equinix.design.value.savings.SavingsEstimate;
import com.eqixiac.equinix.fabric.model.implementation.cloud.CloudProviderType;

SavingsEstimate training = fabric.savingsCalculator()
    .egress(300, DataUnit.TERABYTE)                 // per month, in a month with one full run
    .fromCloud(CloudProviderType.AWS).inRegion("us-east-1")
    .viaMetro(MetroCode.DC)
    .bandwidthMbps(10_000)
    .calculate();
System.out.println(training.toMarkdown());

At the bundled reference rates the egress difference is 300,000 GB x (0.09 - 0.02) = USD 21,000 for that month, before the fixed costs of the connection and the provider port. The fixed costs also apply in a month without a run. The break-even volume in the estimate is the monthly data volume above which keeping the connection costs less than paying internet egress.

Step 3: place inference from the latency budget#

import com.eqixiac.equinix.design.optimizer.enums.OptimizationStrategy;
import com.eqixiac.equinix.design.optimizer.enums.RedundancyTier;
import com.eqixiac.equinix.design.optimizer.enums.WorkloadType;
import com.eqixiac.equinix.design.optimizer.model.OptimizationResult;

OptimizationResult placement = fabric.optimizeMetros()
    .name("inference-serving")
    .addSite("US East users").nearestMetro(MetroCode.NY).weight(5).done()
    .addSite("US West users").nearestMetro(MetroCode.LA).weight(3).done()
    .addSite("EU users").nearestMetro(MetroCode.FR).weight(2).done()
    .requireProvider(CloudProviderType.AWS).done()
    .addWorkload("llm-inference")
        .type(WorkloadType.AI_ML_INFERENCE).bandwidthMbps(2_000).maxLatencyToleranceMs(50.0)
        .dependsOn(CloudProviderType.AWS).done()
    .addWorkload("training")
        .type(WorkloadType.AI_ML_TRAINING).bandwidthMbps(10_000).pinnedMetro("DC")
        .dependsOn(CloudProviderType.AWS).done()
    .constraints().redundancy(RedundancyTier.MULTI_METRO).maxMetros(3).done()
    .strategy(OptimizationStrategy.LATENCY_FIRST)
    .optimize();

placement.getPlacement().placements().forEach(System.out::println);
placement.getConstraintVerdicts().forEach(v -> System.out.println(v.constraintId() + " " + v.satisfied() + " " + v.reason()));

pinnedMetro("DC") holds the training workload where the GPUs are. The optimizer has no data on the power density and cooling of a facility, so the pin supplies the location of the GPU cage.

The latency in the result is network round trip between metros. The time the model takes, such as the time to first token, adds to the network latency.

Step 4: build#

The connections follow Hybrid multicloud. Three points are specific to this design.

• The GPU cloud provider publishes a service profile, which serviceProfiles().search() returns. Create an IP_VC from the router with zSideAccessPointServiceProfile(profileUuid, linkProtocol, "DC"). The provider tells you the authentication key, if it uses one.
• The cage ports for training are 100 Gbps ports, possibly in a LAG. A connection cannot exceed the largest bandwidth tier that Fabric offers for its type, so a higher rate needs several connections with ECMP on the cage routers.
• The Cloud Router package has to cover the route count and the aggregate throughput of the design. cloudRouters().routerPackages() returns the limits of each package.

Step 5: watch utilization during a run#

import com.eqixiac.equinix.fabric.model.Metric;

LocalDateTime now = LocalDateTime.now(ZoneOffset.UTC);
for (Metric metric : fabric.metrics().forConnection(connectionUuid, now.minusHours(6), now)) {
    System.out.println(metric.getName() + " " + metric.getUnit() + " " + metric.getDatapoints().size() + " points");
}

A training connection that stays at its bandwidth limit for hours limits the speed of the run, and one that peaks at 30% of its bandwidth is larger than the run needed. Estate analysis reports both cases from the 95th percentile.

Trade-offs#