Lesson 5/2520%
MODULE 5 OF 25 12 MIN FPGA FUNDAMENTALS

FPGA vs GPU

Contrast FPGA spatial computing with GPU SIMD parallelism — understand when each wins for AI inference, radar DSP, streaming data, and high-frequency trading, and why latency often determines the winner.

Learning Objectives

  • Describe GPU SIMD architecture and explain the CUDA execution model
  • Contrast GPU data-parallel batching with FPGA streaming pipeline processing
  • Quantify GPU latency sources (PCIe transfer, kernel launch, scheduling overhead)
  • Explain why FPGA wins for streaming data and GPU wins for large batch workloads
  • Analyze real-world benchmarks comparing FPGA and GPU for ML inference and DSP
  • Match a given workload to the optimal platform — FPGA, GPU, or a combination

Two Kinds of Parallelism

GPUs and FPGAs are both parallel computing platforms — but they achieve parallelism in completely different ways. Understanding this architectural distinction is the key to making correct platform decisions in AI, signal processing, and data center applications.

A GPU is a SIMD (Single Instruction, Multiple Data) machine. It contains thousands of simple compute cores (CUDA cores in NVIDIA's terminology) organized into Streaming Multiprocessors (SMs). All cores in an SM execute the same instruction on different data simultaneously. To use a GPU efficiently, you must have a large batch of independent data items — the A100 GPU needs 6,912 CUDA cores all busy to reach peak efficiency. If you send it 1 item, 6,911 cores sit idle. The GPU also requires data to be transferred from CPU RAM to GPU VRAM over PCIe, adding 20–100 µs of overhead before any computation begins.

An FPGA is a spatial computing machine. Your algorithm is implemented as a custom hardware pipeline that exactly matches the shape of the data flow. A packet entering the FPGA immediately begins flowing through logic stages — no batch, no kernel launch, no PCIe transfer, no wasted cores. The first bit of data produces the first result in 10–200 nanoseconds. This is why FPGAs dominate in streaming applications where data arrives continuously and must be processed immediately.

Architecture: SIMD vs Spatial Pipeline

GPU — SIMD Model

  • Thousands of simple CUDA/shader cores
  • All cores execute same instruction (SIMD/SIMT)
  • High memory bandwidth (HBM2e: 2–3 TB/s)
  • Requires large data batches for efficiency
  • PCIe interface to host CPU (+20–100 µs latency)
  • GPU VRAM separate from CPU RAM
  • Power: 200–700W (A100: 400W, H100: 700W)
  • Programming: CUDA, OpenCL, ROCm
  • Excellent for: training, batch inference, graphics
  • Weak for: single-item latency, custom IO

FPGA — Spatial Pipeline Model

  • Custom logic fabric — hardware matches algorithm
  • Each pipeline stage has dedicated compute resources
  • Moderate bandwidth (DDR4/HBM: 64–460 GB/s)
  • Processes one item at a time, every clock cycle
  • Direct IO — no host transfer overhead
  • On-chip BRAM + external DDR tightly integrated
  • Power: 5–100W (typical: 15–75W)
  • Programming: VHDL, Verilog, HLS (Vitis)
  • Excellent for: streaming, custom IO, low latency
  • Weak for: training large models, arbitrary code

Execution Model Diagrams

graph LR subgraph GPU_arch["GPU — SIMD (Batch) Model"] Host["CPU Host\n(System RAM)"] -->|"PCIe\n+20-100µs"| GM["GPU\nGlobal Memory\n(VRAM HBM)"] GM --> SM1["SM Core 1\n(CUDA)"] GM --> SM2["SM Core 2\n(CUDA)"] GM --> SM3["SM Core N\n(6912 cores)"] SM1 --> RES["Batched\nResult"] SM2 --> RES SM3 --> RES end subgraph FPGA_arch["FPGA — Spatial Pipeline Model"] IO_in["Input Data\n(Direct IO\nQSFP/PCIe)"] --> P1["Pipeline\nStage 1\n~2ns"] P1 --> P2["Pipeline\nStage 2\n~2ns"] P2 --> P3["Pipeline\nStage N\n~2ns"] P3 --> IO_out["Output\n(nanoseconds\ntotal)"] end style GPU_arch fill:#1a0f40,stroke:#818cf8 style FPGA_arch fill:#0f2d1f,stroke:#22d3ee

The Latency Gap: Where It Comes From

The latency difference between GPU and FPGA is not just about processing speed — it's about the overhead stack that every GPU request must climb before any compute occurs. Understanding each component helps you understand why GPUs cannot serve certain markets regardless of how fast the GPU cores themselves are.

GPU Latency Stack (typical)

Application CPU code~1 µs
CUDA API call overhead~5 µs
PCIe DMA transfer (1KB data)~10–50 µs
GPU kernel scheduling~5–20 µs
Kernel execution~10–100 µs
Result DMA back to CPU~10–50 µs
Total~40–200+ µs

FPGA Latency Stack (direct IO)

IO pin propagation~1 ns
Input register capture~2 ns
Pipeline stage 1~2 ns/stage
Pipeline stages 2…N~2 ns/stage
Output register + IO pin~3 ns
(No PCIe, no OS, no DMA)0
Total~50–500 ns
The GPU's Core Weakness The GPU's biggest architectural weakness is not raw compute — it's latency. A GPU needs thousands of operations batched together to be efficient. Before it can process item #1, you must: call the CUDA API, DMA data over PCIe, schedule the kernel, launch warps, and wait for execution. By the time result #1 arrives, an FPGA has processed thousands of items. For streaming real-time data, this overhead is fatal.

Power Efficiency: A Critical Differentiator

Power is often the deciding factor at the edge — in autonomous vehicles, industrial equipment, telecom radio units, and medical devices where power budgets are tight. A 700W H100 GPU requires a server-grade power supply and cooling infrastructure. An Xilinx Alveo U50 FPGA draws 75W and fits in a standard PCIe slot with passive cooling.

For ML inference at the network edge, the power efficiency difference is dramatic. AMD Xilinx published benchmarks showing Alveo U50 achieving better latency than a V100 GPU for BERT-base inference at 3x lower power. Versal AI Core series achieves up to 400 TOPS while consuming under 100W — compared to A100's 312 TOPS at 400W. The FPGA's ability to implement custom low-precision arithmetic (INT4, INT6, arbitrary fixed-point) further multiplies this advantage, since fewer transistors switch per MAC operation.

Performance Comparison by Workload

Power Consumption Comparison

FPGA vs GPU — Full Comparison

Criteria GPU FPGA Winner
Parallelism Type SIMD — same instruction, many data Spatial — custom pipeline, arbitrary structure Context-dependent
Peak FLOPS (top model) H100: ~2000 TFLOPS (FP8) Versal: ~400 TOPS (INT8) GPU wins
Streaming Latency 40–200+ µs (PCIe + kernel launch) 50–500 ns (direct pipeline) FPGA wins 100–1000x
Batch ML Training Excellent — purpose-built matrix ops Possible but not competitive GPU wins
Edge ML Inference Overkill — too much power/latency Optimal — low power, custom bit width FPGA wins
Packet Processing Not viable — PCIe overhead kills latency Wire-rate with deterministic latency FPGA wins decisively
Radar / Sonar DSP Too high latency, no determinism Real-time pipeline at ns latency FPGA wins
HFT Order Management Not viable (>50 µs latency) ~1.2 µs order-to-send FPGA wins
Video Transcoding (batch) High throughput (NVENC hardware) Real-time encode with custom latency Tie — GPU for batch, FPGA for real-time
Custom Interfaces PCIe/NVLink only Any standard: QSFP, SMPTE, MIPI, custom FPGA wins
Arbitrary Bit Widths FP16, BF16, INT8, FP8 (limited) Any width: INT4, INT6, custom FP FPGA wins
Power Consumption 200–700W per card 5–100W per card FPGA wins 5–50x efficiency
Ease of Programming CUDA well-documented; Python frameworks RTL or HLS — steep learning curve GPU wins significantly
Monte Carlo / Simulation Excellent — massively parallel RNG Possible but GPU is more practical GPU wins
Reconfigurability Programmable but fixed GPU architecture Fully reconfigurable hardware FPGA wins
Engineering Tip — Edge AI For ML inference at the network edge — where power, latency, and cost all matter — FPGAs increasingly beat GPUs. AMD Xilinx Alveo U50 achieves 2x better latency than NVIDIA V100 for BERT-base inference at 1/5th the power. Versal AI Core with its AI Engine array targets sub-millisecond inference for autonomous driving and 5G base station inference workloads where a 400W GPU is simply not an option.
Interview Question Q: Why can't a GPU handle radar signal processing with the same performance as an FPGA?
A: Radar requires processing each pulse return in microseconds with guaranteed deterministic timing — every pulse, every time, with no jitter. A GPU cannot provide this because: (1) PCIe data transfer takes 10–100 µs per transfer; (2) CUDA kernel launch adds 5–20 µs overhead; (3) GPU scheduling is non-deterministic under load; (4) there is no way to directly connect radar RF hardware to a GPU's PCIe interface. An FPGA connects directly to the ADC output, processes each sample in a hardware pipeline with nanosecond latency and absolute determinism.
Common Mistake Do not benchmark FPGA vs GPU for training large neural networks. GPUs were purpose-built for this exact workload — terabytes of training data, massive matrix multiplications, gradient updates across millions of parameters. No FPGA comes close to an H100 for ResNet-50 training. The correct comparison is for inference, streaming DSP, and custom IO workloads, where the architectural trade-offs change the outcome dramatically.

Real-World Performance Data

Published Benchmarks

Workload GPU FPGA FPGA Advantage
BERT-base Inference Latency V100: ~3 ms (batch=1) Alveo U50: ~1.2 ms 2.5x lower latency
BERT Inference Power Efficiency V100: 300W Alveo U50: 75W 4x better perf/watt
HFT Order-to-Send Latency GPU: ~50 µs (software path) FPGA: ~1.2 µs 40x lower latency
100 Gbps Packet Classification Not viable (PCIe bottleneck) Wire-rate (<100 ns/packet) Only FPGA viable
4096-pt FFT at 1 GSPS Not viable (latency/throughput) FPGA: full rate, deterministic Only FPGA viable
ResNet-50 Training (ImageNet) A100: ~7 min/epoch Not practical GPU wins clearly
LLM Inference (Llama 70B) H100: ~60 tokens/sec Not practical (memory constraints) GPU wins for large LLMs

Knowledge Check

1. What type of parallelism does a GPU use?
  • Spatial parallelism — custom hardware for each operation
  • SIMD/SIMT — same instruction executed on thousands of data items simultaneously
  • Sequential pipelining — one operation at a time in a long pipeline
  • Von Neumann — fetch, decode, execute on one data item
Correct! GPUs use SIMD (Single Instruction, Multiple Data) or NVIDIA's SIMT (Single Instruction, Multiple Threads) model. All CUDA cores in a warp execute the same instruction on different data simultaneously. This is extremely efficient for matrix operations but requires large data batches to utilize the thousands of cores.
2. For processing a live 4K camera stream with a latency requirement of under 1 millisecond, which is a better choice?
  • GPU — higher TFLOPS means lower latency
  • FPGA — streaming pipeline with direct sensor IO, nanosecond latency
  • Both are equivalent — latency depends on algorithm complexity, not hardware
  • GPU with NVLink — eliminates PCIe bottleneck
Correct! FPGA wins for low-latency streaming. A GPU must receive data over PCIe, schedule a kernel, execute, and return results — total overhead of 40–200+ µs, which already exceeds a 1 ms budget for complex models. An FPGA connects directly to the camera sensor (e.g., MIPI CSI-2 interface in the programmable IO) and processes each frame through a hardware pipeline with sub-millisecond total latency.
3. A typical high-end GPU (H100) consumes approximately how much power?
  • 50–100W
  • 150–250W
  • 600–700W
  • 1000–1500W
Correct! The NVIDIA H100 SXM5 has a 700W TDP. The A100 is 400W. Even the inference-optimized L40S is 350W. Compare to an Xilinx Alveo U50 at 75W — a 5–9x power advantage for the FPGA at equivalent or better inference latency. For edge deployment, this difference is the difference between feasible and infeasible.
4. For training a large language model (GPT-4 scale) with hundreds of billions of parameters, which platform wins?
  • FPGA — custom bit widths give better efficiency
  • GPU — massive matrix operations, HBM bandwidth, and CUDA ecosystem are purpose-built for this
  • CPU cluster — more memory capacity available
  • ASIC — lowest power per FLOP
Correct! GPU dominates LLM training. GPT-4 scale training required tens of thousands of A100 GPUs running for months. The reasons: (1) training requires FP16/BF16 matrix multiply which GPU Tensor Cores execute at 312 TFLOPS; (2) GPUs have HBM2e memory at 2 TB/s bandwidth; (3) NVLink/InfiniBand allows multi-GPU gradient aggregation; (4) PyTorch/JAX/DeepSpeed provide mature training frameworks. FPGAs have none of this ecosystem for LLM training.
5. FPGAs can implement custom numeric formats like INT4 or 6-bit fixed-point arithmetic. True or False?
  • True — FPGA logic can implement any arbitrary bit width arithmetic
  • False — FPGAs are limited to standard IEEE 754 floating point formats
  • True — but only for powers of 2 (INT1, INT2, INT4, INT8)
  • False — custom arithmetic requires an ASIC
Correct — True! FPGA LUTs implement arbitrary boolean functions, so you can build multipliers and accumulators at any bit width: 4-bit, 6-bit, 9-bit, 17-bit — whatever the algorithm requires. This is a significant advantage for neural network inference, where quantization research has shown INT4 weights work well for many models. Fewer bits per multiply means more MACs fit in the same FPGA resources, multiplying effective throughput.

Practical Exercise — Workload Routing

Platform Selection: GPU or FPGA?

For each workload below, choose GPU or FPGA. Write a 2–3 sentence justification addressing: batch size, latency requirement, power budget, and any custom IO needs.

A
Training ResNet-50 on the full ImageNet dataset (1.2 million images). Target: complete each training epoch in under 10 minutes. Budget: unlimited datacenter power and servers.
B
Edge AI in a factory camera: detect assembly defects in real-time at 60fps, 4K resolution. Latency requirement: <5ms from frame capture to alert. Power budget: 25W total (embedded system, no active cooling).
C
100 Gbps network intrusion detection: classify every TCP/UDP packet on a 100G backbone link. Latency budget: <500 ns per packet. Hardware connects to a QSFP28 optical module.
D
4096-point FFT for an S-band radar: sample rate 1 GSPS, process each FFT in real-time (one new FFT output per 4096 ADC samples = every 4 µs). Connect directly to a 12-bit ADC at 1 GSPS. No batching possible.
E
Monte Carlo simulation for option pricing: run 1 billion random paths, each path is independent, final answer needed in under 30 seconds. Runs in a datacenter with abundant power. No latency constraint — throughput only.
Hint Apply the three questions: (1) Is data arriving in a continuous stream requiring immediate per-item response? → FPGA. (2) Is the workload a large batch where all items can be processed together? → GPU. (3) Is there a custom electrical interface (ADC, QSFP, camera sensor)? → FPGA.

Lesson Summary

Key Takeaways

  • GPU uses SIMD/SIMT parallelism — same instruction on thousands of data items; requires large batches for efficiency
  • FPGA uses spatial pipeline parallelism — custom hardware matches algorithm structure; processes streaming data item-by-item with nanosecond latency
  • GPU latency stack: PCIe transfer + kernel launch + execution = 40–200+ µs. FPGA latency: 50–500 ns (no overhead)
  • GPU wins: batch ML training, Monte Carlo simulation, large language model inference, ray tracing
  • FPGA wins: streaming data processing, edge ML inference, radar/DSP, packet processing, HFT, custom electrical interfaces
  • Power: GPU = 200–700W. FPGA = 5–100W. FPGA provides 5–50x better power efficiency for matching workloads
  • FPGAs support arbitrary bit widths (INT4, INT6, custom fixed-point) — enabling more MACs per watt for quantized neural networks
  • The optimal system often uses both: GPU for batch training and large-scale inference; FPGA for edge inference, IO, and streaming
← FPGA vs ASIC AMD Xilinx FPGA Families →