Understanding ENI, ENA AND EFA in AWS

Network Interfaces in aws EC2 instances-

While working with aws cloud, I am sure you have come across the terms ENI , ENA and EFA. These are various network interface cards which can be used in EC2 VM’s. In this article we will understand their use cases and how they are different from each other.

Elastic Network Interface (ENI)–

It is a virtual network interface that can be attached to an EC2 instance with in a Virtual Private Cloud (VPC). A primary ENI is always attached to an EC2 instance by default on launch. You can always add secondary interfaces depending on the need of your network architecture. They can be in different subnets but same Availibility Zone(AZ). It can be created from management console as well as using aws cli. Unless Delete on Termination flag is set(which is SET by default), the ENI will remain alive and well even after the instance is terminated. It can include following attributes..

A MAC address and a description.
A source/destination check flag.
A primary IPV4 address from the IPV4 address range of your VPC.
One or more secondary IPV4 addresses from IPV4 address range of your VPC (optional).
One elastic IP address per private IPV4 address.
One public IPV4 address (optional).
One or more IPV6 addresses (optional).
One or more security groups
Secondary network interface can be attached to a running instance (hot attach)

Use Cases–

It is the basic network adapter used with ec2 instances when you don’t have any high performance requirements like normal web servers , app servers , db servers etc.
It is used with all instance types.
User managed load balancers, proxy servers, NAT instances etc

Enhanced Network Adaptor (ENA)- It is the next generation network interface card and accompanying drivers which provides enhanced networking on EC2 instances. It is customized to deliver consistent low latency, high throughput and packet per second (PPS) performance on EC2 instances. It does not support the requirements of a tightly-coupled app.

Most of current generation instance types support enhanced networking. Enhanced networking can be enabled using one of the following mechanisms:

Elastic Network Adapter (ENA)- The Elastic Network Adapter (ENA) supports network speeds of up to 100 Gbps for supported instance types.The current generation instances except for T2, C4, D2, and M4 instances smaller than m4.16xlarge use ENA for enhanced networking, .

Intel 82599 Virtual Function (VF) interface-> The Intel 82599 Virtual Function interface supports network speeds of up to 10 Gbps for supported instance types.The following instance types use the Intel 82599 VF interface for enhanced networking(older instance type): C3, C4, D2, I2, M4 (excluding m4.16xlarge), and R3.

Use Cases:

Architectures patterns that require higher bandwidth and lower inter-instance latency.
Supported for limited instance types (HVM only)

Elastic Fabric Adapter (EFA)-

An Elastic Fabric Adapter is an AWS Elastic Network Adapter (ENA) with added capabilities. The EFA lets you apply the scale, flexibility, and elasticity of the AWS Cloud to tightly-coupled HPC apps. It is ideal for tightly coupled app as it uses the Message Passing Interface (MPI). With EFA, tightly-coupled HPC applications have access to lower and more consistent latency and higher throughput than traditional TCP channels, which enables them to scale better.

We can say that an EFA is an Elastic Network Adapter (ENA) with added capabilities. It provides all of the functionality of an ENA, with additional OS-bypass functionality.OS-bypass is an access model that allows HPC and machine learning applications to communicate directly with the network interface hardware to provide low-latency, reliable transport functionality.

With an EFA, HPC applications use MPI or NCCL to interface with the Libfabric API.The Libfabric API bypasses the operating system kernel and communicates directly with the EFA device to put packets on the network.This reduces overhead and enables the HPC application to run more efficiently.

Supported instance type: c5n.18xlarge, c5n.metal, i3en.24xlarge, i3en.metal, inf1.24xlarge, m5dn.24xlarge, m5n.24xlarge, r5dn.24xlarge, r5n.24xlarge, and p3dn.24xlarge. EFA support can be enabled either at the launch of the instance or added to a stopped instance. EFA devices cannot be attached to a running instance

EFA limitations-

You can attach only one EFA per instance.
EFA OS-bypass traffic is limited to a single subnet. In other words, EFA traffic cannot be sent from one subnet to another. Normal IP traffic from the EFA can be sent from one subnet to another.
EFA OS-bypass traffic is not routable. Normal IP traffic from the EFA remains routable.
The EFA must be a member of a security group that allows all inbound and outbound traffic to and from the security group itself.

Use Cases:

High-Performance Computing like computational fluid dynamics (CFD), crash simulations, and weather simulation.
MPI and Machine Learning use cases.
Tightly coupled applications.

Refrences –

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html
https://aws.amazon.com/hpc/faqs/