Hello! Tomorrow I’m running a workshop at work about the humble IP address, so here are some notes about IP addresses and how IP routing on Linux works!
This came up because someone on my team pointed out that there’s actually a LOT going on with IP addresses even though it seems like a simple concept, and they said they’d like to learn more. Here goes!
This post is only about IPv4 because I’ve still never used IPv6.
What’s in the IP header?
Almost every packet your computer sends and receives (with some exceptions like ARP packets) has an IP header.
There are 14 fields in the IP header. The only 3 important ones for most people to know about are:
- the source IP address
- the destination IP address
- the TTL (see: how traceroute works)
One thing you’ll notice isn’t in the IP header is a port! The TCP and UDP protocols both have ports, but that lives at a different network layer. (and is why TCP port 8080 and UDP port 8080 are different ports, and can run different services!)
There’s also a ‘protocol’ field that tells you the protocol (like TCP/UDP).
What’s a subnet?
IP addresses are often grouped into subnets. The main useful thing to know about subnets is to
understand CIDR notation –
220.127.116.11/8 means “all the packets that have the same first 8 bits as
18.104.22.168”. In this case that would be
168.*.*.*, or any packet beginning in 168
(since each of the 4 numbers in an IP address is 8 bits).
When I create a packet on my computer, what happens to it?
Suppose you create a packet with IP address 22.214.171.124 on it. Where does it go? It turns out that this isn’t a super simple question – there are at least 3 possible systems that can affect your packet
System 1: The route table
The most likely system to affect your new packet destined to
126.96.36.199 is the route table. On
Linux, you can view your route table with
ip route list table all. Here’s what the route
table on my laptop looks like:
$ ip route list table all default via 192.168.1.1 dev wlp3s0 proto static metric 600 169.254.0.0/16 dev docker0 scope link metric 1000 linkdown 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 192.168.1.0/24 dev wlp3s0 proto kernel scope link src 192.168.1.170 metric 600 broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
Each of these routes has:
- a subnet (127.0.0.0/8, 172.17.0.0/16, etc)
- a network device (my wireless card
wlp3s0, the virtual loopback device
lo, the virtual docker device
- possibly something like
via 192.168.1.1which means “send all packets like this to IP 192.168.1.1’s MAC address, which happens to be Julia’s router”
- a bunch of other stuff which I don’t understand that well (
proto kernel, etc). Not understanding what those things mean hasn’t hurt me yet.
The main things to pay attention to are the subnet and the network device. So
wlp3s0 means “send packets in the
192.168.1.0/24 range to the
wlp3s0 network device. That’s not
It’s useful to know a little bit about your Linux computer route table if you’re doing container
networking, because with containers you’ll end up with one or more virtual network devices (like
docker0) that packets will end up being sent to.
That’s it for the route table!
System 2: iptables
Having read the above, you might think that the way packets get routed is:
- they come into your computer
- Linux looks at the route table and decides which network device to send to the packet to
- That’s it
That’s often true, but not always!! There are a bunch of secret in between steps (“prerouting”, “output”, “postrouting”) where Linux says “hey, iptables, want to make changes to this packet here?“. When this happens, iptables can change the source or destination IP address on the packet to be something different.
The two main things I’ve used this for are DNAT (“destination NAT”) and SNAT (“source NAT”)
Let’s start with destination NAT! One place this shows up is in this program called
kube2iam. kube2iam is this program that you run on your host
that pretends to be the AWS metadata endpoint (
169.254.169.254). Why you might want this isn’t
important right now, but – how can kube2iam pretend to be this other IP address? That would mean
that we need to magically redirect those packets somehow? How?
It turns out that forcing packets destined for 169.254.169.254 to go somewhere else is totally possible! Here’s the iptables rule that they suggest using:
iptables \ --append PREROUTING \ --protocol tcp \ --destination 169.254.169.254 \ --dport 80 \ --in-interface docker0 \ --jump DNAT \ --table nat \ --to-destination $YOUR_IP_ADDRESS:8181
Usually iptables rules make me want to hide under the couch but in the last year I’ve become a little less afraid of them. Here’s what’s going on with this one:
- it only applies to tcp packets to 169.254.169.254 port 80 that came from the
- it happens at the PREROUTING stage (before the packet gets assigned a network interface)
- it (
What’s this DNAT thing? Basically what this means is that Linux won’t just rewrite packets to 169.254.169.254:80 to go to $LOCAL_IP:8081, it’ll also modify the reply packets from $LOCAL_IP:8081 to make them appear as if they came from 169.254.169.254. So from the perspective of the application receiving the reply, it has no idea that it’s not talking to the IP 169.254.169.254. Lies!!!
To make all this work, Linux needs to keep track of connection state and remember “hey, this reply packet is part of this DNATed connection so I need to rewrite it”
Phew. Hopefully that made any sense.
Source NAT is like destination NAT, except instead of rewriting destination IP address, it rewrites source IP addresses!
The place I’ve used source NAT before is also for container stuff – if you have a bunch of containers with weird virtual container IP addresses sending packets to the outside world, you can’t just let them use those IP addresses!! The outside world (like google) has no idea about your container IPs and will not be able to reply to those packets. So you need to pretend that they come from your host.
If this is you, you probably want an iptables rule something like this on your host:
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
MASQUERADE is a confusing way of saying “use source NAT and pretend the packet is coming from this
host’s IP address”. What this rule does is rewrite the source IP address on every packet to pretend
to be from the host’s IP.
System 3: IPsec
I’m not going to go into this much (I wrote about IPsec the other day),
but a third way packets can end up going to weird places is if you’re using IPsec. You can see
what’s going on there with the
ip xfrm command. It turns out
xfrm stands for “transform”.
some notes on AWS networking
A lot of these concepts (a route table, NAT, IPSec) have their AWS networking analogs in Amazon Virtual Private Cloud (VPC) – much like you can control how packets get routed when they arrive on your Linux computer, you can use these AWS tools to control what happens when you send packets from an instance in a VPC. I looked at the VPC FAQ today and it’s pretty good.
here are some very rough mappings
- route table: VPC route table, which controls how packets are routed in your VPC
- iptables/source NAT: NAT gateway or Internet Gateway. If you have an AWS instance with no public IP address, you need to use source NAT (for the same reasons as we talked about before with containers) to talk to the public internet. The way this works in AWS is you set up a NAT gateway/internet gateway which will rewrite your packets for you.
- IPsec: VPC peering. This isn’t really the same thing (peering connections are only encrypted for cross-region traffic), but it does give you a way to set up private connections between two different VPCs and I think it’s useful in some of the same scenarios.