Per process routing take 2: using cgroups, iptables and policy routing

In a previous article we saw how it's possible to do per process routing using namespaces. In this one we will achieve the same by using cgroups, iptables and policy routing. Perhaps the user case is a bit marginal (see the introduction in the mentioned article) but this article is a tribute to the extreme flexibility of cgroups.

You will need a Linux kernel >= 3.14 and a modern iptables. The former is easy to obtain (at least on Debian via back-ported kernels or directly on Jessie), the later is a bit more difficult. Anyway I prepared a compiled binary, just point the IPT variable to it once unpacked in the root directory:


You have to correctly mount the cgroup file-system, the easiest way on Jessie is by installing the package cgroupfs-mount.

The method will be based on the 3 technologies mentioned in the title:

  • the cgroups net_cls controller will be used to set the classid of the packet originated from the process.
  • iptables will be used to mark the packet. This is possible thanks to the patch by Daniel Borkmann, see this thread for more information. If the entire patch had been accepted,  we would have used the new proposed controller. But the proliferation of cgroup controllers being a bad thing and the unclear semantics of fwmark (because it would be modifiable both by cgroups and iptables) had as a consequence that only the netfilter part of the patch got in the v3.14 kernel.
  • policy routing to define a new routing table wit a different default route that is triggered wit the fwmark.

In my opinion this method has 2 advantage over the one presented in the previous articles:

  • it's much easier to change the default route for processes (even already running) because it's easier to move a process into or out of a control group.
  • you don't need the bridging thing.

The clear disadvantage is that it's built on newer technologies not available out of the box on older distributions, like Debian Wheezy for example.

Now let's see how it works. First define a control group for the net_cls controller:

mkdir /sys/fs/cgroup/net_cls/new_route
cd /sys/fs/cgroup/net_cls/new_route
echo 0x00110011 > net_cls.classid

packet generated by processes in this control group will be annotated with the given 0x00110011 (11:11) classid. Next use iptables to fwmark packets:

$IPT -t mangle -A OUTPUT -m cgroup --cgroup 0x00110011 -j MARK --set-mark 11

note that it's very important to put the rule in this specific table and chain to trigger rerouting. Check out this picture on wikipedia, it's worth more that thousand words in describing the journey of a packet in the Linux network stack. Finally we have to declare an additional routing table for policy routing:

echo 11 new_route >> /etc/iproute2/rt_tables # just once!
ip rule add fwmark 11 table new_route
ip route add default via table new_route

here is the default gateway for the processes in the new_route control group. Now it's really easy to change the default route for a process, just add it to the control group. It's quite entraintaining to have a ping running and see how RTT changes based on the default gateway. You can find the PID for ping in the usual ways (ps is your best friend), let's say it's 2345:

cd /sys/fs/cgroup/net_cls/new_route
echo 2345 > tasks

and you can take it out from the control group easily:

echo 2345 > ../tasks

Keep in mind that when a process in a net_cls control group forks, its child will be in the same one. But if you move the parent, the child will stay there. Normal cgroups semantics applies.

This example gives just another application of the powerful cgroups concept. Others are of course possible, like per-process dynamic firewall rules or traffic control disciplines.

This entry was posted in Linux desktop and tagged , , , . Bookmark the permalink.

6 Responses to Per process routing take 2: using cgroups, iptables and policy routing

Leave a Reply

Your email address will not be published. Required fields are marked *