Automating the protection of a network from denial of service with RTBH, NetFlow, goBGP and FastNetMon

Translations

Overview

In this article, I will introduce Remotely-Triggered Black Hole (RTBH), a technology based on the Border Gateway Protocol (BGP) routing protocol and designed to provide the ability to block unwanted network traffic in order to protect against Distributed Denial-of-Service (DDoS) attacks. The objective here is to share with you an example of usage of RTBH in an automated manner in an environment consisting of Cisco routers and the FastNetMon security solution supported by the GoBGP software and the NetFlow protocol.

Introduction

In computing networks, there are many technologies available to enable telecommunications to be carried out quickly and efficiently. Routing, which aims to determine the best route for a data packet to take from a source to its destination, is one of the key elements. Unfortunately, it also enables attackers to propagate powerful attacks, particularly to undermine the availability of data and services. On the other hand, some technologies are also available to detect and propagate responses in the same way, in order to block attacks as close as possible to their originators.
First, we will look at what denial-of-service attacks are, which are among the most common. We will then look at the BGP routing protocol and the RTBH technology, which is based on BGP and makes it possible to block denial-of-service attacks on a network. Finally, we will demonstrate in a test environment how to implement RTBH using the FastNetMon and GoBGP tools along with the NetFlow protocol.

Denial of service attacks

Denial of Service (DoS) attacks aim to make a host, a network or a service unavailable. To do this, they can be aimed directly at the target by sending one or more malicious data packets designed to exploit a vulnerability on the target system, or they can overload it with requests in order to exhaust its resources. This overload can also be propagated over the network links of a targeted provider or organisation. To make the attack even more effective, several sources of malicious packets are often used, and the denial-of-service is then marked as Distributed DoS (DDoS). Therefore, organisations need to find effective ways of detecting and stopping these attacks as soon as they enter their network infrastructure.

BGP

Border Gateway Protocol (BGP) is a dynamic routing protocol for exchanging routing information between or within Autonomous Systems (AS: administrative entity managing routing prefixes) on the Internet. When BGP is used within an AS, it is referred to as Interior BGP (IBGP), and when it is used between ASes, it is referred to as Exterior BGP (EBGP). BGP is one of the most widespread protocols and is one of the pillars of the Internet. BGP allows routing decisions to be determined by establishing sessions between neighbors / peers via the Transmission Control Protocol (TCP) transport protocol. In these sessions, routing announcements (more precisely, network prefixes and their characteristics) are exchanged, including the gateways or next hop for each route. Each route is associated with a set of attributes (weight, local preference, whether the network is local or originate, AS path, origin, next hop, etc.) which are used to select the best route for a given destination, according to an order of preference. An attribute called community, coded on 4 octets, is used to add additional information on an announced prefix and thus make additional decisions on the processing of BGP announcements.
Routes are exchanged via UPDATE messages containing a*Network Layer Reachability Information (NLRI) field including the prefixes.

Simplified BGP network topology example

Moreover, one of the many advantages of BGP is the ability to define a given next hop in a route advertisement. This makes it possible to send traffic to a fake gateway known as a blackhole.

RTBH

Principle

Remotely-Triggered Black Hole (RTBH) is a technique used with BGP to block malicious traffic. It is generally used by Internet Service Providers (ISPs) to protect their customers from denial-of-service attacks. A trigger is used to send traffic described in a black hole via a dedicated routing advertisement. The description can be made on the destination (target), which is referred to as Destination-based RTBH; or on the source of the traffic, which is referred to as Source-based RTBH.
The component activating the trigger must first detect that the traffic is malicious. For example, it may detect a quantity of traffic that suddenly exceeds a predefined threshold set for normal use. This component can be located on the routing infrastructure or on a dedicated host that interacts with the routing infrastructure. It can also be part of the ISP or customer network.
The next-hop used in the RTBH routing advertisement is usually set to an Internet Protocol (IP) address in the 192.0.2.0/24 subnet. The address used for anti-DDoS purposes is 192.0.2.1. The next hop of the route to 192.0.2.1 is then set to a non-existent interface called discard or null. This route is therefore pre-installed.
RTBH advertisements are not exported to other ASes so as not to impact other organisations, thanks to the use of the NO_EXPORT community. In addition, a specific community for the black hole must be added to these announcements. It must have a decimal value of 666 over its last 2 bytes.

Example of a simplified RTBH application

Automating RTBH

The RTBH concept seems powerful for propagating responses in order to block DDoS attacks via BGP. However, in a large-scale environment, this process can be improved by carrying it out automatically. To do this, there are two mechanisms that need to be automated: analysing the traffic to detect the attack in progress and sending an RTBH response to block it.

The first mechanism can be implemented using packet capture and traffic mirroring such as SwitchPort ANalyzer (SPAN: copying traffic from one interface to another), which can be broken down into Local SPAN (LSPAN: copying takes place within the same device), Remote SPAN (RSPAN: includes multiple interfaces, virtual subnets and devices) or Encapsulated RSPAN (ERSPAN: Cisco proprietary, where captured traffic is encapsulated so that it can be transported intact across multiple networks). That said, capturing and copying traffic consumes resources and is less suitable for detecting attack patterns based mainly on volume alone, such as DDoS. We actually prefer to use it to analyse packets in depth.

Traffic capture with SPAN

In addition, the traffic analysis mechanism can also be implemented by sending samples of the observed traffic, known as flows. A flow is a description of network traffic (a set of packets) defined by several fields (input interface, protocol, source port, destination port, source IP address, destination IP address, the Type of Service field in the IP header). The most common flow sampling technologies are Netflow (Cisco proprietary), sample flow (sFlow) and Internet Protocol Flow Information Export (IPFIX). They are best suited to detecting DDoS attacks in a production environment.

Collecting traffic information with flow

Once the attack has been detected on the flows, the response is propagated via the RTBH mechanism with BGP announcements. In particular, it is then interesting to be able to characterise traffic with a Flow Specification (FlowSpec), which is a set of matching criteria for network traffic. Then this is associated with filtering rules that also contain actions to be taken on the traffic. For example, we might consider blocking traffic, applying traffic rate limits, or marking packets to implement Quality of Service (QoS) policies. These actions are defined by specific BGP communities. These criteria are then encoded in the NLRI field of BGP messages. Thus, it is possible to react quickly and efficiently to control network traffic.

RTBH implementation with Cisco, NetFlow GoBGP and FastNetMon

I’m now going to show you how RTBH can be used in an environment with Cisco routers and with a FastNetMon probe that also uses the GoBGP solution (an open source implementation of BGP written in the Go language). The objective is to demonstrate how a simple denial-of-service attack can be detected via NetFlow collection and analysis on FastNetMon and then automatically remediated using RTBH via GoBGP. The environment was emulated on the Packet Network Emulator Tool Lab (PNETLab) and some machines were prepared by one of my colleagues using the procedure described in this excellent article.

Topology

The network topology set up in the test environment consists of several elements:

  • An Attacker machine running on the Linux Debian system and simulating an attacker;
  • A ProviderEdge1 router running on the Cisco IOS-XE system, simulating an ISP AS router;
  • A ProviderEdge2 router running on the Cisco IOS-XE system and simulating an ISP AS router;
  • A CustomerEdge1 router running on the Cisco IOS-XE system and simulating a customer AS router;
  • A Target machine running on the Linux Debian system and simulating a target;
  • A FastNetMon machine running on the Linux Debian system and running FastNetMon and GoBGP.

Topology of the test environment

Configuration of BGP and RTBH

Here are the configuration lines for the routers and the FastNetMon machine to perform BGP and RTBH.

Configuration of ProviderEdge1:

router bgp 65000
 neighbor 10.10.2.2 remote-as 65000
 neighbor 10.10.2.2 activate
 neighbor 10.10.2.2 send-community
 neighbor 10.10.2.2 route-map RTBH in
 network 10.10.1.0 mask 255.255.255.0
 
ip bgp-community new-format
ip community-list expanded RTBH permit 65000:666
ip route 192.0.2.1 255.255.255.255 Null0

route-map RTBH permit 10 
 match community RTBH
 set community no-export
 set ip next-hop 192.0.2.1
route-map RTBH permit 20

ip icmp rate-limit unreachable 

  • BGP is activated on AS 65000 by advertising the local network 10.10.1.0/24, declaring the neighbor ProviderEdge2 in IBGP and authorising the exchange of BGP information, including the inclusion of the community attribute.
  • The “RTBH” community is described with the value 65000:666.
  • A specific route-map is created which is activated when the “RTBH” community is received in BGP announcements. This route-map sets the next-hop to 192.0.2.1 and only authorises the propagation of these announcements in IBGP.
  • A pre-installed static route sends traffic to 192.0.2.1 on a non-existent interface called Null0.
  • In addition, we set the maximum frequency for sending Internet Control Message Protocol (ICMP) Destination Unreachable responses (sent in particular when traffic is not routed) to 10ms. This is optional, but will be useful for gaining greater visibility during the tests we will be performing below.

Configuration of ProviderEdge2:

router bgp 65000
 neighbor 10.10.2.1 remote-as 65000
 neighbor 10.10.2.1 activate
 neighbor 10.10.2.1 send-community
 neighbor 10.10.2.1 next-hop-self
 neighbor 10.10.3.2 remote-as 65001
 neighbor 10.10.3.2 ebgp-multihop 255
 neighbor 10.10.3.2 activate
 neighbor 10.10.3.2 send-community
 neighbor 10.10.3.2 route-map RTBH in

ip bgp-community new-format
ip community-list expanded RTBH permit 65000:666
ip route 192.0.2.1 255.255.255.255 Null0

route-map RTBH permit 10 
 match community RTBH
 set ip next-hop 192.0.2.1 
route-map RTBH permit 20 

  • BGP is activated on AS 65000 by declaring the neighbor ProviderEdge1 in IBGP and the neighbor CustomerEdge1 in EBGP as well as authorising the exchange of BGP information with the inclusion of the community attribute. The next-hop of routes learned in EBGP thanks to CustomerEdge1 is replaced by the router ProviderEdge2 itself.
  • The “RTBH” community is described with the value 65000:666.
  • A specific route-map is created and is activated when the “RTBH” community is received with a prefix. This route-map sets the next-hop to 192.0.2.1 and only authorises the propagation of these announcements in IBGP.
  • A pre-installed static route sends traffic to 192.0.2.1 on a non-existent interface called Null0.

Configuration of CustomerEdge1:

router bgp 65001
 neighbor 10.10.3.1 remote-as 65000
 neighbor 10.10.3.1 activate
 neighbor 10.10.3.1 send-community
 network 10.10.4.0 mask 255.255.255.0
 redistribute static route-map RTBH
 neighbor 192.168.10.10 remote-as 65001
 neighbor 192.168.10.10 activate

ip bgp-community new-format
ip community-list expanded RTBH permit 65000:666

route-map RTBH permit 10 
 match tag 666
 match community RTBH
 set community 65000:666
route-map RTBH permit 20

  • BGP is activated on AS 65001 by advertising the local network 10.10.4.0/24, declaring the neighbor ProviderEdge2 as EBGP and authorising the exchange of BGP information, with the inclusion of the community attribute, and declaring the neighbor FastNetMon as IBGP.
  • The community “RTBH” is described with the value 65000:666.
  • We create a specific route-map which is activated when the 666 tag (which we use for local tests) or the “RTBH” community are received with a prefix. This route-map sets the next-hop to 192.0.2.1 and only authorises the propagation of these announcements in IBGP. This route-map also sets the community to 65000:666 (in the case where only one tag has been received and recognised).
  • The static routes associated with this route-map are redistributed via BGP.

Configuration of GoBGP on the FastNetMon machine:

$ vi /etc/systemd/system/multi-user.target.wants/gobgpd.service
[Unit]
Description=GoBGP Routing Daemon
Documentation=file:/usr/share/doc/gobgpd/getting-started.md
After=network.target syslog.service
ConditionPathExists=/etc/gobgpd.conf

[Service]
Type=notify
ExecStartPre=/opt/fastnetmon-community/libraries/gobgp_3_12_0/gobgpd -f /etc/gobgpd.conf -d
ExecStart=/opt/fastnetmon-community/libraries/gobgp_3_12_0/gobgpd -f /etc/gobgpd.conf --sdnotify --disable-stdlog --syslog yes $GOBGPD_OPTIONS
ExecReload=/opt/fastnetmon-community/libraries/gobgp_3_12_0/gobgpd -r
DynamicUser=yes
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

$ systemctl daemon-reload
$ systemctl enable gobgpd.service
$ ln -s /opt/fastnetmon-community/libraries/gobgp_3_12_0/gobgp /usr/local/bin/.
$ cat /etc/gobgpd.conf 
[global.config]
    as = 65001
    router-id = "192.168.10.10"
 
[[neighbors]]
  [neighbors.config]
      neighbor-address = "192.168.10.1"
      peer-as = 65001
$ systemctl start gobgpd.service

  • A daemon for GoBGP is created and loads the GoBGP configuration file at start-up as well as a symbolic link to simplify the use of the GoBGP binary.
  • The daemon is activated.
  • The BGP configuration is defined on the AS 65001 by declaring the ProviderEdge2 neighbour as IBGP.
  • The GoBGP daemon is started.

NetFlow configuration

Configuration of CustomerEdge1:

flow record FLOW-RECORD
 match ipv4 protocol
 match ipv4 destination address
 match ipv4 source address
 collect counter bytes
 collect counter packets

flow exporter FLOW-EXPORTER
 destination 192.168.10.10
 source GigabitEthernet3
 transport udp 2055
 
flow monitor FLOW-MONITOR
 exporter FLOW-EXPORTER
 cache timeout active 1
 cache timeout inactive 1
 record netflow ipv4 original-input

interface GigabitEthernet2
 ip flow monitor FLOW-MONITOR input
 ip flow monitor FLOW-MONITOR output

  • A flow record is defined to define the collection of flows (including the number of bytes and packets per flow) on IP traffic.
  • A flow exporter is defined to characterize the target for sending flows, in this case the FastNetMon machine on the User Datagram Protocol (UDP) transport protocol via port 2055.
  • A flow monitor is defined with deliberately low cache values, using the flow exporter and applied to the interface connected to the ProviderEdge2 router (more precisely, on the input and output of the interface from which the attacks could originate).

Configuration of FastNetMon

On the FastNetMon machine, we configure the tool by editing certain lines in the /etc/fastnetmon.conf configuration file:

# Logging level, can be info or debug
logging_level = info
# enable this option if you want to send logs to local syslog facility
logging_local_syslog_logging = on

# Enable/Disable any actions in case of attack
enable_ban = on
# How long (in seconds) we should keep an IP in blocked state
# If you set 0 here it completely disables unban capability
ban_time = 10
# Check if the attack is still active, before triggering an unban callback with this option
# If the attack is still active, check each run of the unban watchdog
unban_only_if_attack_finished = off

# list of all your networks in CIDR format
networks_list_path = /etc/networks_list

# Different approaches to attack detection
ban_for_pps = on
# Limits for Dos/DDoS attacks
threshold_pps = 5
threshold_icmp_pps = 5

ban_for_icmp_pps = on

# Netflow capture method with v5, v9 and IPFIX support
netflow = on

# Configuration for Netmap, mirror, pcap, AF_XDP modes
# For pcap we could specify "any"
# For Netmap we could specify multiple interfaces separated by comma
interfaces = ens3

# We use average values for traffic speed to certain IP and we calculate average over this time periond (seconds)
average_calculation_time = 2

# Delay between traffic recalculation attempts
speed_calculation_delay = 1

# it's possible to specify multiple ports here, using commas as delimiter
netflow_port = 2055

netflow_sampling_ratio = 1

# GoBGP integration
gobgp = on

# Configuration for IPv4 announces
gobgp_next_hop = 0.0.0.0
gobgp_announce_host = on
gobgp_announce_whole_subnet = off

gobgp_community_host = 65000:666

  • Logging is activated locally.
  • The blocking function is activated for a period of 10s when the observed number of packets per second is greater than or equal to 5. Traffic statistics are calculated over 2s and every 1s. Note that this value is low because we are in a test environment and it is not suitable for a production environment.
  • We listen to the traffic on the interface connected to the CustomerEdge1 router.
  • NetFlow support is enabled on UDP port 2055 with a 1:1 sampling.
  • GoBGP is enabled with the ability to make RTBH announcements for hosts and using the community value 65000:666.

Next, in the /etc/networks_list file, we specify the target network to monitor:

$ cat /etc/networks_list 
10.10.4.0/24

Last, the service is started:
$ systemctl start fastnetmon

Test scenario

For the tests, we carry out an ICMP Flooding DoS attack by sending ICMP Echo Request messages from the Attacker machine to the Target machine with 100ms intervals using the ping utility. This allows us to send 10 packets per second. Of course, this is not representative of a real attack, but it’s enough to show you how automated RTBH works.

$ ping 10.10.4.10 -i 0.1
PING 10.10.4.10 (10.10.4.10) 56(84) bytes of data.
64 bytes from 10.10.4.10: icmp_seq=1 ttl=61 time=1.44 ms
64 bytes from 10.10.4.10: icmp_seq=2 ttl=61 time=1.60 ms
64 bytes from 10.10.4.10: icmp_seq=3 ttl=61 time=1.67 ms
64 bytes from 10.10.4.10: icmp_seq=4 ttl=61 time=1.73 ms
64 bytes from 10.10.4.10: icmp_seq=5 ttl=61 time=1.62 ms
64 bytes from 10.10.4.10: icmp_seq=6 ttl=61 time=1.69 ms
64 bytes from 10.10.4.10: icmp_seq=7 ttl=61 time=1.83 ms
64 bytes from 10.10.4.10: icmp_seq=8 ttl=61 time=1.52 ms
64 bytes from 10.10.4.10: icmp_seq=9 ttl=61 time=1.69 ms
64 bytes from 10.10.4.10: icmp_seq=10 ttl=61 time=1.47 ms
64 bytes from 10.10.4.10: icmp_seq=11 ttl=61 time=1.41 ms
64 bytes from 10.10.4.10: icmp_seq=12 ttl=61 time=1.48 ms
64 bytes from 10.10.4.10: icmp_seq=13 ttl=61 time=1.50 ms
64 bytes from 10.10.4.10: icmp_seq=14 ttl=61 time=1.67 ms
64 bytes from 10.10.4.10: icmp_seq=15 ttl=61 time=1.55 ms
64 bytes from 10.10.4.10: icmp_seq=16 ttl=61 time=1.59 ms
64 bytes from 10.10.4.10: icmp_seq=17 ttl=61 time=1.61 ms
64 bytes from 10.10.4.10: icmp_seq=18 ttl=61 time=1.60 ms
64 bytes from 10.10.4.10: icmp_seq=19 ttl=61 time=1.65 ms
64 bytes from 10.10.4.10: icmp_seq=20 ttl=61 time=1.49 ms
64 bytes from 10.10.4.10: icmp_seq=21 ttl=61 time=1.49 ms
64 bytes from 10.10.4.10: icmp_seq=22 ttl=61 time=1.58 ms
64 bytes from 10.10.4.10: icmp_seq=23 ttl=61 time=1.59 ms
64 bytes from 10.10.4.10: icmp_seq=24 ttl=61 time=1.44 ms
64 bytes from 10.10.4.10: icmp_seq=25 ttl=61 time=1.61 ms
64 bytes from 10.10.4.10: icmp_seq=26 ttl=61 time=1.60 ms
64 bytes from 10.10.4.10: icmp_seq=27 ttl=61 time=1.72 ms
64 bytes from 10.10.4.10: icmp_seq=28 ttl=61 time=1.62 ms
64 bytes from 10.10.4.10: icmp_seq=29 ttl=61 time=1.68 ms
64 bytes from 10.10.4.10: icmp_seq=30 ttl=61 time=1.61 ms
64 bytes from 10.10.4.10: icmp_seq=31 ttl=61 time=1.59 ms
64 bytes from 10.10.4.10: icmp_seq=32 ttl=61 time=1.74 ms
64 bytes from 10.10.4.10: icmp_seq=33 ttl=61 time=1.70 ms
64 bytes from 10.10.4.10: icmp_seq=34 ttl=61 time=1.54 ms
64 bytes from 10.10.4.10: icmp_seq=35 ttl=61 time=1.80 ms
64 bytes from 10.10.4.10: icmp_seq=36 ttl=61 time=1.50 ms
From 10.10.1.1 icmp_seq=37 Destination Host Unreachable
From 10.10.1.1 icmp_seq=38 Destination Host Unreachable
From 10.10.1.1 icmp_seq=39 Destination Host Unreachable
From 10.10.1.1 icmp_seq=40 Destination Host Unreachable
From 10.10.1.1 icmp_seq=41 Destination Host Unreachable
From 10.10.1.1 icmp_seq=42 Destination Host Unreachable
From 10.10.1.1 icmp_seq=43 Destination Host Unreachable
From 10.10.1.1 icmp_seq=44 Destination Host Unreachable
From 10.10.1.1 icmp_seq=45 Destination Host Unreachable
From 10.10.1.1 icmp_seq=46 Destination Host Unreachable
From 10.10.1.1 icmp_seq=47 Destination Host Unreachable
From 10.10.1.1 icmp_seq=48 Destination Host Unreachable
From 10.10.1.1 icmp_seq=49 Destination Host Unreachable
From 10.10.1.1 icmp_seq=50 Destination Host Unreachable
From 10.10.1.1 icmp_seq=51 Destination Host Unreachable
From 10.10.1.1 icmp_seq=52 Destination Host Unreachable
From 10.10.1.1 icmp_seq=53 Destination Host Unreachable
From 10.10.1.1 icmp_seq=54 Destination Host Unreachable
From 10.10.1.1 icmp_seq=55 Destination Host Unreachable
From 10.10.1.1 icmp_seq=56 Destination Host Unreachable
From 10.10.1.1 icmp_seq=57 Destination Host Unreachable
From 10.10.1.1 icmp_seq=58 Destination Host Unreachable
From 10.10.1.1 icmp_seq=59 Destination Host Unreachable
From 10.10.1.1 icmp_seq=60 Destination Host Unreachable
From 10.10.1.1 icmp_seq=61 Destination Host Unreachable
From 10.10.1.1 icmp_seq=62 Destination Host Unreachable
From 10.10.1.1 icmp_seq=63 Destination Host Unreachable
From 10.10.1.1 icmp_seq=64 Destination Host Unreachable
From 10.10.1.1 icmp_seq=65 Destination Host Unreachable
From 10.10.1.1 icmp_seq=66 Destination Host Unreachable
From 10.10.1.1 icmp_seq=67 Destination Host Unreachable
From 10.10.1.1 icmp_seq=68 Destination Host Unreachable
From 10.10.1.1 icmp_seq=69 Destination Host Unreachable
From 10.10.1.1 icmp_seq=70 Destination Host Unreachable
From 10.10.1.1 icmp_seq=71 Destination Host Unreachable
From 10.10.1.1 icmp_seq=72 Destination Host Unreachable
From 10.10.1.1 icmp_seq=73 Destination Host Unreachable
From 10.10.1.1 icmp_seq=74 Destination Host Unreachable
From 10.10.1.1 icmp_seq=75 Destination Host Unreachable
From 10.10.1.1 icmp_seq=76 Destination Host Unreachable
From 10.10.1.1 icmp_seq=77 Destination Host Unreachable
From 10.10.1.1 icmp_seq=78 Destination Host Unreachable
From 10.10.1.1 icmp_seq=79 Destination Host Unreachable
From 10.10.1.1 icmp_seq=80 Destination Host Unreachable
From 10.10.1.1 icmp_seq=81 Destination Host Unreachable
From 10.10.1.1 icmp_seq=82 Destination Host Unreachable
From 10.10.1.1 icmp_seq=83 Destination Host Unreachable
From 10.10.1.1 icmp_seq=84 Destination Host Unreachable
From 10.10.1.1 icmp_seq=85 Destination Host Unreachable
From 10.10.1.1 icmp_seq=86 Destination Host Unreachable
From 10.10.1.1 icmp_seq=87 Destination Host Unreachable
From 10.10.1.1 icmp_seq=88 Destination Host Unreachable
From 10.10.1.1 icmp_seq=89 Destination Host Unreachable
From 10.10.1.1 icmp_seq=90 Destination Host Unreachable
From 10.10.1.1 icmp_seq=91 Destination Host Unreachable
From 10.10.1.1 icmp_seq=92 Destination Host Unreachable
From 10.10.1.1 icmp_seq=93 Destination Host Unreachable
From 10.10.1.1 icmp_seq=94 Destination Host Unreachable
From 10.10.1.1 icmp_seq=95 Destination Host Unreachable
From 10.10.1.1 icmp_seq=96 Destination Host Unreachable
From 10.10.1.1 icmp_seq=97 Destination Host Unreachable
From 10.10.1.1 icmp_seq=98 Destination Host Unreachable
From 10.10.1.1 icmp_seq=99 Destination Host Unreachable
From 10.10.1.1 icmp_seq=100 Destination Host Unreachable
From 10.10.1.1 icmp_seq=101 Destination Host Unreachable
From 10.10.1.1 icmp_seq=102 Destination Host Unreachable
From 10.10.1.1 icmp_seq=103 Destination Host Unreachable
From 10.10.1.1 icmp_seq=104 Destination Host Unreachable
From 10.10.1.1 icmp_seq=105 Destination Host Unreachable
From 10.10.1.1 icmp_seq=106 Destination Host Unreachable
From 10.10.1.1 icmp_seq=107 Destination Host Unreachable
From 10.10.1.1 icmp_seq=108 Destination Host Unreachable
From 10.10.1.1 icmp_seq=109 Destination Host Unreachable
From 10.10.1.1 icmp_seq=110 Destination Host Unreachable
From 10.10.1.1 icmp_seq=111 Destination Host Unreachable
From 10.10.1.1 icmp_seq=112 Destination Host Unreachable
From 10.10.1.1 icmp_seq=113 Destination Host Unreachable
From 10.10.1.1 icmp_seq=114 Destination Host Unreachable
From 10.10.1.1 icmp_seq=115 Destination Host Unreachable
From 10.10.1.1 icmp_seq=116 Destination Host Unreachable
From 10.10.1.1 icmp_seq=117 Destination Host Unreachable
From 10.10.1.1 icmp_seq=118 Destination Host Unreachable
From 10.10.1.1 icmp_seq=119 Destination Host Unreachable
From 10.10.1.1 icmp_seq=120 Destination Host Unreachable
From 10.10.1.1 icmp_seq=121 Destination Host Unreachable
From 10.10.1.1 icmp_seq=122 Destination Host Unreachable
From 10.10.1.1 icmp_seq=123 Destination Host Unreachable
From 10.10.1.1 icmp_seq=124 Destination Host Unreachable
From 10.10.1.1 icmp_seq=125 Destination Host Unreachable
From 10.10.1.1 icmp_seq=126 Destination Host Unreachable
From 10.10.1.1 icmp_seq=127 Destination Host Unreachable
From 10.10.1.1 icmp_seq=128 Destination Host Unreachable
From 10.10.1.1 icmp_seq=129 Destination Host Unreachable
From 10.10.1.1 icmp_seq=130 Destination Host Unreachable
From 10.10.1.1 icmp_seq=131 Destination Host Unreachable
From 10.10.1.1 icmp_seq=132 Destination Host Unreachable
From 10.10.1.1 icmp_seq=133 Destination Host Unreachable
From 10.10.1.1 icmp_seq=134 Destination Host Unreachable
From 10.10.1.1 icmp_seq=135 Destination Host Unreachable
From 10.10.1.1 icmp_seq=137 Destination Host Unreachable
From 10.10.1.1 icmp_seq=138 Destination Host Unreachable
From 10.10.1.1 icmp_seq=139 Destination Host Unreachable
From 10.10.1.1 icmp_seq=140 Destination Host Unreachable
From 10.10.1.1 icmp_seq=141 Destination Host Unreachable
From 10.10.1.1 icmp_seq=142 Destination Host Unreachable
From 10.10.1.1 icmp_seq=143 Destination Host Unreachable
From 10.10.1.1 icmp_seq=144 Destination Host Unreachable
From 10.10.1.1 icmp_seq=145 Destination Host Unreachable
From 10.10.1.1 icmp_seq=146 Destination Host Unreachable
From 10.10.1.1 icmp_seq=147 Destination Host Unreachable
From 10.10.1.1 icmp_seq=148 Destination Host Unreachable
From 10.10.1.1 icmp_seq=149 Destination Host Unreachable
From 10.10.1.1 icmp_seq=150 Destination Host Unreachable
From 10.10.1.1 icmp_seq=151 Destination Host Unreachable
From 10.10.1.1 icmp_seq=152 Destination Host Unreachable
From 10.10.1.1 icmp_seq=153 Destination Host Unreachable
From 10.10.1.1 icmp_seq=154 Destination Host Unreachable
From 10.10.1.1 icmp_seq=155 Destination Host Unreachable
From 10.10.1.1 icmp_seq=156 Destination Host Unreachable
From 10.10.1.1 icmp_seq=157 Destination Host Unreachable
From 10.10.1.1 icmp_seq=158 Destination Host Unreachable
From 10.10.1.1 icmp_seq=159 Destination Host Unreachable
From 10.10.1.1 icmp_seq=160 Destination Host Unreachable
From 10.10.1.1 icmp_seq=161 Destination Host Unreachable
From 10.10.1.1 icmp_seq=162 Destination Host Unreachable
From 10.10.1.1 icmp_seq=163 Destination Host Unreachable
From 10.10.1.1 icmp_seq=164 Destination Host Unreachable
From 10.10.1.1 icmp_seq=165 Destination Host Unreachable
From 10.10.1.1 icmp_seq=166 Destination Host Unreachable
From 10.10.1.1 icmp_seq=167 Destination Host Unreachable
From 10.10.1.1 icmp_seq=168 Destination Host Unreachable
From 10.10.1.1 icmp_seq=169 Destination Host Unreachable
From 10.10.1.1 icmp_seq=170 Destination Host Unreachable
From 10.10.1.1 icmp_seq=171 Destination Host Unreachable
From 10.10.1.1 icmp_seq=172 Destination Host Unreachable
From 10.10.1.1 icmp_seq=173 Destination Host Unreachable
From 10.10.1.1 icmp_seq=174 Destination Host Unreachable
From 10.10.1.1 icmp_seq=175 Destination Host Unreachable
From 10.10.1.1 icmp_seq=176 Destination Host Unreachable
From 10.10.1.1 icmp_seq=177 Destination Host Unreachable
From 10.10.1.1 icmp_seq=178 Destination Host Unreachable
From 10.10.1.1 icmp_seq=179 Destination Host Unreachable
From 10.10.1.1 icmp_seq=180 Destination Host Unreachable
From 10.10.1.1 icmp_seq=181 Destination Host Unreachable
From 10.10.1.1 icmp_seq=182 Destination Host Unreachable
From 10.10.1.1 icmp_seq=183 Destination Host Unreachable
From 10.10.1.1 icmp_seq=184 Destination Host Unreachable
From 10.10.1.1 icmp_seq=185 Destination Host Unreachable
From 10.10.1.1 icmp_seq=186 Destination Host Unreachable
From 10.10.1.1 icmp_seq=187 Destination Host Unreachable
From 10.10.1.1 icmp_seq=188 Destination Host Unreachable
From 10.10.1.1 icmp_seq=189 Destination Host Unreachable
From 10.10.1.1 icmp_seq=190 Destination Host Unreachable
From 10.10.1.1 icmp_seq=191 Destination Host Unreachable
From 10.10.1.1 icmp_seq=192 Destination Host Unreachable
From 10.10.1.1 icmp_seq=193 Destination Host Unreachable
From 10.10.1.1 icmp_seq=194 Destination Host Unreachable
From 10.10.1.1 icmp_seq=195 Destination Host Unreachable
From 10.10.1.1 icmp_seq=196 Destination Host Unreachable
From 10.10.1.1 icmp_seq=197 Destination Host Unreachable
From 10.10.1.1 icmp_seq=198 Destination Host Unreachable
From 10.10.1.1 icmp_seq=199 Destination Host Unreachable
From 10.10.1.1 icmp_seq=200 Destination Host Unreachable
From 10.10.1.1 icmp_seq=201 Destination Host Unreachable
From 10.10.1.1 icmp_seq=202 Destination Host Unreachable
From 10.10.1.1 icmp_seq=203 Destination Host Unreachable
From 10.10.1.1 icmp_seq=204 Destination Host Unreachable
From 10.10.1.1 icmp_seq=205 Destination Host Unreachable
From 10.10.1.1 icmp_seq=206 Destination Host Unreachable
From 10.10.1.1 icmp_seq=207 Destination Host Unreachable
From 10.10.1.1 icmp_seq=208 Destination Host Unreachable
From 10.10.1.1 icmp_seq=209 Destination Host Unreachable
From 10.10.1.1 icmp_seq=210 Destination Host Unreachable
From 10.10.1.1 icmp_seq=211 Destination Host Unreachable
From 10.10.1.1 icmp_seq=212 Destination Host Unreachable
From 10.10.1.1 icmp_seq=213 Destination Host Unreachable
From 10.10.1.1 icmp_seq=214 Destination Host Unreachable
From 10.10.1.1 icmp_seq=215 Destination Host Unreachable
From 10.10.1.1 icmp_seq=216 Destination Host Unreachable
From 10.10.1.1 icmp_seq=217 Destination Host Unreachable
From 10.10.1.1 icmp_seq=218 Destination Host Unreachable
From 10.10.1.1 icmp_seq=219 Destination Host Unreachable
From 10.10.1.1 icmp_seq=220 Destination Host Unreachable
From 10.10.1.1 icmp_seq=221 Destination Host Unreachable
From 10.10.1.1 icmp_seq=222 Destination Host Unreachable
From 10.10.1.1 icmp_seq=223 Destination Host Unreachable
From 10.10.1.1 icmp_seq=224 Destination Host Unreachable
From 10.10.1.1 icmp_seq=225 Destination Host Unreachable
From 10.10.1.1 icmp_seq=226 Destination Host Unreachable
From 10.10.1.1 icmp_seq=227 Destination Host Unreachable
From 10.10.1.1 icmp_seq=228 Destination Host Unreachable
From 10.10.1.1 icmp_seq=229 Destination Host Unreachable
From 10.10.1.1 icmp_seq=230 Destination Host Unreachable
From 10.10.1.1 icmp_seq=231 Destination Host Unreachable
From 10.10.1.1 icmp_seq=232 Destination Host Unreachable
From 10.10.1.1 icmp_seq=233 Destination Host Unreachable
From 10.10.1.1 icmp_seq=234 Destination Host Unreachable
From 10.10.1.1 icmp_seq=235 Destination Host Unreachable
From 10.10.1.1 icmp_seq=236 Destination Host Unreachable
From 10.10.1.1 icmp_seq=237 Destination Host Unreachable
From 10.10.1.1 icmp_seq=238 Destination Host Unreachable
From 10.10.1.1 icmp_seq=239 Destination Host Unreachable
From 10.10.1.1 icmp_seq=240 Destination Host Unreachable
From 10.10.1.1 icmp_seq=241 Destination Host Unreachable
From 10.10.1.1 icmp_seq=242 Destination Host Unreachable
From 10.10.1.1 icmp_seq=243 Destination Host Unreachable
From 10.10.1.1 icmp_seq=244 Destination Host Unreachable
From 10.10.1.1 icmp_seq=245 Destination Host Unreachable
From 10.10.1.1 icmp_seq=246 Destination Host Unreachable
From 10.10.1.1 icmp_seq=247 Destination Host Unreachable
From 10.10.1.1 icmp_seq=248 Destination Host Unreachable
From 10.10.1.1 icmp_seq=249 Destination Host Unreachable
From 10.10.1.1 icmp_seq=250 Destination Host Unreachable
From 10.10.1.1 icmp_seq=251 Destination Host Unreachable
From 10.10.1.1 icmp_seq=252 Destination Host Unreachable
From 10.10.1.1 icmp_seq=253 Destination Host Unreachable
From 10.10.1.1 icmp_seq=254 Destination Host Unreachable
From 10.10.1.1 icmp_seq=255 Destination Host Unreachable
From 10.10.1.1 icmp_seq=256 Destination Host Unreachable
From 10.10.1.1 icmp_seq=257 Destination Host Unreachable
From 10.10.1.1 icmp_seq=258 Destination Host Unreachable
From 10.10.1.1 icmp_seq=259 Destination Host Unreachable
From 10.10.1.1 icmp_seq=260 Destination Host Unreachable
From 10.10.1.1 icmp_seq=261 Destination Host Unreachable
From 10.10.1.1 icmp_seq=262 Destination Host Unreachable
From 10.10.1.1 icmp_seq=263 Destination Host Unreachable
From 10.10.1.1 icmp_seq=264 Destination Host Unreachable
From 10.10.1.1 icmp_seq=265 Destination Host Unreachable
From 10.10.1.1 icmp_seq=266 Destination Host Unreachable
From 10.10.1.1 icmp_seq=267 Destination Host Unreachable
From 10.10.1.1 icmp_seq=268 Destination Host Unreachable
From 10.10.1.1 icmp_seq=269 Destination Host Unreachable
From 10.10.1.1 icmp_seq=270 Destination Host Unreachable
From 10.10.1.1 icmp_seq=271 Destination Host Unreachable
From 10.10.1.1 icmp_seq=272 Destination Host Unreachable
From 10.10.1.1 icmp_seq=273 Destination Host Unreachable
From 10.10.1.1 icmp_seq=274 Destination Host Unreachable
From 10.10.1.1 icmp_seq=275 Destination Host Unreachable
From 10.10.1.1 icmp_seq=276 Destination Host Unreachable
From 10.10.1.1 icmp_seq=277 Destination Host Unreachable
From 10.10.1.1 icmp_seq=278 Destination Host Unreachable
From 10.10.1.1 icmp_seq=279 Destination Host Unreachable
From 10.10.1.1 icmp_seq=280 Destination Host Unreachable
From 10.10.1.1 icmp_seq=281 Destination Host Unreachable
From 10.10.1.1 icmp_seq=282 Destination Host Unreachable
From 10.10.1.1 icmp_seq=283 Destination Host Unreachable
From 10.10.1.1 icmp_seq=284 Destination Host Unreachable
From 10.10.1.1 icmp_seq=285 Destination Host Unreachable
From 10.10.1.1 icmp_seq=286 Destination Host Unreachable
From 10.10.1.1 icmp_seq=287 Destination Host Unreachable
From 10.10.1.1 icmp_seq=288 Destination Host Unreachable
From 10.10.1.1 icmp_seq=289 Destination Host Unreachable
From 10.10.1.1 icmp_seq=290 Destination Host Unreachable
From 10.10.1.1 icmp_seq=291 Destination Host Unreachable
From 10.10.1.1 icmp_seq=292 Destination Host Unreachable
From 10.10.1.1 icmp_seq=293 Destination Host Unreachable
From 10.10.1.1 icmp_seq=294 Destination Host Unreachable
From 10.10.1.1 icmp_seq=295 Destination Host Unreachable
From 10.10.1.1 icmp_seq=296 Destination Host Unreachable
From 10.10.1.1 icmp_seq=297 Destination Host Unreachable
From 10.10.1.1 icmp_seq=298 Destination Host Unreachable
From 10.10.1.1 icmp_seq=299 Destination Host Unreachable
From 10.10.1.1 icmp_seq=300 Destination Host Unreachable
From 10.10.1.1 icmp_seq=301 Destination Host Unreachable
From 10.10.1.1 icmp_seq=302 Destination Host Unreachable
From 10.10.1.1 icmp_seq=303 Destination Host Unreachable
From 10.10.1.1 icmp_seq=304 Destination Host Unreachable
From 10.10.1.1 icmp_seq=305 Destination Host Unreachable
From 10.10.1.1 icmp_seq=306 Destination Host Unreachable
From 10.10.1.1 icmp_seq=307 Destination Host Unreachable
From 10.10.1.1 icmp_seq=308 Destination Host Unreachable
From 10.10.1.1 icmp_seq=309 Destination Host Unreachable
From 10.10.1.1 icmp_seq=310 Destination Host Unreachable
From 10.10.1.1 icmp_seq=311 Destination Host Unreachable
From 10.10.1.1 icmp_seq=312 Destination Host Unreachable
From 10.10.1.1 icmp_seq=313 Destination Host Unreachable
From 10.10.1.1 icmp_seq=314 Destination Host Unreachable
From 10.10.1.1 icmp_seq=315 Destination Host Unreachable
From 10.10.1.1 icmp_seq=316 Destination Host Unreachable
From 10.10.1.1 icmp_seq=317 Destination Host Unreachable
From 10.10.1.1 icmp_seq=318 Destination Host Unreachable
From 10.10.1.1 icmp_seq=319 Destination Host Unreachable
From 10.10.1.1 icmp_seq=320 Destination Host Unreachable
From 10.10.1.1 icmp_seq=321 Destination Host Unreachable
From 10.10.1.1 icmp_seq=322 Destination Host Unreachable
From 10.10.1.1 icmp_seq=323 Destination Host Unreachable
From 10.10.1.1 icmp_seq=324 Destination Host Unreachable
From 10.10.1.1 icmp_seq=325 Destination Host Unreachable
From 10.10.1.1 icmp_seq=326 Destination Host Unreachable
From 10.10.1.1 icmp_seq=327 Destination Host Unreachable
From 10.10.1.1 icmp_seq=328 Destination Host Unreachable
From 10.10.1.1 icmp_seq=329 Destination Host Unreachable
From 10.10.1.1 icmp_seq=330 Destination Host Unreachable
From 10.10.1.1 icmp_seq=331 Destination Host Unreachable
From 10.10.1.1 icmp_seq=332 Destination Host Unreachable
From 10.10.1.1 icmp_seq=333 Destination Host Unreachable
From 10.10.1.1 icmp_seq=334 Destination Host Unreachable
From 10.10.1.1 icmp_seq=335 Destination Host Unreachable
64 bytes from 10.10.4.10: icmp_seq=336 ttl=61 time=1.34 ms
64 bytes from 10.10.4.10: icmp_seq=337 ttl=61 time=1.61 ms
64 bytes from 10.10.4.10: icmp_seq=338 ttl=61 time=1.52 ms
We can see that requests 1 to 36 received ICMP Echo Reply responses from the Target machine for about 3.6s, then ICMP Destination Host Unreachable responses from the ProviderEdge1 router for about 29.9s. Finally, responses from the Target machine were received again. RTBH thus automatically blocked the attack attempt, and the diagram below summarizes what happened at all levels.

RTBH procedure for the attack carried out in the test environment

The attack was detected within a reasonable and expected timeframe, and the blocking lasted almost 30s instead of the 10s configured on FastNetMon. This is due to the internal reactivity time of the software and the propagation time of the BGP RTBH announcements. It is interesting to check what is happening on each piece of equipment at the time of the attack.

On FastNetMon, the detection of the attack and the RTBH reaction can be seen in the logs located in the /etc/fastnetmon.log file:

2024-05-16 14:39:37,431 [INFO] We run execute_ip_ban code with following params  in_pps: 9 out_pps: 9 in_bps: 842 out_bps: 842 and we decide it's o
utgoing attack
2024-05-16 14:39:37,432 [INFO] Attack with direction: outgoing IP: 10.10.4.10 Power: 9
2024-05-16 14:39:37,432 [ERROR] Can't print attack details to file
2024-05-16 14:39:37,432 [INFO] Call GoBGP for ban client started: 10.10.4.10
2024-05-16 14:39:37,432 [INFO] Call to GoBGP for ban client is finished: 10.10.4.10
2024-05-16 14:39:37,433 [INFO] announce 10.10.4.10/32 to GoBGP
2024-05-16 14:39:39,483 [INFO] Attack with direction: outgoing IP: 10.10.4.10 Power: 9 traffic samples collected
2024-05-16 14:39:39,484 [ERROR] Can't print attack details to file
2024-05-16 14:39:52,934 [INFO] We will unban banned IP: 10.10.4.10 because it ban time 10 seconds is ended
2024-05-16 14:39:52,934 [INFO] Call GoBGP for unban client started: 10.10.4.10
2024-05-16 14:39:52,934 [INFO] Call to GoBGP for unban client is finished: 10.10.4.10
2024-05-16 14:39:52,934 [INFO] withdraw 10.10.4.10/32 to GoBGP
The attack is detected here after 9 packets per second being observed, GoBGP is called to block the source of the attack via RTBH (sending a specific BGP announcement) and after about ten seconds, GoBGP is called again to cancel the blocking (sending a new BGP announcement).
We can see that during the attack, a route was added on FastNetMon to 10.10.4.10/32 with community 65000:666 in the IBGP routing table:
$ gobgp global rib
   Network              Next Hop             AS_PATH              Age        Attrs
*> 10.10.1.0/24         10.10.3.1            65000                1d 21:33:44 [{Origin: i} {Med: 0} {LocalPref: 100}]
*> 10.10.4.0/24         192.168.10.1                              1d 21:33:44 [{Origin: i} {Med: 0} {LocalPref: 100}]
*> 10.10.4.10/32        0.0.0.0                                   00:00:15   [{Origin: i} {Communities: 65000:666}]

On the CustomerEdge1 router, we can see that a prefix for 10.10.4.10/32 (the Target machine) was received from 192.168.10.10 (the FastNetMon machine) with a next-hop to the Target machine (seen as local because the next-hop sent was 0.0.0.0) and with the 65000:666 community.

CE1#show ip bgp 10.10.4.10
BGP routing table entry for 10.10.4.10/32, version 411
Paths: (1 available, best #1, table default)
  Advertised to update-groups:
     6         
  Refresh Epoch 1
  Local
    192.168.10.10 from 192.168.10.10 (192.168.10.10)
      Origin IGP, localpref 100, valid, internal, best
      Community: 65000:666
      rx pathid: 0, tx pathid: 0x0
      Updated on May 16 2024 12:39:37 UTC

Note that thanks to the configuration we have put in place, we could have installed the prefix for 10.10.4.10/32 manually on CustomerEdge1:
CE1(config)#ip route 10.10.4.10 255.255.255.255 Null0 tag 666
This way, the 666 tag would be recognised and the 65000:666 community would be propagated in the same way. Obviously, this isn’t very useful when you have a detection and response system like FastNetMon and GoBGP, but I wanted to show you how to do it manually anyway.

On the ProviderEdge2 router, we can see that a prefix for 10.10.4.10/32 (the Target machine) has been received from 10.10.2.2 (itself) with a next-hop to 192.0.2.1 and with the community 65000:666. The route to 192.0.2.1 points to a non-existent interface (Null0).

PE2#show ip bgp 10.10.4.10
BGP routing table entry for 10.10.4.10/32, version 402
Paths: (1 available, best #1, table default)
  Advertised to update-groups:
     6         
  Refresh Epoch 29
  65001
    192.0.2.1 from 10.10.3.2 (3.3.3.3)
      Origin IGP, localpref 100, valid, external, best
      Community: 65000:666
      rx pathid: 0, tx pathid: 0x0
      Updated on May 16 2024 12:39:36 UTC
PE2#sh ip route 192.0.2.1
Routing entry for 192.0.2.1/32
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1
On the ProviderEdge1 router, we can also see that a prefix for 10.10.4.10/32 (the Target machine) has been received from 10.10.2.2 (ProviderEdge2) with a next-hop to 192.0.2.1. The route to 192.0.2.1 again points to a non-existent interface (Null0).
PE1#show ip bgp 10.10.4.10
BGP routing table entry for 10.10.4.10/32, version 224
Paths: (1 available, best #1, table default, not advertised to EBGP peer)
  Not advertised to any peer
  Refresh Epoch 5
  65001
    192.0.2.1 from 10.10.2.2 (10.10.3.1)
      Origin IGP, metric 0, localpref 100, valid, internal, best
      Community: no-export
      rx pathid: 0, tx pathid: 0x0
      Updated on May 16 2024 12:39:36 UTC
PE1#sh ip route 192.0.2.1
Routing entry for 192.0.2.1/32
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1
The ProviderEdge1 router will therefore be the first router in the path to block the flow and respond with an ICMP Destination Unreachable packet.

Conclusion

Distributed denial-of-service attacks can be devastating for an organisation. Fortunately, response mechanisms exist such as RTBH which uses the power of the BGP protocol to propagate routing announcements stopping the transmission of malicious traffic. When this is coupled with a detection solution supported by flow collection such as NetFlow, the response can be even faster and more accurate. I also urge you to read this excellent article written by a colleague who is exploring RTBH with sFlow and FlowSpec. Incidentally, I would like to extend my warmest thanks to the colleagues who provided me with technical support in setting up the test environment.

Sources

Translations