RPKI ROV Sickle in Full Swing

The month of June saw a spike in Resource Public Key Infrastructure (RPKI) and BGP Route Origin Validation (ROV) activity. This compelled me to curate some of the events that caught my eye. By increased activity, I'm referring to both increased incidents of routing security improvement and, social occasions focused on promoting RPKI and raising awareness. However, this blogpost is about the latter. The purpose began as drafting a personal summary of all the compressed RPKI knowledge consumed. It's now evolved into what I think could be useful for those that are new to the concept and may have missed some of these important resources that are shared freely on the internet.

TL;DR

You can skip reading the rest of this article and dig straight into the following fantastic four:

• RPKI Zero To Hero Live Demo a.k.a Implementing RPKI: It's easier than you think

• Mikrotik announces RouterOSv7 Beta8 with RPKI support and someone takes it for a ride

• ICOMM2020 - Securing The Routing System with MANRS

• Measuring Route Origin Validation

Since the break out of the COVID-19 global pandemic, it had been a while since techies that help keep the beast that is the global internet breathing, came together to share knowledge of this magnitude. On the 19th of June, some of these champions live-streamed a demonstration of how quick and easy it can be to set up the different components of an RPKI-fluent network. The line up represented a balanced mix of vendors and organizations that play significant roles in the make-up of the internet. This included Apstra, Arista, Cisco, Cloudflare, Juniper Networks, NLnet Labs, Nokia, Orhan Ergun LLC and RIPE NCC.

Moderated by Orhan Ergun and Jeff Tantsura, the Zoom session was also live streamed via YouTube and Facebook - a move that should be celebrated for making the live session more accessible to the global community. I personally experienced challenges joining the Zoom session so this proved handy.

With a goal of achieving the following in 2 hours, the session turned out much shorter than expected:

• Creating Route Origin Authorizations (ROAs) via your Regional Internet Registry (RIR) member portal
• Installing a validator in your network
• Configuring your router to enable RPKI and perform BGP ROV
  

Nathalie Trenaman from RIPE NCC kicked off with creating a ROA via their member portal. She also shared some best practices, like how you should be conservative with maxLength of your ROAs, all within 5 minutes. Her introduction and background about RPKI (which I think was planned to be presented by Jeff Tantsura) actually took longer than creating a ROA. Assuming that most networks are familiar with RPKI terminology and a bit of its theory, the excuse of not having time to create ROA(s) was proved to be invalid. 🧀🧀🧀

Validator installation demos included RIPE NCC's RPKI Validator, routinator 3000 from NLNetlabs and Cloudflare's OctoRPKI. Yes, no FORT. Another representative from RIPE NCC, Ties de Kock demonstrated the RIPE Validator and RTR Server installations. After installation of the 2 packages, making some necessary configuration adjustments and starting up the services which took less than 5 minutes, he explained that the validator can take about 15 minutes to be ready with data downloaded from the different repositories.

Alex Band then took the reigns to demonstrate a routinator installation. Unlike RIPE Validator, there are no packages to download for installation and routinator is built from source code - a conscious decision they made because of its frequent releases. To set up the environment for routinator, you need to install rsync, the C toolchain and Rust (the programming language that routinator is written in). While waiting for the routinator installation to complete, Alex spoke about their transition from using rsync to https for rrdp. The final step is to start the application as an RTR server and wait about 10 minutes for data to be downloaded from the difference repositories. The demonstration took roughly 7 minutes.

Loius Poinsignon from Clouflare then demonstrated OctoRPKI. Similar to the one from RIPE NCC, the validator and RTR server are separate packages in the form of OctoRPKI and GoRTR respectively. Loius explained that this has the advantage of having GoRTR running on a different machine closer to your routers while the validator is hosted elsewhere safe. His demonstration was also within 7 minutes.

Enabling Routers to Speak RPKI

In what seemed like a friendly tournament of the router vendors, there were demonstrations from Juniper Networks by Melchior Aelmans, Nokia's Greg Hankins, Florian Hibler from Arista, and Cisco's duo Vinay Shankarkumar and Jakob Heitz. Setting up their routers to a point of dropping Invalids and showing off some troubleshooting commands took less than 10 minutes each. I've summarized how long each activity took below. Of course a beginner would take slightly longer than that because they'd have to RTFM first, but these numbers still highlight how smooth the process has become. This in no way trivializes the process.

Creating ROA	5 minutes	Installing Routinator Validator	7 minutes
Installing RIPE Validator	5 minutes	Installing OctoRPKI Validator	7 minutes
Configuring JUNOS for ROV	5 minutes	Configuring Arista EOS for ROV	8 minutes
*Configuring Cisco IOS XR for ROV	16minutes	Configuring Nokia SR OS for ROV	5 minutes

* The Cisco session had Q&A during the demonstration unlike the others who held it off until their configurations were completed

The total amount of time reflected in the table is only about an hour of a 1-hour-45-minute session. There are many gems of Q&A throughout the session including an interesting wrap up. I strongly encourage a watch of the entire video.

Mikrotik roller coaster

It was a pity that there wasn’t a Mikrotik Zero To Hero demonstration given that earlier in the month the industry was excited about the announcement that RouterOS had a beta release of an RPKI ROV ready image and Massimiliano Stucchi, under his personal ASN AS58280 had taken it for ride a few days before the RIPE event. These were big news for the industry given that Mikrotik has a large footprint in small and startup networks, especially in developing countries and continents.

Unfortunately, this Mikrotik party was crushed in another RPKI studded session held a week later. Enter "InterCommunity: Securing Global Routing" featuring Melchior Aelmans (this time wearing a moderator mask), Abdul Awal of the Bangladesh National DataCentre, Mark Tinka of SEACOM, Kevin Blumberg from TORIX, Jorge Cano from NIC.mx, and Tashi Phuntsho from APNIC. It was during Tashi's slot that it was revealed that the RPKI implementation on RouterOSv7.0beta8 is broken, by pointing us to a Mikrotik Forum discussion where his team raised the issues experienced and Mikrotik also confirmed the bug.

This Internet Society (ISOC) #ICOMM2020 event involved the panelists sharing their experiences from their various contributions in making global routing more secure. Awal's presentation on his project RPKI Deployment in South Asia was a good place to start and he dropped many gems on how to mobilize RPKI adoption in your region. He also published a write-up on the same topic in June.

Mark Tinka (Head of Engineering at Seacom) took us back to when they tried to implement RPKI in 2014 only to discover bugs in Cisco IOS XE and that not many networks had deployed ROV. Dropping invalids while your competitors allow such traffic through can put you at a business disadvantage. Together with another IP Transit network provider in the same region, Workonline Communications, they went live with RPKI in April 2019. He found that IOS XR and JUNOS worked well while IOS XE was still buggy. This was supported by Kevin Blumberg (President of the Toronto Internet Exchange) who shared their history of ticking all the boxes of the MANRS actions as the largest IXP in Canada. Echoing what was shared by many in the Zero To Hero event, Kevin runs 2 RPKI validator servers in parallel and wants to see how they differentiate for the foreseeable future.

Representing the open source software development efforts of NIC Mexico, Jorge Cano took us through an introduction to FORT Zero To Hero gap-filler which is their most recent contribution to the community. Again the running of 2 or more validators in your network was stressed.

Similar to Mark Tinka, Tashi Phuntsho took us back to January 2014 where they started to implement RPKI ROV using IOS XE and things broke spectacularly to the point of upsetting the local king who happened to be their customer 😂. Tashi stressed the need to test your validators by showing us an example of the asymmetry between Validated ROA Payloads (VRPs), extracted a couple of hours before the webinar, from FORT and Routinator. Please watch the video and see how significant the difference can be. He also shares some of the brilliant outreach work they've been doing.

It's worth mentioning that Tashi was also involved in an online APNIC webinar (Securing Internet Routing Tutorial) on the same topic earlier in the month. I personally didn't attend it but I'm sure it was useful given the topics listed in the agenda. If you have a link to the actual content and have permission to distribute it please leave it in the comments below and I'll have it added here.

Short Break (not an advertisement/sponsored content):

There was also a major update to Krill. Check out Krill Gains Powerful ROA Management Based on BGP Routing by Alex Band.

I also need to make mention of this Excuse Me, Your BGP Is Leaking episode by the #theInternetReport which has some interesting global routing security incidents and RPKI news for June.

Measuring Route Origin Validation

Last and certainly not least, there's a brilliant stock-take of how far we've come with ROV by the internet veteran and APNIC Chief Scientist Geoff Huston. This new way of measuring ROV is where I should respectively end this post. Geoff Huston and Joao Damas challenge you to think deeper about the numbers we often see being reported and the implications of dropping Invalids from an end-user perspective. It's definitely worth your time. Also look out for the surprising findings about Africa in his article.

Bonus: You can couple Geoff's blog post with a him being interviewed by Mehmet Akcin in the same month where he drops some fascinating insights.

blogger-node#show post summary

More awareness could have been raised in my continent/region about the Zero to Hero RIPE event. There was an alert shared about it on the ZANOG mailing list but I don't think there was anything beyond that. The session covered all the basics and, with an AFRINIC ROA creation example perhaps, it would be a great addition to the workshops that are being run in promotion of RPKI by various organizations in the region. If you're from my region, you had to have been on the RIPE mailing lists or eagerly following feeds about RPKI on social media to know about it.

The #ICOMM2020 event was well marketed in my region. I'm guessing this is because the organizing team consisted of ISOC representatives from Africa. It would be great to see Mikrotik release a working version of their code (one that also doesn't break IPv6) very soon. Overall, I wish that the momentum gained in the past month is not just an annual climax and hope that it will be kept up until that all-invalids-dropped day.

May your Valids live long and prosper!