Cleaning Up RPKI Invalids In The AFRINIC Service Region

Introduction

With the growing adoption of RPKI based ROV to make the internet more secure and resilient, it is becoming increasingly important for IP network operators or IP prefix owners to ensure that their BGP advertisements are not seen as RPKI Invalid on the internet. Recent years have seen a significant increase in IP Transit providers (across tiers) and IXPs, implementing ROV and filtering Invalid prefix-origin AS pairs.

This article describes an initiative taken to alert and propose corrective measures to network operators with RPKI Invalid prefix-origin pairs in the AFRINIC service region.

Background

Although the concept dates back a decade, in the past 3 years Resource Public Key Infrastructure (RPKI) based Route Origin Validation (ROV) (RFC7115) has been on the lips of every network operator that cares about the resilience of the internet. Many Tier1 and Tier2 networks have implemented ROV fully enabled by dropping RPKI Invalid prefixes.

A prefix received via Border Gateway Protocol (BGP) can either be RPKI Valid, NotFound, or Invalid (RFC6811). RPKI Invalid prefix-origin pairs can occur due to a BGP "route leak" (RFC7908) or incorrect creation of Route Origin Authorizations (ROAs). These are in many cases due to a "fat finger", design oversight or not fully understanding the implications of certain ROA configurations.

At the time of implementing this initiative, there were just above 100 RPKI Invalid prefixes (IPv4) being announced that either belong to AFRINIC Autonomous System Numbers (ASNs) and are announced by non-AFRINIC ASNs, OR are announced incorrectly by the rightful AFRINIC ASN. The latter case occurs when a prefix is deemed RPKI Invalid due to the MaxLength property of a ROA - A network owner may have created a ROA with a MaxLength that is less than the length of the prefix being advertised. This is usually an internal mistake that the operator may not be aware of.

A network operator may advertise a prefix with different lengths to different upstream providers or BGP peers based on a routing policy that best suites their business. Having the more specific prefix (longer length) being dropped by networks with RPKI based ROV enabled, means that the intended routing behavior becomes sub-optimal.

For example, if a network operator has multiple paths for a prefix, they could advertise the longest length (eg A.B.C.D/24) to the preferred shorter and cheaper path, and advertise a less specific prefix length to the secondary path (eg A.B.C.D/16). If the network operator has a ROA with MaxLength of 16 for example, the A.B.C.D/24 advertisement would be RPKI Invalid due to MaxLength. Upstream network operators who receive both prefixes from different sources, but have RPKI based ROV fully enabled, would therefore only use the secondary path. This is an oversimplified example but I hope it demonstrates the unintended cost implications for the network operator and/or the compromised end-user experience that can arise from this oversight.

There is therefore some value to be gained in alerting network operators of instances of their RPKI Invalid prefix-origin pairs and, cleaning them up is good for the general health of the internet.

Methodology

The procedure followed to implement this initiative can be broken down to 4 simple steps:

• Collecting a list of AFRINIC RPKI Invalids
  
• Gathering contact details of ASNs involved
  
• Contacting the network operators with diagnosis and suggested remedies
  
• Monitoring operator feedback or changes to status

Collecting a List of RPKI Invalids for AFRINIC Service Region

There are a number of platforms that one can use to analyse RPKI statistics depending on specific research requirements. NLnet Labs maintains a useful list on their RPKI readthedocs Resources page. My case required a tool that would allow me to filter a list of RPKI Invalid prefix-origin pairs by Regional Internet Registry (RIR). My closest match was the US National Institute of Standards and Technology (NIST) RPKI Monitor.

Figure1 : Screenshot of parts of the NIST RPKI Monitoring Tool Menu with "Invalid Prefix-Origin Pairs"

When I initially wanted to start this project, I found that the NIST tool's feature for filtering by RIR was buggy. After a number of calls for help on various social media platforms such as Twitter and the RPKI Community on Discord, someone at NIST must have heard my cry and fixed it as part of release version 2 of the tool in May 2021.

Glad to see that NIST fixed this bug... shaves off some time from my after-hours project this month to get some of Africa's RPKI Invalids cleaned up. https://t.co/OKUxuWHIgu

— Peter Peele (@peterpeele) June 2, 2021

The NIST data used in this initiative was collected on and for the 22nd of June 2021. A bonus match of my requirements from the NIST Monitoring tool, is that its analysis features include the ability to expand an RPKI Invalid Prefix-Origin Pair to show the covering prefixes (Figure2 below). That saved me some digging time, but a sanity check I did was to confirm the existence of the associated ROA through another web-based RPKI Validator like AFRINIC's deployment of the Routinator Validator.

Figure2: A Detailed View of The Analysis Feature of Each RPKI Invalid Prefix-Origin Pair on the NIST RPKI Monitoring Tool

Further details of each involved ASN and peering relationships were extracted from bgp.he.net. It's also important to note that the above exercise was only performed for IPv4.

Collecting Network Operator Contact Details

An indirect achievement of this initiative was being able to verify whether the network operator's PeeringDB records included a working e-mail address. This speaks to Action3 of the Mutually Agreed Norms for Routing Security (MANRS) for Network Operators, which requires an operator at minimum to have up-to-date contact information on PeeringDB.

In a case where a network operator had no contact information on PeeringDB, I scraped through the operator's IRR objects (RFC2650).

Contacting Network Operators

With a diagnosis of the problem and some options for the operator to consider to remedy the problem, I contacted each operator via e-mail. The following is an example of an email I sent to an operator whose RPKI Invalid was due to MaxLength:

 

Figure3: A screenshot of a typical e-mail sent to relevant network operators. Specific details have been removed to protect the reputation of the network operators involved.

Monitoring

Monitoring just included regularly checking the RPKI status on NIST Monitoring tool for these recorded prefixes.

Results

As mentioned earlier, there were 107 AFRINIC RPKI Invalid prefix-origin pairs when the data was initially collected. 39 of the pairs were due to the ROA MaxLength. These 39 were deemed low-hanging fruit as they required the least complicated solutions. In many of the cases an ASN would have multiple pairs of these RPKI Invalid prefix-origin and therefore needed to be contacted once for multiple prefixes, significantly reducing the amount of work for the 39.

There were instances where a network operator that owns multiple ASNs has the complexity of advertising different subnets from different ASNs but having created a single ROA covering the different subnets. As you can guess, these are some large networks where organizational complexity can also influence such messy situations. I wasn't holding my breath for a response on this one, but I proposed that they undo the quick&easy shortcut of defining a ROA with MaxLength 24. This MaxLength=24 method is common because it requires less planning and maintenance, however it can come back to bite you when your network design gets sticky as in this case and it also makes you vulnerable to ASN spoofing attacks. 

There were some networks without PeeringDB records. Through IRR records, I could gather some e-mail addresses and also asked the operators to update their PeeringDB contact details in addition to correcting their RPKI data.

Only 10 networks were contacted. Some networks were just not worth contacting in the rest of the list - From studying the ASNs involved, you could detect cases of badly managed IP brokering or undesirable dealings between ASNs that could stray me from the objective of this initiative. Perhaps an investigation and post for the day I can afford bodyguards...

Out of the 10 networks contacted, only 2 had responded to my e-mail at the time of writing this article. One of the messages was in good faith and was from someone who also cares about the health of the internet. Both networks promised to address the issues raised.

Update(1 July): The 2 network operators who responded to my e-mail had fixed their ROAs. In addition to that, 20 out of 25 RPKI Invalid prefixes belonging to one ASN had been fixed even though the network operator never responded to my e-mail. I sent them a "Thank You" e-mail anyway.

Update(5 July): An additional network operator heeded my call and fixed 1 out of the 5 RPKI Invalid prefixes I had asked them to look into.

Discussion and Conclusion

The low count of RPKI Invalid prefix-origin pairs found in this exercise should not be celebrated as a sign of cleanliness in the region. RPKI adoption in the AFRINIC service region is still fairly low. For example, RPKI NotFound makes up 89% of the total unique prefix-origin pairs in the AFRINIC region compared to the global 68.9% for IPv4.

Figure4: AFRINIC Prefix-Origin Pairs; source NIST RPKI Monitor

This exercise assisted to verify that network operators in the AFRINIC region who have implemented Action4 of the recommended MANRS actions, have done so in a way that does not harm their business.

Beyond this initiative, I think regularly checking the status of prefix-origin pairs in your region is necessary to further promote the adoption of RPKI based ROV. It ensures that Network Operators don't have self-inflicted business harming configurations in their network and end up joining the RPKI dark side.

Ideally, this should be a regular exercise done by upstream network operators who have ROV enabled to ensure that their downstream customer networks don't have any RPKI Invalid Prefix-origin pairs. I'm guessing that if an operator is contacted by another operator, they'd heed the call better than when contacted by some random internet individual.

Why The Shumiphone Is Not Widely Celebrated By Black South African Techies

 

First of all, yeah I'm a hater. First of all, yeah I'm anti-black-progress. First of all, yeah I'm about to commit some ICT black-on-black crime!

 

Now that we've got all the first-of-alls out of the way, I'm writing this as a Computer Engineer who genuinely cares about real growth and advancement of technology in Africa. I'm also highly sensitive to what progress we celebrate and the associated songs of praise.

What's with all the politicking? Let's start with these tweets:

Meet 26 year Lesley Ncube founder of Thatha, a tech company based in KZN. Last month he graduated with a degree in Computer Science and he already created his owned smartphone called Shumi. He designs the software and hardware by himself. pic.twitter.com/Bx4ofd3kPQ

— Tumelo Warona (@DjNewAfrica) July 27, 2020

Meet 26 year ol Lesley Ncube. Founder of Thatha, a tech company based in KZN. Last month he graduated with a degree in Computer Science and he already created his owned smartphone called Shumi!

We are young and we are in business! pic.twitter.com/Dr0Uq60wVq

— Youth in Business South Africa (@YouthinBizSA) August 10, 2020

 

When you qualify as a black ICT professional, you're immediately elevated to IT Guru/Oracle among family and friends in the black community. With this promotion, comes the responsibility of being the Go-To Expert for all things technology, including kettles that stop working in the dead of winter. This also carries weight when there are any news about African technological advancement.

 

We've all been sent an article about an African child that invented their own helicopter in their backyard and asked why we're not celebrating it enough, if at all. It's great that people fact check, but the why is often from a position of "this is fact and why are you being a hater" rather than "is there anything missing here"?

 

In the most recent case, a number of highly influential media houses published articles spreading inaccuracies about a phone being developed in South Africa and left our mailboxes in shambles. I even saw it posted in a social media group of ICT professionals. These media houses include, amongst others:

 

• DRUM (news24) (in my opinion the biggest culprit as a lot of social media posts, blogs and articles reference them)

• The South African (Subtitled - Lesley Ncube graduated in June and already has created his own smartphone and a software and computer hardware company. 😒)

 

• Briefly.co.za

 

It even managed to convince one of the Executive Producers at the South African Broadcasting Corporation and made it to both local television and radio stations:

 

So here's the Shumi smartphone (image saved from Thatha's website): 

It's going for R4k (at the time of posting). It looks good. I wouldn't hesitate to promote it in the name of supporting local business (as I've already done here).

So what's your problem? Lies detected:

 

Yes, the man is selling his own smartphone BUT

 

A number of the articles say he developed the hardware and software himself. I have a number of friends and associates who started their own smartphone startups locally. They're all essentially self-branded off-the-shelf phones. None of them however sell a story of having developed the hardware and software themselves.

Assembled the hardware here might have been more convincing...

On his Thatha website, Lesley Ncube lists the CPU used in the phone as:

MTK6761 Cotex-A53 Quad Core CPU

 

Why then does he publicly claim that he developed the hardware himself to those who typically don't read beyond tweets and videos?

I can point you to many smartphones in the continent and beyond that use this CPU and don't claim to have developed it themselves. There isn't even a need to go that deep into the hardware. Just looking at a few low-cost phones with a similar design should get an investigative mind ticking. I don't understand how this slipped through so many editorial offices made up of employees who get paid to do this stuff.  

This is the Vernee M7. I didn't have to Google-dig to find it here.

How about the CUBOT P30 below. Look familiar yet?

"With Thatha, I decided to go the traditional route by starting with developing hardware, then moving on to software. That’s when we developed the Shumi smartphone."

He didn't at least say that he just assembled the pieces locally. At his young age, having established a plant/workshop to assemble components here would have been a major achievement. A lot of such products get assembled outside the continent. The Maraphone and AfriOne are some of the very few local smartphones that can boast this manufacturing capability. Sadly, if you look under the bonnet for both, you'll discover that they're not really African companies.

Developed own software?

The phone is running on Android. I'll leave open the possibility that there may be some customization on top of Android and therefore we can say that the software was locally "developed". Actually, fuck that, the software was NOT developed here! These wild claims are mildly upsetting so let's not dissect the phone any further.

 

Straight outta varsity?

For someone to make such wildly inaccurate claims made me suspicious about their qualifications so I did a bit of searching in that area. The articles and tweets keep promoting this phenomenon as someone who just recently graduated from Tshwane University of Technology (TUT).

The TUT student newspaper, BUA, published an article about this recent graduate just this month (August 2020), stating that he's currently studying for an Advanced Diploma in Computer Science at TUT so I decided not to search any further. Assuming that they at least did some confirmation that he's really one of them.

What caught my eye out of this exercise though, was that TUT seems to have published an article about this smartphone earlier and later removed the post.

Screenshot from Google Search Results

Why would the institution take down the article? Did someone advise them of the inaccuracies of this story and so they chose to distance themselves as an institution?

 

Screenshot of page after clicking link on Google Search Results

I'm willing to bet that some of the media houses I listed earlier will either edit their posts or remove them entirely as the truth reveals itself.

Why Haven't We Manufactured our own African Phone from scratch?

Manufacturing circuit boards is not as simple as the chip looks. Even if we were to recruit hundreds of locals at cheap labour costs to sit around the clock with soldering irons, we'd still see little return. Mass production of phones is not a parent's garage or backyard job.

Asia is well ahead of us in terms of hardware manufacturing. It's going to take many government-corruptionless years to catch up with China and his neighbours on manufacturing circuit boards for instance. They have far cheaper labour than all of Africa and a greater output of Electronic and Computer Engineers/Scientists for us to catch up or present a business case worth pumping money into. Unless we're looking into next generation technology instead of re-inventing the wheel, we'll never catch up.

To maximize profit and try to be "first-to-market", the rest of the world (including first world countries) still goes to the Far East for manufacturing and there's absolutely nothing wrong with that capitalistic approach. It also allows businesses to focus their energy on everything else that supports the product and brand.

In other parts of Africa, the trend and adoption of these low-cost unknown-brand phones is much greater. The Maraphone in Rwanda for instance is faced with a steep run against the entrenched and cheaper Chinese brands like Tecno. It will be a long time before ordinary users start spending their money on our phones, especially with an upcoming brand conscious generation.

Parting Sanitizer Shot

It actually took a while to get myself to write about this even though it was bothering me for some weeks. A part of me felt that we were starved for good stories (even before the global covid-19 pandemic) and I should just let this story slide in hope that it doesn't cause any harm. The most it's done is get a few non-tech people excited for a day and it will quickly be forgotten. I see it, however, popping up in WhatsApp groups and being believed by too many people close to me. I had to get it off my chest. If it dies it dies.

The way I see it, this whole stunt is driving traffic to his Thatha website where he also promotes his Development (Mobile Apps, Websites, etc) business services. I hope that's working well for him.

RPKI ROV Sickle in Full Swing

The month of June saw a spike in Resource Public Key Infrastructure (RPKI) and BGP Route Origin Validation (ROV) activity. This compelled me to curate some of the events that caught my eye. By increased activity, I'm referring to both increased incidents of routing security improvement and, social occasions focused on promoting RPKI and raising awareness. However, this blogpost is about the latter. The purpose began as drafting a personal summary of all the compressed RPKI knowledge consumed. It's now evolved into what I think could be useful for those that are new to the concept and may have missed some of these important resources that are shared freely on the internet.

TL;DR

You can skip reading the rest of this article and dig straight into the following fantastic four:

• RPKI Zero To Hero Live Demo a.k.a Implementing RPKI: It's easier than you think

• Mikrotik announces RouterOSv7 Beta8 with RPKI support and someone takes it for a ride

• ICOMM2020 - Securing The Routing System with MANRS

• Measuring Route Origin Validation

Since the break out of the COVID-19 global pandemic, it had been a while since techies that help keep the beast that is the global internet breathing, came together to share knowledge of this magnitude. On the 19th of June, some of these champions live-streamed a demonstration of how quick and easy it can be to set up the different components of an RPKI-fluent network. The line up represented a balanced mix of vendors and organizations that play significant roles in the make-up of the internet. This included Apstra, Arista, Cisco, Cloudflare, Juniper Networks, NLnet Labs, Nokia, Orhan Ergun LLC and RIPE NCC.

Moderated by Orhan Ergun and Jeff Tantsura, the Zoom session was also live streamed via YouTube and Facebook - a move that should be celebrated for making the live session more accessible to the global community. I personally experienced challenges joining the Zoom session so this proved handy.

With a goal of achieving the following in 2 hours, the session turned out much shorter than expected:

• Creating Route Origin Authorizations (ROAs) via your Regional Internet Registry (RIR) member portal
• Installing a validator in your network
• Configuring your router to enable RPKI and perform BGP ROV
  

Nathalie Trenaman from RIPE NCC kicked off with creating a ROA via their member portal. She also shared some best practices, like how you should be conservative with maxLength of your ROAs, all within 5 minutes. Her introduction and background about RPKI (which I think was planned to be presented by Jeff Tantsura) actually took longer than creating a ROA. Assuming that most networks are familiar with RPKI terminology and a bit of its theory, the excuse of not having time to create ROA(s) was proved to be invalid. 🧀🧀🧀

Validator installation demos included RIPE NCC's RPKI Validator, routinator 3000 from NLNetlabs and Cloudflare's OctoRPKI. Yes, no FORT. Another representative from RIPE NCC, Ties de Kock demonstrated the RIPE Validator and RTR Server installations. After installation of the 2 packages, making some necessary configuration adjustments and starting up the services which took less than 5 minutes, he explained that the validator can take about 15 minutes to be ready with data downloaded from the different repositories.

Alex Band then took the reigns to demonstrate a routinator installation. Unlike RIPE Validator, there are no packages to download for installation and routinator is built from source code - a conscious decision they made because of its frequent releases. To set up the environment for routinator, you need to install rsync, the C toolchain and Rust (the programming language that routinator is written in). While waiting for the routinator installation to complete, Alex spoke about their transition from using rsync to https for rrdp. The final step is to start the application as an RTR server and wait about 10 minutes for data to be downloaded from the difference repositories. The demonstration took roughly 7 minutes.

Loius Poinsignon from Clouflare then demonstrated OctoRPKI. Similar to the one from RIPE NCC, the validator and RTR server are separate packages in the form of OctoRPKI and GoRTR respectively. Loius explained that this has the advantage of having GoRTR running on a different machine closer to your routers while the validator is hosted elsewhere safe. His demonstration was also within 7 minutes.

Enabling Routers to Speak RPKI

In what seemed like a friendly tournament of the router vendors, there were demonstrations from Juniper Networks by Melchior Aelmans, Nokia's Greg Hankins, Florian Hibler from Arista, and Cisco's duo Vinay Shankarkumar and Jakob Heitz. Setting up their routers to a point of dropping Invalids and showing off some troubleshooting commands took less than 10 minutes each. I've summarized how long each activity took below. Of course a beginner would take slightly longer than that because they'd have to RTFM first, but these numbers still highlight how smooth the process has become. This in no way trivializes the process.

Creating ROA	5 minutes	Installing Routinator Validator	7 minutes
Installing RIPE Validator	5 minutes	Installing OctoRPKI Validator	7 minutes
Configuring JUNOS for ROV	5 minutes	Configuring Arista EOS for ROV	8 minutes
*Configuring Cisco IOS XR for ROV	16minutes	Configuring Nokia SR OS for ROV	5 minutes

* The Cisco session had Q&A during the demonstration unlike the others who held it off until their configurations were completed

The total amount of time reflected in the table is only about an hour of a 1-hour-45-minute session. There are many gems of Q&A throughout the session including an interesting wrap up. I strongly encourage a watch of the entire video.

Mikrotik roller coaster

It was a pity that there wasn’t a Mikrotik Zero To Hero demonstration given that earlier in the month the industry was excited about the announcement that RouterOS had a beta release of an RPKI ROV ready image and Massimiliano Stucchi, under his personal ASN AS58280 had taken it for ride a few days before the RIPE event. These were big news for the industry given that Mikrotik has a large footprint in small and startup networks, especially in developing countries and continents.

Unfortunately, this Mikrotik party was crushed in another RPKI studded session held a week later. Enter "InterCommunity: Securing Global Routing" featuring Melchior Aelmans (this time wearing a moderator mask), Abdul Awal of the Bangladesh National DataCentre, Mark Tinka of SEACOM, Kevin Blumberg from TORIX, Jorge Cano from NIC.mx, and Tashi Phuntsho from APNIC. It was during Tashi's slot that it was revealed that the RPKI implementation on RouterOSv7.0beta8 is broken, by pointing us to a Mikrotik Forum discussion where his team raised the issues experienced and Mikrotik also confirmed the bug.

This Internet Society (ISOC) #ICOMM2020 event involved the panelists sharing their experiences from their various contributions in making global routing more secure. Awal's presentation on his project RPKI Deployment in South Asia was a good place to start and he dropped many gems on how to mobilize RPKI adoption in your region. He also published a write-up on the same topic in June.

Mark Tinka (Head of Engineering at Seacom) took us back to when they tried to implement RPKI in 2014 only to discover bugs in Cisco IOS XE and that not many networks had deployed ROV. Dropping invalids while your competitors allow such traffic through can put you at a business disadvantage. Together with another IP Transit network provider in the same region, Workonline Communications, they went live with RPKI in April 2019. He found that IOS XR and JUNOS worked well while IOS XE was still buggy. This was supported by Kevin Blumberg (President of the Toronto Internet Exchange) who shared their history of ticking all the boxes of the MANRS actions as the largest IXP in Canada. Echoing what was shared by many in the Zero To Hero event, Kevin runs 2 RPKI validator servers in parallel and wants to see how they differentiate for the foreseeable future.

Representing the open source software development efforts of NIC Mexico, Jorge Cano took us through an introduction to FORT Zero To Hero gap-filler which is their most recent contribution to the community. Again the running of 2 or more validators in your network was stressed.

Similar to Mark Tinka, Tashi Phuntsho took us back to January 2014 where they started to implement RPKI ROV using IOS XE and things broke spectacularly to the point of upsetting the local king who happened to be their customer 😂. Tashi stressed the need to test your validators by showing us an example of the asymmetry between Validated ROA Payloads (VRPs), extracted a couple of hours before the webinar, from FORT and Routinator. Please watch the video and see how significant the difference can be. He also shares some of the brilliant outreach work they've been doing.

It's worth mentioning that Tashi was also involved in an online APNIC webinar (Securing Internet Routing Tutorial) on the same topic earlier in the month. I personally didn't attend it but I'm sure it was useful given the topics listed in the agenda. If you have a link to the actual content and have permission to distribute it please leave it in the comments below and I'll have it added here.

Short Break (not an advertisement/sponsored content):

There was also a major update to Krill. Check out Krill Gains Powerful ROA Management Based on BGP Routing by Alex Band.

I also need to make mention of this Excuse Me, Your BGP Is Leaking episode by the #theInternetReport which has some interesting global routing security incidents and RPKI news for June.

Measuring Route Origin Validation

Last and certainly not least, there's a brilliant stock-take of how far we've come with ROV by the internet veteran and APNIC Chief Scientist Geoff Huston. This new way of measuring ROV is where I should respectively end this post. Geoff Huston and Joao Damas challenge you to think deeper about the numbers we often see being reported and the implications of dropping Invalids from an end-user perspective. It's definitely worth your time. Also look out for the surprising findings about Africa in his article.

Bonus: You can couple Geoff's blog post with a him being interviewed by Mehmet Akcin in the same month where he drops some fascinating insights.

blogger-node#show post summary

More awareness could have been raised in my continent/region about the Zero to Hero RIPE event. There was an alert shared about it on the ZANOG mailing list but I don't think there was anything beyond that. The session covered all the basics and, with an AFRINIC ROA creation example perhaps, it would be a great addition to the workshops that are being run in promotion of RPKI by various organizations in the region. If you're from my region, you had to have been on the RIPE mailing lists or eagerly following feeds about RPKI on social media to know about it.

The #ICOMM2020 event was well marketed in my region. I'm guessing this is because the organizing team consisted of ISOC representatives from Africa. It would be great to see Mikrotik release a working version of their code (one that also doesn't break IPv6) very soon. Overall, I wish that the momentum gained in the past month is not just an annual climax and hope that it will be kept up until that all-invalids-dropped day.

May your Valids live long and prosper!

Kenya and Nigeria IXPs Approaching the Final Stages of a Developed Internet Ecosystem

Eight years ago, only 30% of Kenya and Nigeria’s internet traffic remained in-country with Google Global Cache providing the only static international content. Today they have roughly 70% of traffic being exchanged locally. The exchange of this traffic occurs at Intenet Exchange Points (IXPs), namely KIXP and IXPN in Kenya and Nigeria respectively.

Stages of Internet Ecosystem Development

 

Source - Anchoring the African Internet Ecosystem: Lessons from Kenya and Nigeria’s Internet Exchange Point Growth by Michael Kende (Internet Society)

The Internet Society (ISOC) defines 3 stages of internet ecosystem development, determined by what percentage of the total internet traffic is localized, as seen in the above figure. In 2012, Kenya and Nigeria were in Stage1 and are today at the advanced phases of Stage2. A report by Michael Kende was launched earlier this week by ISOC detailing how such exponential growth was achieved.

The ISOC’s team in Africa had set themselves a goal of 80% localized traffic by 2020. Back in 2010 when this goal was set, anyone who follows African news would have dismissed it as pie in the sky. This great achievement calls for a detailed read of the report and forms a blueprint that can be adopted by other African countries who are still building their internet ecosystem.

The coming years will focus on seeing 80% of Africa internet traffic being locally accessed. The following are some of the key action points that need to be addressed to achieve this:

• Raise awareness of the benefits of local content hosting and peering at IXPs
• Local content developers who are currently hosting their content outside their countries should bring the content back home and have consumers enjoy lower latency
• Smaller Internet Service Providers (ISPs) should have presence at IXPs and peer with others as efficiently as possible
• Aggregation of demand for backbone capacity in and between cities. A pool of small ISPs can buy backbone capacity in bulk and enjoy volume discounts for instance
• Aggregate local content providers and acquire hosting services from them
• Extend domestic backbone infrastructure beyond the main landing point for submarine cables and main population center into other population centers
• Further strengthen the foundation of trust and collaboration amongst stakeholders

We often complain about Africa’s undocumented achievements. The amount of evolution of the internet ecosystems in these countries in such a short period should be widely celebrated. Both the work done in the past 8 years by all stakeholders and the reporting of it by ISOC are important milestones in the continent’s Growth and History of the Internet.

Why I Chose Blender For Simple Video Editing

Summary:

• With less than 3 months of the year left, Blender presents a new personal challenge to learn a new tool used by professionals and popularly known for being complicated with a steep learning curve.
• The tool is Open Source and therefore free to use. Should I ever need to enhance or extend its powers, it allows me to do that through Python, one of the languages I'm already familiar with.
• I have a belief that one day, as my self mastery improves, I'll create time to address my 3D animation itch.

The Need to Chop Scenes From A Movie

In Star Wars Episode IV, A New Hope, the Jawas on lifting R2-D2 complain that "uyasinda lomsunu" - A South African phrase meaning "This machine looks interesting but may be a challenge to dismantle". This is what I believe I hear(d) and my ears refuse to hear otherwise. I've posted about what I hear(d) on various social media platforms and have been called out as a liar or simply looking for action where there is none.

I needed to chop these few seconds from the movie to prove my super hearing ability and a number of video editing tools that I tried failed me. The complete movie file was about 5 Gigabytes in size and some free video editing tools would crash from just trying to load the file. Others would crash while in the middle of chopping the scene that I want. The purpose of this post is not to bash other tools so there won't be any names dropped. Safe to say that I tried a umber of tools both on Windows and Linux platforms. Chances of proving that the Jawas say "uyasinda lomsunu" were looking bleak.

A Tool For Professionals - No Cutting Corners

With a bit of "Top Tools To Do X" blog-posts-crunching, my Open Source and Flexibility filter left me with Blender. Most reviewers who just needed a tool for quick video editing without wanting to know what's under the bonnet were throwing No-Go flags. A number of Introductory videos on YouTube showed Blender to be a bit on the complicated side. The fact that the tool is boldly sold as "used by professionals" captured me. Needless to mention that it was used to create the pre-visuals of the movie Spiderman 2. I'm not aiming for professional or expert status with this but it should be fun to have a crack at learning the basics of a tool that professionals use.

The cherry on top is that it runs on Linux and is completely Open Source. I'm a dual-boot kind of guy but I prefer more of my time spent on the Linux side. I do some work on Python whenever required and the fact that I can edit or extend the power of Blender using Python just sealed the deal for me.

Know Thy Specs

My experience so far has been drastically different from other tools I've tried before. It took a couple of online videos for me to get comfortable with the user interface. It also required me to know technical specifications of the end product that I want.

With easy-to-use tools, like Windows Movie Maker, you don't need to know technical details such as resolution and the file type you're creating. When you're done editing you just click on button that creates a file for the target platform like YouTube or Facebook. The tool figures out the resolution and file format to use for you.

With Blender, I had to know the correct technical specifications of the file for the target platform and specify these settings in a number of not-very-obvious places during the editing process, before trying to export the final product. In my case, I needed to export files to Twitter, Instagram and Facebook with different formats supported. More painful was the Twitter requirement to generate different formats for mobile and PC.

Overall it's a great free tool for video editing and I'm looking forward to many hours of learning about its nuts and bolts. Kodwa unzima lomsunu.

A Lonely Afternoon with a Facebook Clone

One lazy afternoon I got a Facebook Messenger greeting from what appeared to be one of my aunts. All the cells in my body had to sit up because, in my reclusive life, a message from an aunt usually means I have to prepare for a major event like a funeral, wedding or tombstone unveiling. It's very rare that they just check up on my disappointing life.

Three interactions in, I started suspecting that my aunt had gone over the limit of her grape juice intake for the day. Her grammar was horrible and we skipped the introductory pleasantries too quick even for someone like me who hates such.

 

The “80,000.00 cashier check” was a clear sign that I was speaking to someone wearing a gold chain, gold tooth, crucifix tattoo and running multiple Facebook chats from stolen laptops. I had a long boring night ahead of me so I played along.

 

This dude sends me to a Facebook group which I think would look legit for a non-techie. If smarties can fall for MMM then this page was up to scratch. I mean who would doubt any of this after being shouted at with an “OKAY THANKS” to seal the deal right?

Or miss an opportunity to be edited into this picture?

I then launched a complaint/report to Facebook in the meantime, only to realise that it would take a day for them to bring this man down. That's enough time to attack my aunt's entire list of Facebook friends. Also enough reason to continue wasting his time...

 

Small test to confirm whether this dude has any South African bone in him:

The rest of the conversation was just me bouncing some startup ideas off him. He eventually gave up on me and probably concluded that I would be a crappy nephew to have.

Using tkinter in Netbeans 8.1

Environment

Operating System: Windows 10

IDE: Netbeans 8.1 using Jython 2.7 platform as Python plugin

I had also already installed Python 3.5.2 and its IDLE.

Background

I'm porting a friend's old code from Python to Java and had issues getting it to show me what it does. The code uses Tkinter (a Python interface for the Tk GUI toolkit) for its Graphical User Interface. The Tkinter library comes standard with official Python releases. The Python plugin for Netbeans however doesn't support Tkinter (at the time of publishing this post). This Netbeans plugin, developed by the nbPython community, uses Jython 2.7 as a platform, which would require usage of another GUI library.

Problem

I can run the old code with no issues when using the Python 3.5.2 IDLE but when I run the same code in Netbeans I get:

importError: no module named tkinter

It turns out that tkinter is used in a lot of Python courses so it was strange that, after a whole year of Netbeans 8.1 being officially released, nobody had posted a straightforward solution to this problem. Searches reveal that the problem is parked for later official plugin releases and, treated as a legacy coder's problem since everyone else is using more modern and advanced GUI libraries. The solution is however very simple and targeted at Netbeans newbies like myself.

Before trying my method, check if you're not facing the case sensitivity/renaming issue. A common problem people face is the change from Tkinter to all-lowercase tkinter between Python 2.7 and Python 3. It's all a case of RTFM really.

Solution

As in my case, you should have Python (besides the Netbeans Python plugin) installed already. Under the project's properties, change the Platform from Jython 2.7 to Python 3:

If you're changing platforms for the first time you're unlikely to find the Python 3 option in the Drop-down List. To add Python 3, click on “Manage” and then “Auto Detect” in the next dialog box. Netbeans will pick up your other Python installations.

Yes, that's it.

List of African Fake News Websites

africannewsupdates.com
buzznigeria.com
buzzsouthafrica.com
citysun.co.za
dailylive.co.za
dailytimes.co.za
dodgysaministers.com
eyenews.co.za
glonews360.com
hinnnews
hivisasa.com
howafrica.com
imzansi.co.za
livemonitor.co.za
mzansilive.co.za
mpasho.co.ke
nationalnewsbulletin.com
news24-tv.com
newsdaily.co.za
Newslo.com
pressgh.com
southafricainsider.co.za
southafricalatestnews.co.za
smithrise.com
spynigeria.com
theguard1an.com
times1ive.co.za
viroshare.com
yesiyesighana.com
zimbotoday.com
zimeye.net


Parody or news satire can be great entertainment tools. The problem starts when people spread it as gospel truth or when the producers become insensitive to the extent of announcing someone's "death".

I'm building a crowdsourced list of African fake news sites. The aim is not to take food off the website owners' table. I'm only hoping to increase awareness of news satire and for people to start sharing these "news" only for what they are.

Please help me to grow the list whenever you find any site that isn't listed. You can add it in the comment section below and I'll update the list.

This is also not about generating traffic to my blog. I just thought it would be useful for those who want a quick credibility check of source before sharing local "news". I often copy and paste the list into WhatsApp/Facebook groups when someone shares a link to a satirical article as credible source.

Just Credit The Source Man!

Social Media (lead by Twitter) recently went ablaze with a video of South Africa's globally renowned DJ Black Coffee slapping South African rapper AKA's manager. While the video went viral, its owner also dished out some virtual slaps in a different ring at two competitor media houses which shared the video without crediting them.

Stop stealing content @yoMzansi @SocialSA_ Stop being lazy. Conduct interviews. Attend events. Take photos. Take your readers SERIOUSLY.

— Andile Mathobela (@AndileMathobela) September 25, 2016

Y'all have a nerve @yoMzansi cropping out @Zkhiphani branding like that & posting the video as if its your own. Bunch of THIEVES.

— Andile Mathobela (@AndileMathobela) September 25, 2016

What got me interested in the story was that not only did one of the media houses not credit the source, they also removed all traces of the source (tags/logos) from the video. This amounts to stealing and goes against one of the most important ethics code followed by bloggers, vloggers, curators, etc : Prominently credit the original source.

WATCH VIDEO: Black Coffee slaps AKA's team member (road manager) at #dstvirock Festival — https://t.co/5VPqPhB8nF | pic.twitter.com/lZ2820b6nW

— yomzansi.com (@yoMzansi) September 25, 2016

Having spent some time doing Content Production myself, I've noted with dismay a number of these cases involving Africans. Our continent has a history of having our art stolen and often reproduced without any credit given to Africans. Should we even be discussing why we shouldn't steal from each other? I might be exaggerating by classifying the work we share on social media in the same category as African art, but do we really need to preach ethics in content sharing between African digital media startups?

Watermarks

People always complain about the @Zkhiphani watermark, what they don't understand is that people like @yoMzansi steal our work

— Andile Mathobela (@AndileMathobela) September 25, 2016

I can understand why people would complain about Zkhiphani.com's watermark. I didn't see this in their videos, but I've seen how intrusive their watermark can be on their pictures. I see nothing wrong with the placement of their logo or tag in the corners of their videos, however their watermarks on pictures often run across the main subject and have prominence.

Global knowledge says we should apply our watermarks without being obtrusive. Obtrusive watermarks also affect the rate at which our content is shared. Some people don't like to share content that may seem to have too much advertising.

Some of the world's decorated photographers don't use watermarks, while the style and quality of their work identifies them. If we're failing to make our watermarks non-intrusive, we may need to explore other creative ways of identifying ourselves. As we've seen with yomzansi, our watermarks can be completely edited out anyway.

This watermark discussion entertains this idea of stealing. Just credit the source man!

Benefits of crediting the source

Crediting the source is flattering. Often, the content owner will retweet the tweet containing their credit. At the time of publishing, Zkhiphani.com had 63K Twitter followers while yomzansi had 33.1K. The opportunity of yomzansi reaching new followers from Zkiphani.com far outweighs the amount of work required to edit and share the content as their own.

I'm not sure if there was any beef between the 2 startups prior to this incident, but the crediting/flattering can also lead to building relationships and collaborations in the same industry. Compared to other continents we have far less consumers of our digital media. It gets worse when our content is focused on pop culture and entertainment for a specific African country. Being ethical, collaborating and sharing our networks in the African context can be more beneficial than the dirty games that come with traditional competition.

Just credit the source man!

Why Net Neutrality In Afrika Is a Pipe Dream - The Internet Belongs To All Who Live In It

This article was originally posted on iAfrikan.

My first internet experience was through a school library PC in the 90s. I only knew of yahoo.com at the time and I was told that I could search for whatever information I wanted, including underground Hip Hop artists who wouldn't otherwise be featured on our main source of information at the time: Word Up! Magazine or The Source. This capability alone had me willing to spend a portion of profit from my Fireballs Candy business just to rent 15 minutes a day on the library PC.

Browsing through Yahoo was enough experience to get me hooked on this web of information. For most in Africa, first contact with the internet is through mobile phones and I assume as captivating, if not more, as my experience was. Accessing 3 websites of relevance is more than enough to get one hooked and wanting to surf more.

Unfortunately for Mme Xhanthi who lives off $1 a day, in an era where #dataMustFall is so serious a hashtag that it makes it all the way up to parliament, those few sites is where the road ends since she relies on zero-rated services like Free Basics by Facebook. 

#datamustfall

Do you think Mme Xhanthi would give up the little experience she gets from the free access to these few sites, in the name of a concept such as Net Neutrality? The likes of Mme Xhanthi makes up the majority of the population of South Africa. Yet, a minority privileged techie group insists on deciding for this majority not to have this limited free access because they believe everybody's internet should be served on an all or nothing plate.

This is why the likes of Joe Mucheru (Cabinet Secretary of Kenya's Ministry of Information, Communications and Technology) say

“It’s like saying someone has no food, but if someone brings them bread we are not going to allow them to have the bread because they must have a balanced diet,”

I'm all for net neutrality. In fact, I've turned away offers to get involved in projects that had a slight hint of creating islands or a compromised menu of the internet. My opening paragraphs might have you thinking I'm here to defend net non-neutrality. However, I wrote these notes in trying to understand the position of most African governments on the subject.

Demographics

Africa has the highest percentage of population that is offline – almost 75% according to the ITU's “ICT Facts and Figures 2016”.

Our internet penetration rates are below 30 while all other continents are above that. Would our governments believe that we're in position to be sharing the same access philosophies with Europe, whose penetration rate is in the 80s?

Education

• Governments face a major challenge of getting information to its people. From their developmental programmes or educational campaigns to progress reports, reaching the rural or poverty stricken majority is costly and produces low results.
• The quality of education in most African countries is deteriorating. Supplementing this with free access to learning platforms such as beSmart, worldreader andwikipedia can lessen the negative effects somewhat.
• Besides being oversubscribed, most African families also cannot afford the costs that come with basic and tertiary education.

The above are just some points governments consider when deciding on the applicability and legality of zero-rated services. Free Basics by Facebook, for instance, provides access to free information about health, employment, starting a business and other local information on your mobile phone. Scholars4Dev gives access to a list of international scholarships.

Under Free Basics there are a number of other valuable sites that one can access freely to self-develop and fight poverty. Unless theirs is to keep its people uneducated and oppressed, it would be difficult for any African government to reject these free services.

Innovation and Competition

We often argue that not following net neutrality laws aids the monopoly of the zero-rated services over competition and stifles innovation. This is undeniable.

Let's assume that there are 10 free services that are offered via these zero-rated platforms. This means that 10 different areas of online services/solutions will experience unfair competition. Compare this to hundreds of African solutions that could sprout from people having access to information that they would not have ordinarily had.

A popular saying at startup or tech events is

“Africa doesn't need another Facebook or Twitter. We need solutions that address unique African problems.” Some-Motivational-Speaker-Turned-Tech-Guru

The number of rural and/or uniquely African challenges, we need to find solutions for, outweigh the few tech startups that could potentially experience unfair competition as a result of zero-rated services.

We've all seen Maslow’s Hierarchy of Needs in the Internet Age. The already connected urbanites have jokingly placed internet access as more necessary than food. When governments give free meals to school children from poor communities with a preference for brandX, we don't complain about unfair competition for brandY.

If we follow Maslow, we shouldn't complain when those who can't afford it are given some form of internet. I admit this is a pathetic attempt at expanding Joe Mucheru's earlier analogy, but it feels relevant here.

Politics

Net Neutrality activists have watched in paralysis, a couple of times, as Ugandan President Yoweri Museveni isolated social media traffic from the rest of the internet in his country and put this net island into darkness. The last time he did it, the confirmed government order cleared his path to a two thirds majority election victory.

It highlighted how quick and easy it is for governments to switch these islands on and off. This happened in many other African countries before and after that. Most African governments are not willing to give this power up.

There have been exceptions like Egypt who suspended Free Basics at the end of 2015 for instance. Just as our hopes were raised, Moroccans were blocked from using VoIP services a week later.

Why Did Morocco Ban VoIP & Why Is South Africa Looking To Regulate OTT Services?

What this further highlights is that the direction of African governments' swing in terms of how they control the internet can’t be determined or generalised. Neither can this be done for developing countries in general. The world witnessed how India rejected Free Basics earlier in 2016, which prompted an emotional rant from Marc Andreessen that painted him and his team's Free Basics as fake philanthropy and new age colonization. I believe our governments are well aware of this.

The consequences of a free-to-do-as-you-like environment on the internet for governments can be disastrous in the long term. It only takes the manipulation and imagination of a loony dictator to ruin the internet experience for the already connected. After these notes, I was left sympathizing with governments whose intentions are good and face tough challenges to get their people online. Each African country needs to consider some not-so-neutral laws in the short term. Pure net neutrality is definitely essential when we're all online.