NDN Video Streaming on the ndn6 Network

The ndn6 network, my own global scale Named Data Networking network, came back earlier this year. I moved my NDNts video streaming app into the ndn6 network, to reduce dependency on the NDN testbed. How well is it performing?

QUIC ⇒ HTTP/3

In my last article "NDN video streaming over QUIC", I used Chrome browser's experimental QuicTransport feature to perform video streaming over Named Data Networking. The analysis revealed that QUIC transport was generally performing better than WebSockets in this application, according to metrics including video resolution and startup latency.

Web technologies are constantly evolving. QuicTransport was in Origin Trial status at the time, but it was discontinued as of Chrome 91. WebTransport was introduced in its place. The main difference is that, WebTransport uses HTTP/3 as the underlying network protocol, while QuicTransport uses QUIC datagrams.

Since HTTP/3 runs over QUIC, I expect no performance difference between the two. I promptly registered for the WebTransport Origin Trial, and updated my gateways and NDNts libraries to use the new API.

Return of the ndn6 Network

In 2014, I installed NDN Forwarding Daemon (NFD) router on a tiny 128MB virtual machine. I named this node ndn6: IPv6 NDN router, because the virtual machine, purchased from the original Low End Spirit forum for €3.00/year, was an IPv6-primary service. I idled this router for three years, and then shut it down in 2017.

I created NDNts: NDN libraries for the modern web in 2019. Since then, I have been publishing my own content over Named Data Networking, most prominently the NDN push-ups. NDNts does not require a local forwarder, so that I operated video repositories by directly connecting to a nearby testbed router via UDP tunnel. Shortly after, I started experimenting with QUIC transport, which involved deploying several NDN-QUIC gateways to translate between NFD's plain UDP packets and Chrome's QUIC transport protocol.

One day, I realized: my content is sent to the global NDN testbed, and then retrieved back to my servers for delivery to browsers over QUIC. My video repository in Buffalo and NDN-QUIC gateway in Montreal are quite close to each other, but the packets are taking a detour to Boston, increasing latency by at least 10ms. Also, since I statically assign a testbed router for each application, a downtime of that router would bring my application offline as well. I thought, instead of operating isolated applications and gateways, I should setup my own NDN network.

Setting up a new NDN network is no small feat. NFD and NLSR implement forwarding and routing, but I also need to:

  • Decide on a topology between different routers.
  • Assign a name prefix to each router.
  • Install and update software in each router.
  • Generate configuration files for NFD and NLSR, and modify them as the topology changes.
  • Monitor the network and know about ongoing problems.

Face and PacketHandler in NDNph

Face is an overloaded term in Named Data Networking (NDN). My last article explained what is a face in NDN forwarders and NDN libraries, and then described the endpoint design in my NDNts library. This time, I'll introduce a unique face API design in my NDNph library.

NDNph is a C++ header-only library that enables low level application development. It supports multiple platforms, but is primarily designed for microcontrollers with limited hardware resources. In particular, RAM capacity is very limited, with typical values ranging from 50KB (ESP8266) to 320KB (ESP32). This necessitates a different API design for the face.

Overhead of a Traditional Face

Traditionally, a face in NDN libraries has the following features:

  • send and receive NDN network layer packets
  • match incoming Data against outgoing Interests
  • keep track of Interest timeouts
  • dispatch incoming Interests to producer callback functions

NDN Video Streaming over QUIC

NDN over WebSockets == TCP over TCP

Named Data Networking (NDN) was first ported to web browser environment in 2012. At that time, a browser-based JavaScript application can communicate with the Internet via either XMLHTTPRequest or WebSocket. Feeling that WebSocket is a better match for the NDN implementation at the time, I wrote the initial code for a CCNx WebSocket proxy.

Web applications would connect to this proxy over TCP, negotiate a WebSocket connection, and send NDN packets in WebSocket frames. The proxy then decapsulates these frames, and delivers the NDN packets to ccnd forwarder over TCP.

NDN-over-WebSockets survived multiple protocol changes over the years, and made its way into the NDN Forwarding Daemon (NFD). It worked fine for simple NDN web applications, such as status pages and text chat, and even file retrievals.

Recently, with the rise of video streaming on the NDN testbed, congestion control functionality starts to show up in NDN libraries. Then, a question popped into my mind: WebSockets run over TCP, and NDN congestion control algorithms are largely borrowed from TCP, would this cause any problems?

What is a "Face" in Named Data Networking?

Face is an overloaded term in Named Data Networking (NDN). Most developers have some feeling of what a face is, but often find it hard to explain. This article attempts to demystify the concept of face in NDN.

"Face" as Defined in Publications

The original NDN paper, Networking Named Content, introduces the term face in a footnote:

We use the term face rather than interface because packets are not only forwarded over hardware network interfaces but also exchanged directly with application processes within a machine.

NFD, the original NDN forwarder software, explains in the NFD Developer Guide:

The Reality of NDN Video Streaming

As of 2019, video accounts for over 60% of downstream traffic on the Internet. It is believed that video streaming could benefit from the in-network caching feature of Named Data Networking (NDN), which would reduce the total traffic volume and bring cost savings for Internet service providers and content publishers. Far Cry: Will CDNs Hear NDN's Call?, a paper published at ACM-ICN 2020 conference, is the latest attempt on NDN video streaming.

How iViSA Works

In Far Cry, the authors implemented iViSA, a browser-based video streaming application that runs on the global NDN testbed, and then performed some comparison study between this application and similar HTTP video streaming application deployed on commercial CDN services.

It's said that if you want reproducible science, the software needs to be open source. The authors released most of their source code, and we can get a peek into how iViSA actually works.

The backend server repository contains:

NFD nightly APT repository

Last year, I started building NFD nightly packages in GitHub Actions. So far, installation is a manual procedure: the user must manually download the ZIP files from nfd-nightly.ndn.today, decompress them, and figure out the dependency among various .deb packages. Starting today, I'm publishing NFD nightly packages in an APT repository, and you can install them with apt-get command.

Add the NFD nightly APT repository

Add the repository with the command that matches your platform:

# Ubuntu 18.04 (bionic), amd64 (laptops and servers)
echo "deb [trusted=yes] https://nfd-nightly-apt.ndn.today/ubuntu bionic main" \
  | sudo tee /etc/apt/sources.list.d/nfd-nightly.list

# Ubuntu 20.04 (focal), amd64 (laptops and servers)
echo "deb [trusted=yes] https://nfd-nightly-apt.ndn.today/ubuntu focal main" \
  | sudo tee /etc/apt/sources.list.d/nfd-nightly.list

# Debian 10 (buster), amd64 (laptops and servers)
echo "deb [trusted=yes] https://nfd-nightly-apt.ndn.today/debian buster main" \
  | sudo tee /etc/apt/sources.list.d/nfd-nightly.list

# Debian 10 (buster), armv7 (Raspberry Pi 3 or 4)
echo "deb [trusted=yes] https://nfd-nightly-apt.ndn.today/debian buster main" \
  | sudo tee /etc/apt/sources.list.d/nfd-nightly.list

# Debian 10 (buster), armv6 (Raspberry Pi Zero W)
echo "deb [trusted=yes] https://nfd-nightly-apt.ndn.today/raspberrypi buster main" \
  | sudo tee /etc/apt/sources.list.d/nfd-nightly.list

If your operating system and CPU architecture combination is not listed, it is not supported by NFD nightly. See the previous post on how to request a new platform.

Self-Hosted NDNts Nightly Build

NDNts nightly build is a set of NPM-compatible tarballs compiled automatically from the development branch of NDNts, Named Data Networking (NDN) libraries for the modern web, distributed on https://ndnts-nightly.ndn.today website. Users can install NDNts nightly build following these instructions.

However, this website only stores the latest version of NDNts packages. This has been causing installation conflicts when NPM tries to look for previous versions. Moreover, as I have declared, I don't care much about backwards compatibility. With NPM, all published versions are stored indefinitely, so you can continue using an older version without being affected by breaking changes. On the other hand, once a new nightly build is uploaded, the previous version is overwritten and no longer available for downloads. You are then forced to cope with the breaking changes I introduce from time to time, possibly at higher frequency than you would like to.

Today, I'm introducing two methods for self-hosting NDNts nightly build. Both methods allow you to build a specific version of NDNts codebase from a checkout of the NDNts monorepo, and generate a set of tarballs that you can host locally on a server under your control. Afterwards, you can install NDNts packages from this server, without relying on my website and without being affected by my breaking changes.

Self-Hosted NDNts on an HTTP Server

The following steps allow you to create and host tarballs of one specific version of NDNts.

NDN-DPDK: NDN Forwarding at 100 Gbps on Commodity Hardware

Presented at: 7th ACM Conference on Information-Centric Networking (ICN 2020)

Since the Named Data Networking (NDN) data plane requires name-based lookup of potentially large tables using variable-length hierarchical names as well as per-packet state updates, achieving high-speed NDN forwarding remains a challenge. In order to address this gap, we developed a high-performance NDN router capable of reaching forwarding rates higher than 100 Gbps while running on commodity hardware. In this paper we present our design and discuss its tradeoffs. We achieved this performance through several optimization techniques that include adopting better algorithms and efficient data structures, as well as making use of the parallelism offered by modern multi-core CPUs and multiple hardware queues with user-space drivers for kernel bypass. Our open-source forwarder is the first software implementation of NDN to exceed 100 Gbps throughput while supporting the full protocol semantics. We also present the results of extensive benchmarking carried out to assess a number of performance dimensions and to diagnose the current bottlenecks in the packet processing pipeline for future scalability enhancements. Finally, we identify future work which includes hardware-assisted ingress traffic dispatching, dynamic load balancing across forwarding threads, and novel caching solutions to accommodate on-disk content stores.

Read full paper at ACM Digital Library: NDN-DPDK: NDN Forwarding at 100 Gbps on Commodity Hardware

NDN-DPDK logo

NDNts Demo at NDN Community Meeting 2020

NDN Community Meeting is an annual event that brings together a large community of researchers from academia, industry, and government, as well as users and other parties interested in the development of Named Data Networking (NDN) technology. Having no peer review process, it is a prime opportunity to showcase my personal projects to the community. I demo'ed my ndn-js home surveillance camera at NDNcomm 2018. This time, I decide to demo my flagship product, NDNts: NDN Libraries for the Modern Web.

The Demo Video

NDNts is a set of libraries with many different features, where do I start? I decide to select a subset of unique features that are not found in any other library:

  • The Endpoint API that enhances face by automatically handling repetitive tasks such as Interest retransmissions and packet signing/verification, so that app developers can focus on the application logic.
  • An implementation of trust schemas.
  • NDN Certificate Management protocol implementation, including a graphical user interface for the certificate authority component.

I also threw in two web applications: