Deep Atlantic Storage: Streaming Bits

I'm bored on 4th of July holiday, so I made a wacky webpage: Deep Atlantic Storage. It is described as a free file storage service, where you can upload any file to be stored deep in the Atlantic Ocean, with no size limit and content restriction whatsoever. How does it work, and how can I afford to provide it?

This article is the third of a 3-part series that reveals the secrets behind Deep Atlantic Storage. The first part revealed that the uploaded file is sorted which drastically reduces its storage demand, and introduced the bit sorting algorithm. The second part covered how to process the upload in the browser using Web Workers. Now I'd continue from there, and explain where I store the files and how I offer downloads with reasonable costs.

Storage in the URL

Deep Atlantic Storage sorts the bits in every uploaded file. After sorting, each file can be represented by two numbers: the number of 0 bits, the number of 1 bits. Given these two numbers, the sorted file can be reconstructed.

I could make a database or use one of those fancy NoSQL thingy to store those numbers that represent the files, but I prefer my websites stateless so that I don't need to take backups. Therefore, I decided to encode those numbers in the URI.

Deep Atlantic Storage: Sorting Bits

I'm bored on 4th of July holiday, so I made a wacky webpage: Deep Atlantic Storage. It is described as a free file storage service, where you can upload any file to be stored deep in the Atlantic Ocean, with no size limit and content restriction whatsoever. Since Chia currency farming became popular in May, hard drive prices went up significantly. How can I afford to operate an unlimited free storage service?

"Advanced Sorting Technology"

One of the benefits listed on Deep Atlantic Storage webpage is:

  • Advanced sorting technology keeps your data neatly ordered.

What this means is that, content in the uploaded file would be sorted before being stored.

Face and PacketHandler in NDNph

Face is an overloaded term in Named Data Networking (NDN). My last article explained what is a face in NDN forwarders and NDN libraries, and then described the endpoint design in my NDNts library. This time, I'll introduce a unique face API design in my NDNph library.

NDNph is a C++ header-only library that enables low level application development. It supports multiple platforms, but is primarily designed for microcontrollers with limited hardware resources. In particular, RAM capacity is very limited, with typical values ranging from 50KB (ESP8266) to 320KB (ESP32). This necessitates a different API design for the face.

Overhead of a Traditional Face

Traditionally, a face in NDN libraries has the following features:

  • send and receive NDN network layer packets
  • match incoming Data against outgoing Interests
  • keep track of Interest timeouts
  • dispatch incoming Interests to producer callback functions

Introducing NDNph and New Version of esp8266ndn

NDNph is my latest Named Data Networking (NDN) client library. This article gives an overview of this library.

History and Motivation

In 2016, I started esp8266ndn. It contains a copy of ndn-cpp-lite, UCLA REMAP's C++ library that does not use dynamic memory allocations. I then added integrations with ESP8266's network stack and crypto functions, making esp8266ndn the first NDN library that works on the ESP8266 microcontroller. Using this library, I built several projects, including a wearable jewelry and a call button. University of Memphis also deployed several sensor nodes using esp8266ndn.

Over the years, esp8266ndn gained many new features, and widened platform support to include ESP32 and nRF52. However, using ndn-cpp-lite as the core is becoming problematic:

  • New protocol features show up slowly, because ndn-cpp-lite author would not add a feature until ndn-cxx has it, and design discussions for ndn-cxx sometimes take several years.
  • There is no generic TLV encoder/decoder, making it difficult to support TLV structures in application layer.
  • ndn-cpp-lite is bloated with obsolete features, due to their backwards compatibility guarantees. Consequently, binary code size is unnecessarily large.
  • Although I can add patches to ndn-cpp-lite during importing into esp8266ndn, it has been difficult to test these patches. This isn't ndn-cpp-lite's fault, but is still an issue.