My Experience at Hack Arizona 2016

Hack Arizona is the largest collegiate hackathon in southwestern United States. I attended Hack Arizona 2016 and had a great experience, and I want to share what I experienced during this event.

Why I didn't attend in 2015

I heard about Hack Arizona when it started in 2015, but I decided against attending last year because 37 sleepless hours is harmful for health and won't produce high quality project.

Many of my friends went in 2015, and they shared their experiences and showed me their projects. The situation sounds less scary than I imagined:

  • Although you are provided enough coffee and Red Bull energy drinks to stay up, you are permitted to leave and re-enter at anytime, and you can sleep in the venue as well.
  • There's free food, and it's not just pizza.
  • Projects aren't of poor quality.

In October shortly before Hack Arizona 2016 opens for applications, the organizers gave a presentation to introduce the event. They introduced the rules, and emphasized how good the food would be.

The hackathon is hosted at University of Arizona, Science-Engineering Library, which is a 15-minute walk from my apartment. I'm not locked inside the library. And there's free food over the whole weekend. So, I applied to attend Hack Arizona 2016, just for free food.

But I ought to have a project, right?

The organizers are bringing in some fancy hardware such as drones, virtual reality headsets, and smart watches. The sponsors are also providing free access to their APIs or cloud services. Last year's winner list reveals that most prizes are awarded to projects that utilize either fancy hardware or sponsor APIs. But I don't want to use these: I want my project to stay alive after the hackathon is over, regardless of whether I could win a prize and keep the hardware or API access.

I chose my project on Jan 18, four days before the hackathon: I want to allow everyone to replicate traceroute bad.horse with his or her favorite lyrics. In case you don't know, traceroute bad.horse went viral on Twitter in Sep 2015: when you type traceroute -m255 bad.horse on a Linux terminal, you will see lyrics of Bad Horse:

root@vps2:~# traceroute -m255 bad.horse
traceroute to bad.horse (162.252.205.157), 255 hops max, 60 byte packets
 1  162.211.65.2 (162.211.65.2)  0.034 ms  0.019 ms  0.019 ms
 2  colo-lax6.as8100.net (96.44.180.33)  0.229 ms  0.211 ms  0.197 ms
 3  any2-la.cr1.lax0.amcbb.net (206.72.210.125)  28.473 ms  28.522 ms  0.305 ms
 4  ve10.cr2.lax0.amcbb.net (208.78.31.249)  28.554 ms  28.684 ms  29.085 ms
 5  ve1450.cr2.lga3.amcbb.net (108.60.128.73)  119.003 ms ve996.cr1.lga2.amcbb.net (69.9.33.149)  94.461 ms  66.357 ms
 6  e4-1.cr2.lga1.amcbb.net (208.68.168.149)  94.411 ms e2-1.cr2.lga1.amcbb.net (108.60.132.125)  102.458 ms e4-1.cr2.lga1.amcbb.net (208.68.168.149)  66.375 ms
 7  sandwichnet.dmarc.lga1.atlanticmetro.net (208.68.168.214)  98.141 ms  66.270 ms  93.815 ms
 8  bad.horse (162.252.205.130)  66.876 ms  98.439 ms  66.624 ms
 9  bad.horse (162.252.205.131)  98.342 ms  102.784 ms  98.222 ms
10  bad.horse (162.252.205.132)  80.838 ms  106.176 ms  107.279 ms
11  bad.horse (162.252.205.133)  81.537 ms  112.245 ms  112.113 ms
12  he.rides.across.the.nation (162.252.205.134)  85.035 ms  119.047 ms  122.138 ms
13  the.thoroughbred.of.sin (162.252.205.135)  91.999 ms  91.985 ms  91.974 ms
14  he.got.the.application (162.252.205.136)  132.601 ms  132.589 ms  132.576 ms
15  that.you.just.sent.in (162.252.205.137)  138.937 ms  139.004 ms  138.992 ms
16  it.needs.evaluation (162.252.205.138)  145.405 ms  145.393 ms  136.923 ms
17  so.let.the.games.begin (162.252.205.139)  115.580 ms  110.923 ms  110.869 ms
18  a.heinous.crime (162.252.205.140)  147.236 ms  115.370 ms  147.138 ms
19  a.show.of.force (162.252.205.141)  121.619 ms  162.346 ms  158.279 ms
20  a.murder.would.be.nice.of.course (162.252.205.142)  162.116 ms  130.679 ms  162.093 ms
21  bad.horse (162.252.205.143)  170.553 ms  136.238 ms  130.661 ms
22  bad.horse (162.252.205.144)  136.776 ms  140.851 ms  170.616 ms
23  bad.horse (162.252.205.145)  176.074 ms  176.094 ms  176.079 ms
24  he-s.bad (162.252.205.146)  147.480 ms  182.529 ms  188.084 ms
25  the.evil.league.of.evil (162.252.205.147)  191.525 ms  187.940 ms  151.492 ms
26  is.watching.so.beware (162.252.205.148)  195.965 ms  195.930 ms  195.893 ms
27  the.grade.that.you.receive (162.252.205.149)  204.823 ms  199.702 ms  198.020 ms
28  will.be.your.last.we.swear (162.252.205.150)  199.499 ms  199.833 ms  199.371 ms
29  so.make.the.bad.horse.gleeful (162.252.205.151)  209.442 ms  174.198 ms  205.254 ms
30  or.he-ll.make.you.his.mare (162.252.205.152)  179.639 ms  209.053 ms  208.988 ms
31  o_o (162.252.205.153)  216.543 ms  216.532 ms  179.407 ms
32  you-re.saddled.up (162.252.205.154)  221.956 ms  221.964 ms  187.144 ms
33  there-s.no.recourse (162.252.205.155)  226.859 ms  226.846 ms  190.445 ms
34  it-s.hi-ho.silver (162.252.205.156)  195.975 ms  199.293 ms  227.819 ms
35  signed.bad.horse (162.252.205.157)  227.660 ms  227.226 ms  227.570 ms

This looks cool. As a computer networking student, it doesn't take me long to figure out how they did it:

  1. In DNS, resolve bad.horse to the destination IP 162.252.205.157.
  2. Have some sort of network appliance to add 27 hops on the network path between the Internet and the destination IP.
  3. In DNS, resolve the IP of each hop to a line of lyrics.

Now imaging how cool it would be if I could have my own bad.horse, or allow anyone to setup one.

The technical preparation

There are two difficulties for this project: how to get so many IP addresses, and how to add the network hops. I'd have these sorted out first, before the hackathon. According to hackathon rules, I can't write any code before the official start time, but learning the APIs and setting up servers aren't forbidden.

As we all know, the world is running out of IPv4 addresses. Reserving a /24 needs a "justification" sent to IANA, and can accommodate up to 256 lines of lyrics. This clearly won't allow anyone to setup their traceroute target. So I went to IPv6.

I'm using the cheapest Virtual Private Server (VPS) with OpenVZ virtualization. Although my VPS have native IPv6, routed IPv6 isn't supported in OpenVZ. Thus, I need to get routed IPv6 over an IPv4 tunnel from Tunnel Broker. After a few configuration changes and a few rounds of ticket exchanges with the VPS provider, I got routed IPv6 working.

To add the network hops, simulating them behind a TUN interface would be a good idea. I had experience with TAP interface in a computer networking class project, and TUN is very similar to TAP.

I also choose a name for the project, TraceArt, which stands for "traceroute artwork", and then setup necessary DNS delegation records so that DNS queries for the IPv6 block go to my server.

With these non-code components in place, I'm confident that I can proceed with the coding part during Hack Arizona 2016.

Shall I work solo?

In the past, I didn't have a good experience working in small teams without a manager:

  • When my teammates have as experienced as me, we too frequently get into technical debates, and cannot get anything done.
  • When my teammates are inexperienced, I become frustrated when they "mess things up" although it's not their intention, and I engage in micro-management which means I'm spending more time fixing their minor mistakes than I would do the tasks myself.

In bigger teams, or in a team with a manager, these can be settled by having a group meeting to discuss the issue, or letting the manager make the decision. But at a hackathon, it's always a small team (of no more than four), with different technical backgrounds, and without a manager. And the problems above would kill my productivity.

Thus, I'm planning on working solo on the project.

Hack Arizona organizers had one last introduction session on Jan 20. When asked about team size, they did confirm that a team is between one and four, which means working solo is permitted. But after hearing my plan, one of the organizers advised me to get a team, because:

  • Having a team (of more than one) is part of the hackathon experience.
  • The sponsors (which are also potential employers) want to see you are able to work in a team.

After this meeting, in the days leading up to the hackathon, I kept debating with myself on whether to work solo or get a team.

I have a team!

I made my decision the night before hackathon, when one of my Facebook friends, Ruoyu Li, posted in the Hack Arizona Attendees group:

Hey everyone, I'm new to programing and know a little about Python. I thought this hackathon would be a great opportunity for me to learn more about programing. I'm willing to do anything that I can do to help teammates. I'm wondering that would anyone could let a beginner to join their team. I'll be so so grateful if anyone willing to give me a chance!

Although I don't remember why I added him on Facebook, he's not a total stranger as we have been Facebook friends since Aug 2014. This looks like a great chance for me to interact with people. After all, as a beginner, he won't blame me for designing or executing a poor project and diminish his chance of winning a prize. On the other hand, my project is easy enough that I could finish myself even if he couldn't do anything. Thus, I invited him onto my (previously non-existent) team.

Two more attendees expressed interest in working with me, even without knowing what I wanted to build. I accepted Mark, who has a bit of JavaScript experience, to help with frontend webpage. I didn't accept a fourth person because there are only three modules in the design.

The check-in

Hack Arizona check-in is hosted at University of Arizona's newest building, ENR2. I heard there would be swags at sponsor booths, so I packed light, to maximize the backpack capacity for all the swags.

What did I get?

  • Hack Arizona bag
  • Hack Arizona shirt
  • Hack Arizona sticker
  • Recruiting Ventures shirt
  • Raytheon mug
  • Raytheon's raffle ticket for a chance to win a mini drone
  • Cognizant headset
  • Rincon Research Corporation measuring tape
  • USAA flash drive

There are more swags available for grabs, but I don't need them, so I'd leave the chance to others.

I approached Cisco booth and introduced the idea of TraceArt, and they said it's "interesting", although their booth is mainly promoting Cisco's newest product Cisco Spark, a real-time collaboration platform. I also enjoyed a chat with USAA about their aviation products, such as the ADS-B transponder installed in aircrafts.

Then it's an uneventful 90-minute opening ceremony in a lecture hall, during which I'm feeling thirsty after drinking my bottle of water.

TraceArt day 1

I managed to meet both my teammates before the opening ceremony. I showed the effect of traceroute bad.horse with an SSH client on my smartphone. Using pen and paper, I tried my best to explain what is traceroute and how it works, and introduce the architecture of my app. Yes, this kind of explanation is best done with pen and paper, or mark and whiteboard; they are more effective than using a computer. When they seem to understand, we left each other our phone numbers and email addresses, and scheduled the next meeting to be after 20:00 when hacking time officially begins, before each of us heads into the opening ceremony.

The opening ceremony ended around 19:30. I ran to the hacking space, Science-Engineering Library. Then I made one of the worst decisions during the weekend: I chose to wait for my teammates at the gate, before heading upstairs to find a room. As a result, after I'm joined by Ruoyu 15 minutes later and texted by Mark saying he'll come later, all the study rooms in the library are taken, and we can barely find a desk that has power outlets nearby. But in any case, we have a "base" now.

I created the GitHub repository when Administration Building's bell rang for 8pm.

Since Ruoyu is a beginner with only one semester's Python experience in the context of class assignments, I decide to assign him the simplest module in the app: converting lines of lyrics to DNS configuration files. I believe that as long as I give him the exact specifications and examples of inputs and outputs, he would be able to program this correctly.

But the output needs to be interpreted by bind9 the DNS server. I can't trust him with root privilege on my VPS which runs several other applications, and I couldn't get HomeBrew's bind package working on his Macbook. But Macbook is powerful, so a Vagrant environment with Ubuntu 14.04 is a good choice, as long as the WiFi speeds are reasonable. Downloading VirtualBox, Vagrant, and the ubuntu/trusty64 template took less than 15 minutes, and we have bind(9) running a few minutes later.

One of the consequences of choosing to use Vagrant is: there's no graphical interface in the virtual machine. Even if we install Lubuntu, the text editor won't be his favorite. After trying to install sshfs, I realized that that's totally unnecessary: Vagrant has the /vagrant shared folder that is stored on the host machine and accessible from the guest machine, so Ruoyu could use his favorite editor to program in the shared folder, and execute them in Ubuntu. I pulled out pen and paper, and wrote down all the commands he would need to copy files between bind9 configuration folder and /vagrant.

Mark, our third team member, showed up around 22:00. It's evident that he has some experience. He got his assignment of writing the frontend webpage, and said he'll push up the code when he finishes.

How I missed the dinner

Dinner is to be served at 20:00. I was busy helping Ruoyu setting things up at that time, so I decided to eat at 20:30 while he's downloading. At 20:30, I saw a line of over 30 people, so I decide to eat "later". When we finished the setup, it's already 21:30. I'm glad to find that the line disappeared, but then I'm told there's only salads and breads left. Thus, I had a big plate of salads and a few slices of Italian breads for dinner. There's no protein. And I'm still hungry.

This is another bad decision I made. Since this incident, I learned a lesson: no matter what, go for food on time. After all, I signed up to attend Hack Arizona "just for free food".

The sleep

I put health as a priority. I have decided to maintain normal sleep hours, at the time I signed up. So I'll do just that.

I got back to my apartment at 22:30, cooked a plate of dumplings to fill my still hungry stomach, and went to bed.

With the green wristband still on me, I'm obviously too excited to sleep. During the 7 hours in bed, I only had around 3 hours of effective sleep. In the other 4 hours, I'm reading technical documents on the smartphone, to prepare for the second day of work.

I also feel thirsty during the entire night, and had to get up for water frequently. There's probably some kind of dehydration going on. I decide to drink more water during the second day.

TraceArt day 2

I arrived at the library just after 08:00 on Saturday. Breakfast is served, but there's no protein, and I have to balance all these fruits on a tiny little plate. I had my only cup of coffee during the entire Hack Arizona 2016 at this breakfast.

I brought my primary laptop. But when I enter the library, I went straight to the library computers: they have dual screens! As I said before, my productivity is the square root of the number of windows I could keep visible on my workstation.

Over a VNC session to my office computer, I started working on the most difficult component: simulating the network hops. The whole project would be down the drain if I cannot get this component working. The RFC 1071 checksum is again a headache, even if I got it right when I took the network class in 2011. But after I change the code numerous times, Wireshark stopped complaining "checksum incorrect", and traceroute6 is showing correct IPv6 addresses.

Ruoyu showed up at 10:30. Despite being a beginner, he's a quick learner, and can finish each programming assignment in an hour or less. Of course, I'm breaking down the programs into small and easy pieces, sometimes at the cost of efficiency. Both of us are happy with his achievements.

Neither of us knows how to accept HTTP requests in Python. I wrote a PHP script that invokes his Python scripts on command line.

The teammate that disappeared

As Ruoyu and I are working on the project, we still haven't heard from Mark or see his commits on GitHub. He's not answering my phone calls. He's not replying my emails. He eventually sent me a Facebook message saying he's "finishing up" with another project, but didn't confirm when he could come.

I'm well-rounded and previously wanted to work solo, so the disappearance of our frontend guy won't kill my project. HTML5! jQuery! Regular expressions! I know all these. No sweat.

But I don't know how to make a logo. How about, the photo of a handwritten logo?

TraceArt: the submission

We completed TraceArt by 17:00. It's singing Jingle Bells:

sunny@vps2:~$ traceroute6 -m255 000020.u.traceart.yoursunny.cn
traceroute to 000020.u.traceart.yoursunny.cn (2001:470:1f11:4a0:5441:7274:0:2012), 255 hops max, 80 byte packets
 1  2602:fff6:1::1 (2602:fff6:1::1)  0.169 ms  0.096 ms  0.091 ms
 2  2607:fcd0::602c:b465 (2607:fcd0::602c:b465)  0.228 ms  0.209 ms  0.210 ms
 3  10gigabitethernet1-3.core1.lax1.he.net (2001:504:0:3::6939:1)  36.294 ms  36.640 ms  15.334 ms
 4  100ge11-1.core1.lax2.he.net (2001:470:0:72::2)  5.513 ms  5.681 ms  5.631 ms
 5  10ge14-1.core1.den1.he.net (2001:470:0:15d::1)  31.115 ms  31.109 ms  31.003 ms
 6  10ge4-3.core1.chi1.he.net (2001:470:0:1af::1)  57.915 ms  56.582 ms  56.331 ms
 7  tserv1.chi1.he.net (2001:470:0:6e::2)  54.434 ms  52.038 ms  59.855 ms
 8  yoursunny-1-pt.tunnel.tserv9.chi1.ipv6.he.net (2001:470:1f10:4a0::2)  61.763 ms  51.369 ms  51.203 ms
 9  Dashing.through.the.snow (2001:470:1f11:4a0:5441:7274:0:2001)  57.596 ms  57.497 ms  57.451 ms
10  In.a.one.horse.open.sleigh (2001:470:1f11:4a0:5441:7274:0:2002)  57.848 ms  57.791 ms  57.737 ms
11  O.er.the.fields.we.go (2001:470:1f11:4a0:5441:7274:0:2003)  57.113 ms  57.071 ms  51.021 ms
12  Laughing.all.the.way (2001:470:1f11:4a0:5441:7274:0:2004)  51.620 ms  51.670 ms  49.991 ms
13  Bells.in.bobtails.ring (2001:470:1f11:4a0:5441:7274:0:2005)  51.189 ms  51.165 ms  61.755 ms
14  Making.spirits.bright (2001:470:1f11:4a0:5441:7274:0:2006)  61.475 ms  62.190 ms  62.163 ms
15  What.fun.it.is.to.ride.and.sing (2001:470:1f11:4a0:5441:7274:0:2007)  51.176 ms  51.432 ms  61.497 ms
16  A.sleighing.song.tonight (2001:470:1f11:4a0:5441:7274:0:2008)  50.353 ms  60.957 ms  50.235 ms
17  Jingle.bells (2001:470:1f11:4a0:5441:7274:0:2009)  51.618 ms  49.736 ms  49.659 ms
18  Jingle.bells (2001:470:1f11:4a0:5441:7274:0:200a)  51.797 ms  52.072 ms  51.619 ms
19  Jingle.all.the.way (2001:470:1f11:4a0:5441:7274:0:200b)  49.665 ms  50.589 ms  61.364 ms
20  Oh.what.fun.it.is.to.ride (2001:470:1f11:4a0:5441:7274:0:200c)  61.785 ms  51.801 ms  49.990 ms
21  In.a.one.horse.open.sleigh (2001:470:1f11:4a0:5441:7274:0:200d)  61.112 ms  49.460 ms  60.608 ms
22  Jingle.bells (2001:470:1f11:4a0:5441:7274:0:200e)  50.065 ms  52.629 ms  60.307 ms
23  Jingle.bells (2001:470:1f11:4a0:5441:7274:0:200f)  59.621 ms  51.533 ms  59.480 ms
24  Jingle.all.the.way (2001:470:1f11:4a0:5441:7274:0:2010)  59.049 ms  51.268 ms  61.524 ms
25  Oh.what.fun.it.is.to.ride (2001:470:1f11:4a0:5441:7274:0:2011)  51.757 ms  51.136 ms  60.668 ms
26  In.a.one.horse.open.sleigh (2001:470:1f11:4a0:5441:7274:0:2012)  49.724 ms  50.690 ms  59.996 ms

Submission to DevPost is not as easy as pasting a link. It needs screenshots, story, and description of each teammate's contribution. I'm glad that we are not doing the submission in the last 5 minutes before deadline.

#TechTalk

When most attendees are busy working on their projects, the organizers / sponsors / mentors are giving presentations about various technical topics, for those hackers who wish to have a break. I attended two tech talks: Intuit QuickBooks and Ionic Framework.

Frankly, these tech talks are pretty useless. They give me an impression on how an API / framework looks like, but it's no better than reading the documentation by myself. And it would be too late to incorporate the API / framework into a project, given that most projects are already half done on day 2.

The second project

After dinner and submission, I left the library at 19:00, and went to bed before 22:00. I had a nice 6-hour sleep. And I'm awake at 04:00 on Sunday.

I still have my wristband on, and there are still 4 hours before the 8am deadline. Why not build another project?

And I did. Another home surveillance camera, integrated with Cisco Spark. I hook up a webcam on my 11-year-old laptop, set up motion detection, have motion(1) invoke my script which sends pictures of "thieves" to the Cisco Spark app.

Admittedly, this project is not completely spontaneous. I have been running a home surveillance camera for two weeks already. It allows me to see the pictures from anywhere, through my NDN-based HomeCam app. It has motion detection, which writes pictures onto the disk. But it does not have notification capability when someone "breaks in" my apartment.

When I learned about Cisco Spark during the check-in, the idea of sending notification through Cisco Spark platform has planted a seed in my mind. So I built it in 3 hours, and spent the last hour writing the submission. It's named "HomeCamSpark"; it's not a smart name.

Last breakfast

The last breakfast is served on 08:00, just after the submission deadline. Burritos! This is the best meal of Hack Arizona 2016. And there's no line at all.

I walked around the library, and found many chose sleeping over the damn good breakfast. Some have stayed up for 36 hours, so they deserve this 1-hour nap before reporting to the project expo.

And that's the last chance to grab swags:

  • 2x Major League Hacking stickers
  • NameCheap shirt
  • NameCheap sticker

Two tables at project expo

The project expo is hosted in Student Union Memorial Center Grand Ballroom. Rows of tables are lined up in the room. Power strips are under the tables. This must be the hard work of Hack Arizona volunteers.

I'm informed that I can only demo one project, which I protested. Both projects are my kids, and I can't kill either of my kids, even if it's built in 3 hours. Then I learned that each project is assigned a table, judges are looking for projects by table number, and I can't demo both projects because I can't be at two tables at the same time. What if I can?

So they give me two tables. They are not consecutive, because table numbers are pre-assigned without considering who submitted the projects. But they are not too far apart, so I can manage that.

I'm glad that I brought enough computers and phones, so that each table can have separate devices. I'm glad that I brought a power strip, because there aren't enough power strips for everyone, and one of my laptops doesn't have a battery and must keep plugged in. I set the two tables up, and called Ruoyu to come in quickly to stand at table 3 for TraceArt, while I roam between table 3 and table 56 for HomeCamSpark. This science fair is on!

Demo not working?

Murphy's law says: anything that can go wrong, will go wrong. The demos aren't exceptions.

  • At some point, TraceArt fails to provision the latest lyrics submission, but all earlier submissions are working. Adding sleep(1) somewhere in the backend fixes the problem. I guess my VPS is too slow.
  • HE.net Looking Glass is blocking our IP address after running too many traceroutes on its web interface. We are able to continue the demo by SSHing into a server with native IPv6 where we can run traceroute6 directly.
  • The 11-year-old laptop for HomeCamSpark decides to disconnect from WiFi suddenly. I spent 20 minutes trying to reconnect, during which one of the judges came and said "demo not working?" then left with a promise to "come later", but he never came again.

We demoed TraceArt to over 15 patrons, including judges and the public. I demoed HomeCamSpark to about 10 interested viewers.

The end

Project expo and judging closed at 12:15. There are plenty of leftover breakfast, for my lunch and dinner. I slept 2 hours in the afternoon, which I believe is much less than most other attendees.

I read each and every submission in the evening, and casted my vote.

Winners are announced the next morning. I won nothing. That's okay, I'm not frustrated.

I'm an unique Hack Arizona 2016 attendee:

  • I have probably the cheapest laptop, an RCA Cambio W101.
  • I have probably the oldest laptop, an DELL INSPIRON 600m from 2005.
  • I'm the only one working on a library computer.
  • I'm the only one who submitted two projects.
  • I did not drink any Red Bull, and had only one cup of coffee.
  • My project is the only one serving webpages over TLS among public websites. All others (except GitHub) use insecure HTTP.
  • My project is one of a few that has proper commit messages in GitHub repository.

And I've learned a lot:

  • It's not scary to have a team.
  • Having a beginner on the team can be productive, if assignments are simple enough, and some guidance is given.
  • Don't be frustrated when someone disappears. You don't pay them, so you can't force them to work for you.
  • It's possible to build a cool project in 3 hours.
  • Food, water, and sleep are important, and they should be enjoyed on time.

Will I attend Hack Arizona 2017?

Very likely.

Tags: hackathon life