How to make GIFs of sites using WayBackMachine

So… I like following fivethirtyeight’s interesting 2016 Election Prediction page. It shows the ups and downs and the general mood of the election. I’ve been staring at it for so long that I wanted to collect the daily changes and make a nice GIF. I know the Internet Archive’s WayBack Machine collects archives of popular websites, so I went there and found that the Election Prediction page is on there too.

So, I started looking for ways to make a GIF from the WayBack machine. There were some node and ruby scripts and applications which didn’t really work. But then I landed on waybacklapse. Its developer – Kyle Purdon – works for bitly and has built two versions of waybacklapse. The older one is python, node, imagemagick and then some. The newer one is python3 and docker. Eww. I followed the steps of the tutorial for the older version, with a few notable exceptions –

  1. The tutorial is for OS X and is a little dated. What I have on hand is an Ubuntu 15.04 VM, so I went ahead and used apt-get install instead of brew
  2. The tut tells you to use the command “git checkout -t v1.1.0”, but it should be “git checkout -b v1.1.0”. Technically v1.1.0 is a tag, not a branch, but I didn’t know that and just used -b, which worked, so why mess with a good thing, amiright?
  3. You need to have node installed, but not the new node. Install old node with “apt-get install nodejs-legacy” and use the command “nodejs app.js” when you’re running screenshot-as-a-service
  4. The tut doesn’t mention that you need to actually *run* screenshot-as-a-service. I went to the github page for the service and found out that I need to run the above “nodejs app.js” command in order to run a server on the localhost. Technically, waybacklapse has code in it to warn you that the server isn’t running. But that didn’t work so well for me.
  5. The user prompts for waybacklapse only allow for monthly or yearly snapshots. But fivethirtyeight has only been running the site for about 3 months, with daily updates, so those didn’t make sense to me. I wanted to get all the changes. So, after installing waybacklapse with pip, I went ahead and modified the code inside /usr/local/lib/python2.7/dist-packages/waybacklapse/waybacklapse.py with one small change to get all the screenshots instead of just monthly or yearly ones –
    1. In the create_payload function, I commented out the collapse variable as follows –

All was well and good, but not really. Turns out, screenshot-as-a-service pulls a screenshot of the entire page, not just above the fold. Which is great, and not so much. I was looking at a GIF that was way too long to be palatable. So, I needed a way to extracts parts of the screenshots so I could make a nice, clean and small-ish GIF. Luckily, waybacklapse made me install imagemagick. So I looked around and made the following script.

It must sit inside the screenshot folder. It parses through the screenshots and converts them into smaller versions of themselves. Finally, I found the command inside waybacklapse which creates the GIF. I modified it a bit and used it to recreate the GIF.

convert -delay 30 /root/fivethirtyeight/2016081011081470853418/final-*.png /root/fivethirtyeight/2016081011081470853418/timelapse/2016electionforecastss.gif

Now, I could go about changing waybacklapse and submitting the code to the author, but he’s moved on to docker and in-house solutions for the dependencies, so I doubt it’ll be a benefit to anyone. Instead, I’ll just leave these notes here so I can reference them in the future. If they helped you, shout out in the comments section. Oh, and I’ll leave you with the GIF I made. –

FiveThirtyEight's Election Forecast in a GIF

On A Question of Ethics

This happened a few days ago but I have been delaying writing on this topic primarily since I had requested my friend Shyam Chandrasekar to write on it as a guest speaker on my blog. He has refused my offer and hence, I must write it myself. The reason why he should have written it is that the incident largely involved his resolve to do the right thing.

Few days back, upon a strong craving of ice cream, I and a group of friends had hit Boulder’s Pearl Street to find and plunder the tasty Glacier Ice cream company’s stall. We found a girl managing the stall and asked her for various flavors and made our order. She completed the delivery and whipped out her cellphone to tally the bill and tell us the whole amount but I insisted that she write it down on paper as one of us was going to pay and then everyone would give their share back to that person…

She agreed to the demand and looked around helplessly to find some paper to write on. I took a pen out of my pocket and gave her some tissue paper from her own stall to write down the bill on. She quickly summarized a bill of about $19.40 and I took the money from our today’s bakra(We still need to give our share of the money back to this person!!!) and handed a $20 bill to the girl. She casually commented that it was so late and she was so tired after having studied at the University that her mind was just not working. Having said this, she gave me the change and thus ended our transaction.

As we walked off, my friends enjoyed a nice, cold, tasty ice cream treat at the expense of our bakra while I wondered as to who it really was at whose expense we were really enjoying! You see, there were a total of 6 people in my group and most of us had opted for a single scoop of ice cream with a waffle cone which is slightly costlier than usual. So a total of under $20 was just quite not right. As I pondered over this, my friends chided me for going into deep thought again, as is my habit and asked me to join them in the conversation. I did that, but before doing so, I called Shyam to one side and told him about this dilemma which we now faced. I did not know the correct prices of a waffle cone but I knew that we had under-paid the woman by a huge amount. Shyam too expressed his suspicion of the same and having been more resourceful, quickly summarized the actual bill. We found that we had to pay this vendor a total of $28, that is, we had paid her $8 less!

Now it was time for either action or debate. While Shyam was more interested in the former, I chose the latter and brought the matter out to my friends. I told them that it was a matter of not paying what was due but also reminded them that it may be eight dollars for someone but it’s nearly Four hundred rupees for us! Most of them chimed in with the opinion that we should not pay the money back and someone even went on to say that we’re already paying through our nose just by being here for our studies so why not save some money while we can! While I was enjoying this debate and the reactions of my peers, we were joined by my friend and roommate, Abhjit Kaul, who had till now been talking to his family members on phone. When we told him about our situation, he instantly voted in favor of returning the money, pointing out the most important thing which tipped the scales of the discussion – that it may be eight dollars for us, but it’s an hour’s work for the girl!

That did it. Those reluctant were now quiet and Shyam quickly went back to present the money to the girl. He did not return for quite some time.

Oh, he did return eventually and told us triumphantly that the girl had been nice enough to give him a dollar off on the bill for his honesty!

Thus nearly ended a night when we had a taste of good ice cream and crowd mentality. We saw that Gelato is good and greed is bad. Finally, we saw that it’s not the amount of money which matters, it’s the amount of work which goes into making that money which counts and this reminds me, I better talk to our bakra and give back the money! 🙂