How to make GIFs of sites using WayBackMachine

So… I like following fivethirtyeight’s interesting 2016 Election Prediction page. It shows the ups and downs and the general mood of the election. I’ve been staring at it for so long that I wanted to collect the daily changes and make a nice GIF. I know the Internet Archive’s WayBack Machine collects archives of popular websites, so I went there and found that the Election Prediction page is on there too.

So, I started looking for ways to make a GIF from the WayBack machine. There were some node and ruby scripts and applications which didn’t really work. But then I landed on waybacklapse. Its developer – Kyle Purdon – works for bitly and has built two versions of waybacklapse. The older one is python, node, imagemagick and then some. The newer one is python3 and docker. Eww. I followed the steps of the tutorial for the older version, with a few notable exceptions –

  1. The tutorial is for OS X and is a little dated. What I have on hand is an Ubuntu 15.04 VM, so I went ahead and used apt-get install instead of brew
  2. The tut tells you to use the command “git checkout -t v1.1.0”, but it should be “git checkout -b v1.1.0”. Technically v1.1.0 is a tag, not a branch, but I didn’t know that and just used -b, which worked, so why mess with a good thing, amiright?
  3. You need to have node installed, but not the new node. Install old node with “apt-get install nodejs-legacy” and use the command “nodejs app.js” when you’re running screenshot-as-a-service
  4. The tut doesn’t mention that you need to actually *run* screenshot-as-a-service. I went to the github page for the service and found out that I need to run the above “nodejs app.js” command in order to run a server on the localhost. Technically, waybacklapse has code in it to warn you that the server isn’t running. But that didn’t work so well for me.
  5. The user prompts for waybacklapse only allow for monthly or yearly snapshots. But fivethirtyeight has only been running the site for about 3 months, with daily updates, so those didn’t make sense to me. I wanted to get all the changes. So, after installing waybacklapse with pip, I went ahead and modified the code inside /usr/local/lib/python2.7/dist-packages/waybacklapse/waybacklapse.py with one small change to get all the screenshots instead of just monthly or yearly ones –
    1. In the create_payload function, I commented out the collapse variable as follows –

All was well and good, but not really. Turns out, screenshot-as-a-service pulls a screenshot of the entire page, not just above the fold. Which is great, and not so much. I was looking at a GIF that was way too long to be palatable. So, I needed a way to extracts parts of the screenshots so I could make a nice, clean and small-ish GIF. Luckily, waybacklapse made me install imagemagick. So I looked around and made the following script.

It must sit inside the screenshot folder. It parses through the screenshots and converts them into smaller versions of themselves. Finally, I found the command inside waybacklapse which creates the GIF. I modified it a bit and used it to recreate the GIF.

convert -delay 30 /root/fivethirtyeight/2016081011081470853418/final-*.png /root/fivethirtyeight/2016081011081470853418/timelapse/2016electionforecastss.gif

Now, I could go about changing waybacklapse and submitting the code to the author, but he’s moved on to docker and in-house solutions for the dependencies, so I doubt it’ll be a benefit to anyone. Instead, I’ll just leave these notes here so I can reference them in the future. If they helped you, shout out in the comments section. Oh, and I’ll leave you with the GIF I made. –

FiveThirtyEight's Election Forecast in a GIF

Tech Bloggers should sell their articles

tl;dr – Tech bloggers should sell articles to News companies, much like Reuters and AP have done in the past.

GigaOm Pro is an interesting service. While most of the tech blogging industry is ad-supported, it has a subscription based model. It’s not unheard of, but surprises me that it works. I don’t have any numbers, but the fact that the industry pays good money to read and listen to GigaOm Pro analysts seems to point to a healthy business model. Why does GigaOm Pro work? Because it is attached to a name, that of Om Malik. That name carries weight in tech reporting circles. People care about what he has to say.

Why is this relevant? Because good writing is always rewarded. Newspapers around the world have always depended on wire services like Reuters, AP and BBC, to name a few, to fill their columns and inform their readers of news from around the globe. This makes sense for two reasons – it is cost-effective and companies such as Reuters and AP can be trusted to do the right reporting in a timely and impartial manner. I have seen news reports being replicated in different newspapers without any change simply because the headline was followed by a reporter’s name and a small (Reuters) mention.

Today, the Internet, in its goal to be an open sharing platform, has skewed this reporting standard. Companies such as ABC News and NYT often quote a blog post or a tweet and wrap their own story or analysis around that. This works so long as the original writer isn’t a paid professional writer, like when I write on this blog. But when the quoted links are tech blogs and independent writers who lose out on page views; and thus ad revenue, this becomes a lousy proposition for the bloggers.

I believe that tech blogs should become similar to Reuters and AP in their reporting. Traditional news companies cannot afford to send reporters to every tech event, nor are they invited to do so. Tech blogs have reached the level where if you’re not writing well or covering the latest topics, you’ll get laughed off the Internet quicker than it takes to set up a Tumblr blog. So, instead of keeping in-house writers, news outlets can take articles from tech blogs for a fixed price, along with deeper analyses and more contextual content. They could do this without compromising the quality of the writing. Not every tech blogger would qualify for a payout, but those who have proven their worth will be able to earn more than just page views on such a program.

Why would this work for news companies? Right now it’s a free-for-all. They can quote anyone, attach a link and bet that no average reader will click on it, all for free. The benefit of paying up? Syndication. Right now, tech news trickles down from everyone who was on the scene and this means that everything links back to the tech blogs. By offering to pay for their writing, media companies will be able to get their hands on exclusive content without having to link to any blogs and without having to worry about legal issues related to trackbacks.

The negatives for tech bloggers are not negligible. The first is that many companies will require bloggers to officially become journalists to protect them in case of First Amendment issues. This will require that the following question be addressed – is a blog post talking about a recent event a news report or an opinion? I myself believe that it is an opinion, since the writer does not necessarily have the required training to be impartial to the subject. This debate has often played out against bloggers. The other issue is that upon paying money, the company will require the blogger to not publish the article on their own blog, as this would duplicate the syndicated article. This hits the blogger exactly where it hurts. Tech bloggers write on their own platform for the single purpose of gaining popularity; which can then be converted into money using the traditional means of sponsored posts or advertisements. The work around would be for tech bloggers to either accept less money in exchange for permission to post the article on their blog after a fixed period of time or write a short article on their own blog while syndicating a longer, much more detailed version for the news media. The final problem is that this move would take us away from the open nature of the Internet where RSS feeds and pingbacks allow a level of sharing that doesn’t exist on any other media platform. I do not know how negatively this will affect the Internet, but it would not be a small change.

In conclusion, I believe that tech bloggers should strike deals with traditional news media to provide them with syndicated news feeds. This can mean that tech bloggers do not need to depend solely on advertisements on their own platform to earn a living. This will benefit news media outlets because they will get accurate, real-time news and analysis from people who are in the field and understand the context very well. There are still a lot of issues that need to be resolved before such a step can be taken, but the idea is worth looking into.

Interesting sub note – The Wikipedia page for Syndication lists print syndication as “where individual newspapers or magazines license news articles, columns, or comic strips” but web syndication as “where web feeds make a portion of a web site available to other sites or individual subscribers”. In web syndication, there is no mention of licensing, money or purchase of rights. This is what Riptide has concluded to be the problem with news reporting on the web.

Editor: Anna Tarkov [ADN|Twitter]
Note: I’d love to thank Anna for her help in editing this article. She provided me with invaluable help in getting the message across and pointing out some of my idiosyncrasies. Anna is a journalist from the Chicago area and she’s steeped in the digital life. She runs a personal blog here.

Save yourself from the Ephemeral

As users of the Internet, we change a lot. We move email IDs, we jump from one social networking fad to another, we change bookmarking and read-it-later sites and even crash, delete or just forget blogs that we write on.

Most of the stuff I’ve done in the past 10 years or so on the Internet has been pretty personal. Emails, Orkut or Facebook where privacy settings allowed me to block external users or bookmarking sites that were private by default. But recently, most of my contribution to the Internet has been public – twitter and App.net, my blogs and even my bookmarking has been public. So is true for most of us out there. With the shift in social networks’ view of what data should be totally private, there’s a lot of data that’s in the public domain. This also means that there’s equally that much data that can be lost or can stagnate when an eventuality occurs – a web service shuts down because of acquisition or drying up of funds, your blog crashes and you have to start from scratch, you leave a social network and even though you download all your data and invite all your connections to the new one, some don’t join or you can’t upload any of that data anywhere else (how many social networks out there are interchangeable? None.) or maybe you just stop using a site or service and that data just sits there, alone and forgotten (just ask my bookmarks on del.icio.us). Continue reading

The Three Blind Men

Recently, I was in a shopping mall heading towards McDonald’s for a dinner. As I walked past a showroom, I noticed an odd occurrence. Three Men, holding each other by the arms were walking in a straight line. One look and I had dismissed this sight. But then I paused and looked at them again. Here were the three blind men, white sticks in hand, goggles covering their eyes at 9 in the night, walking steadily in the middle of the mall.

I went over to them and inquired as to where they were headed. They said they wanted to go to the loo and the lead amongst them confirmed from me that it was in  the direction ahead. I corrected him by saying in a matter of fact way that he would have to turn to the right after walking down a bit further. He thanked me and started heading out in the initial direction. Instantly, it occurred to me that in the most insensitive way, I had told them the directions but not the distances. I latched on to the first one and took them to the corner where they had to turn right and pointed them on to start walking in the said direction. Again, they thanked me and started off.  Satisfied that I had done a good job, I walked off but stopped before having gone more than a few steps. I cussed at myself for being that stupid and ran back to them in order to guide them further to the exact door. When I reached, they were fumbling around a fire escape, looking for the correct door. Many onlookers were passing by them, surprised to see them try to find their way in the maze of objects and paths we “sighted” people take for granted. Before I could reach though, another fellow had arrived and helped them get to the restroom. I realized as I looked from a distance that it was the lead who alone had to go to the loo but the other two had no option than to follow him. They stopped at the door and waited for him to return.

As I walked off, a thought came to my mind. As the world around us progresses and technology allows us to become greater than ourselves, the society we live in does not change, the plight of the people does not change and the conditions which affect us do not change. We may be running after cures for cancer and aging but the blind man still cannot see and the deaf can still not hear. There is technology to enable them in doing these too, but that technology is not available to the present at a price which does not hit the pride of the man bearing the cost. Those blind men were not in rags but seemed suitably dressed. A poor blind man would have been stopped at the doors of the mall and forced to look for a loo elsewhere but the guard let them in and guided their path too, in his own insensitive way. That goes to say that the latest gadgets which help the blind(no euphemisms here, say it as it is) are within their reach but not there yet.

As a tech enthusiast and a software programmer, I can say that anything is possible in the world of technology. As a hardware designer and a practical man, I can say that we are not looking at the right ventures. I can blame many for not following up on this, from Steve Jobs to the Indian Government, but till some brilliant yet cheap technology comes into the possession of these disabled people, let’s make sure that the three blind men you see on the road next time reach home safely. Let us make sure that the stark contrast between consumerism of the most blatant kind and the simple reality that life has not yet changed despite Science’s greatest contributions be diminished by Humanism, for that too is as simple as extending your hand and guiding the path of those who cannot see.