Reuters takes offense at hacked apps in iOS

It is unclear how much revenue the pirate distributors are siphoning away from Apple and legitimate app makers.

Source: Software pirates use Apple tech to put hacked apps on iPhones | Reuters

It’s taken a long time and another massive Facebook privacy scandal for the news media to discover this underbelly of hacked apps chugging along happily due to Apple’s Enterprise Apps program.

I’ve used one on and off – Instagram++

I must say, it’s a liberating experience – I see no ads on Instagram, I see no random “Suggested Friends to Follow” crap.

I had to resort to this because my Instagram experience is vastly worse off than my wife’s and my friends’. I see, on average, 3x more ads on Instagram than others around me. How many ads does my wife see? None.

So to my mind, using Instagram++ makes perfect sense. If I can hack my way to a better UX, why shouldn’t I? It’s the same as using an adblocker.

I don’t support piracy of services. There’s no legit reason to not pay for Spotify.

As for hacked games, well, cheats and hacks have always existed, and will continue to exist, despite the alarmed voice of this Reuters article.

Also, the article got one thing wrong – I’ve observed Apple kick out the Enterprise cert almost once a month, sometimes two or three times a month. They seem to make it sound as if Reuters alerting Apple was the only thing that forced Apple into action.

They’re very much aware of the problem and can’t or won’t do much about it. Talking about it as if it’s the end of the App Store is just noise.

As for how much revenue these services generate? Not close to enough. They do seem to have a comfortable existence, and so might be able to get around Apple’s 2FA proposal by just buying a bunch of phone numbers in China. But do they run a massive profit? You bet that if they did, Apple would be all over them.

This is the same as the jailbreak community in some senses – only a small percentage of users are actually trusting these services not to misuse the extensive powers that Enterprise certs give them. Out of that small percentage, a further small percent is paying for it.

It’s sad that large companies like Facebook pulling the shit that they do often also bring to light little players that are just trying to provide a good service to users.

Now, the technical aspect of this – Instagram++ is available online for download as an IPA if you want to use your own developer account. If you don’t have a dev account, Apple now allows side-loading, but it is a cumbersome process that expires after 7 days. Apple’s earlier sideloading used to be 30 days. When Apple made it free for everyone to sideload (not just if you’re a $99/year paying developer), they reduced the time frame of the cert to 7 days, which in my mind is a total d*ck move.

If Apple really wants to combat Enterprise cert misuse while letting users do whatever they want with their systems, they can just legitimize sideloading and let me choose when my cert would expire, but Apple isn’t that generous.

Till a good solution presents itself, services like TweakBox, Tutu, and AppValley will continue to operate by hook or by crook. So be it.

The Original iPad mini and Apple’s fluid vision

It is meaningless, unless your tablet also includes sandpaper, so that the user can sand down their fingers to around one-quarter of the present size. Apple’s done extensive user-testing on touch interfaces over many years, and we really understand this stuff. There are clear limits of how close you can physically place elements on a touch screen before users cannot reliably tap, flick, or pinch them.

Source: A Look Back at the Original iPad mini – MacStories

 

It seems like Steve Jobs and Apple understood that you can’t place things too close inside the screen, but forgot that you can’t place the screen and the edge too close either, because it’ll cause hours of headaches by unwanted swipes, taps, and hard presses. The Apple of today thinks bezels are bad and it is wrong. Steve Jobs might have said the above, but he’s also the one constantly touting that they made their devices thinner, which reduces battery life and also the ‘holdability’ of mobile devices.

(Proof of the holdability issue – When was the last time you held your iPhone 7 or above naked, without a case, and felt confident that you’ll not drop it? It’s been months for me and when I did it last night, it felt alien. It seems Apple has outsourced the job of holdability to the cases that we inevitably put on our sold-a-kidney-for devices.)

I love my original iPad Mini and still use it. It’s a very well built device. The iPads of today make me feel like Apple just wants to make the jump to touch ‘computers’ instead of trying to keep the iPad what it is – a touchscreen tablet that feels different than anything else on the market.

The difference between a touch computer and a touch tablet? The former, you keep on your desk and work on using a keyboard (think Surface). The latter, your kids hold while they’re watching YouTube videos on in the car.

But this gives Apple a great new diversification strategy – do you want an iPad to work or an iPad to play?

Till now, they’ve kept these two together. But maybe, bowing to market forces, they’ll break these two use cases apart and give us two iPads that do very different things. That’ll require Apple to stop treating the iPad like it’s just the overgrown brother of the iPhone. Let’s see if they do that.

p.s. With iOS 12, if Apple is truly committed to making software releases that don’t completely destroy older devices, that’s also relevant to corporate uses. Companies don’t keep updating everyone’s hardware every two years ‘because the software got old’. So if Apple wants an iPad on every office desk (as they should), they really need to get their software updates game right, which they seem to be on a path to.

p.p.s I was going to call the title “Apple’s faltering vision” (because clickbait!) but Apple’s vision is rather fluid. If they see a market segment responding well, they go after that, instead of doubling down on losing segments like some other companies do.

Security vs Usability

I’ve come to a point where I do **not** update apps, plugins, software in general. I know that’s a regressive approach to safety, but safety can’t keep trumping usability all the time.

Source: My comment on Stephen’s Notebook

 

Every few days, I have a conversation about security vs usability somewhere. With my iPad Mini, I blindly trusted Apple to do the right thing and they’ve screwed me over. It’s a beloved device, destroyed completely by iOS 9.

So I’ve basically given up on this bullshit harp that companies sing of ‘security’ to shove software updates down our throats. Sometimes it’s their stupidity, and sometimes it’s just them being sinister. The new Microsoft is the old Microsoft. The benevolent Apple is an insidious Apple. Don’t get me started on Facebook, twitter, and Google. Gmail is just the latest casualty of our overzealous overlords.

Yes, security is a big problem. Yes, it needs constant vigilance. But just like national defense budgets, one key phrase doesn’t allow organizations to completely railroad people’s expectations, asks, hopes, and in this case, UX.

If you’re concerned that by not updating software, you’re living on the edge, restrict the things you do on that device, while keeping other devices that are completely updated and secured. Use only frequently updated third party browsers instead of the default options. Read up on the latest security scares on the Internet and just be aware of the situations you can get into. But most importantly – back up. Make frequent backups of things you care about. I don’t care if it’s as much as letting iCloud run its course every night, and Google Photos siphoning off your pics. Just do it so that if you brick your device, or get hacked, you’re not set back a hundred years.

99% of security is just keeping your eyes open.

How do you like them upgrades?

Every few days, my iPhone politely but firmly nudges me to ‘downgrade’ my iOS from iOS 10 to iOS 11. I say downgrade because that’s what iOS 11 is to me – a crappy OS that was shoved out with half baked ideas which work well for the latest and greatest iPhone, but not at all for any other device Apple supposedly still supports. Getting rid of that prompt requires careful jumping through a confusing menu that makes it too easy to accept a “sure go ahead with this change at night when no one is watching” option. Most of the time, I am able to do just that. But last night, in a haze of trying to actually use my phone, I must have hit the wrong button, because when I woke up, my phone had restarted and was magically on iOS 11.4.1. Yay.

Before I talk about iOS 11, I just want to say why I didn’t want to get on it –

  1. It’s terribly built – simple features such as the ability to close apps quickly (in a few years time, Apple will reveal that just like their battery nonsense, closing apps DOES actually increase the speed of the phone, as empirically witnessed by a Bajillion people), the ability to turn off the wifi completely through the Control Center, the ability to actually use the phone for half an hour without draining the battery completely (my wife got on iOS 11 as soon as it released and she had the worst experience possible with that OS) were nice to have in iOS 10.
  2. I won’t be able to use all my apps – Apple, with iOS 11, waged a war on 32 bit apps. Now, most apps (99.9% I’d say) were smart about it and went 64bit, but I still have 4 apps on my phone, two of which I was using every few days till yesterday, which are 32 bit. So long Stress Baal and Sunstroke. You will be sorely missed.
  3. It will most certainly screw up my Apple Watch – I have a Series 0 (zero) Apple Watch. When will I buy the new one? Probably not for another few years. It’s a watch. It’s somewhat smart and lets me see messages and cut phone calls, but that’s about it. Do I need LTE? If AT&T pays me $15/mo instead of charging it from me, I might. But one minute into using the new OS, I was told to update my Watch from version 3.2.3 to 4.3.2 and told that if I do not, the phone will force unpair my watch and reset it. Thanks Obama. I exited the Watch app on my phone and plan on opening it at some point in the future. My watch is no longer getting notifications and isn’t able to send heart rate data to the phone (so much for Apple’s “we’re helping you take care of your health” crap. If the data collection is conditional, it’s not really helpful, is it?). But I know that watchOS 4 will screw up the watch, the third party apps, the battery usage. Basically, this is Apple’s way of making you buy a new watch. NO.

Now, coming to iOS 11. I immediately noticed that most apps seem to work differently – Google Maps had some new and interesting UI changes, Egg Inc had AR, the photos app had an irritating number of new features it had to tell me about before it let me use the app, the screenshots were showing up at the bottom (which is nice), etc.

Oh wait, backup. AR. That gleaming, new, awesome technology that’s changing the world! Yeah, I used it. For about 30 seconds. Then I was done.

Literally the only thing I could imagine using AR for – Egg, Inc. With that, my AR experience has ended. Well done, Apple.

Incidentally, I only recently watched this rather interesting video about how Apple will eventually launch AR glasses and they will be more successful than Google’s half-ass attempt because, well, Apple. It’s worth a watch 🙂 –

The rest of the stuff, is as I expected – meh. The app switcher can now close apps (yay!). The wifi stupidity that Apple propagated with iOS 11 is still there (so it’s always going to drain your battery no matter what). The animations and speed of launching apps is meh. Apple really wanted to make you feel something different, and well, I feel it, but I don’t care for it. It’s more a disruption than a nice addition. Plus, if you close an app that sits at the top of the screen vs at the bottom, the animation helps you see where the app is ‘going to’, but that’s really a rather stupid thing to care for Apple. I say that because I’m sure anyone who has as many apps as I do uses the search bar to get to apps instead.

Oh, yeah, that might be the silver lining – in iOS 10, I would swipe down, type out the name of an app I want, and the phone would just sit there, like a dunce, unsure of what I want it to do. Something was really borked in the code there and sometimes the search would work perfectly and other times it would go completely for a toss. Hopefully, that experience will be more consistent with iOS 11. If not, I’ll know that Apple did not even bother improving the Siri search code underneath and just dressed it in iOS 11 style. Typical Apple. Let’s see.

I’m no Luddite. I like experimenting with new stuff. But I really was hoping to go directly from iOS 10 to iOS 12. When iOS 12 drops, it’ll most likely not support my Series 0 watch. But at least it’s purported to be better than this monstrosity Apple threw our way. It’s OK to skip an OS, it’s OK to turn off auto-upgrades and auto-updates and watch your ‘to update’ App Store list burgeon to 197 apps. It’s OK to let the latest and greatest go while developers work on hardening releases. We all do it in some sphere of our lives. It’s just that my sphere was the one I’m staring at the most during my day – my phone. I want it to be consistent, familiar, and with less fluff. Sometimes people stick to a particular iPhone for a lot longer than they can, because they like the form factor and the materials used. Well, iOS 10 was that for me. But now my phone has moved past it. Time to adopt the new and shiny and see what changes this brings. Hopefully some nice AR filters.

Sourcing information

We all do most of our browsing on our phones. When we come across something we don’t know about, we google it to find out more. More often than not, the link that gives us the most information is either Wikipedia or a news site.

If it’s current affairs, it’s a news site. If it’s general information, Wikipedia. Then why do we still google the thing? Why waste time on the middleman? Is it force of habit? Is it because we believe that google will give us the most comprehensive information and links? Is it just laziness?

Perhaps it’s all of the above. Google is our one stop shop for all information. Whether we’re looking to buy something, looking for a website which we don’t often go to, looking for some news, or solving some mystery on the web, google will give you the knowledge you’re looking for. That’s a great product, regardless of any other implications on privacy, advertising, politics etc.

So why should we opt to change this excellent workflow? (Need information, ask google, get information)

Because it’s worth it to go to the source.

  • Google often scrapes data from Wikipedia, but most of the time, it’s incomplete. It’ll be the first line or paragraph in a topic that’s complex and needs some more study to understand. Or, google will tell you a part of the information, expecting you to select a link to learn more from. So why not go to the source directly?
  • When the topic is a current affair, Google will show you links that it judges to be of your interest, or of value to them (advertising, collaborations with sites like twitter which will be surfaced above others). Instead, if you go to a solution such as Apple News (or Google News perhaps) and search for the topic you’re looking for, you’ll see a more balanced perspective because all Apple News is doing is collecting links from various news sources and presenting those to you. Notice that I didn’t say you should go to a particular news site for this, because if you want real news, you’d better be looking at more than one source.

Now, how do we make this easier? How do we give up our google habit and go to the source? On mobile, the simplest way to do this is to move your apps around. On my phone, the Wikipedia app sits on the main home screen and the Apple News app sits inside a folder on the dock (most of the time, I end up searching for the news app on spotlight search, but I’m trying to get rid that habit too).

This is not ideal. In an ideal world, I would not have to go to each app individually to search for the topic at hand. I would be able to select a word or phrase and use the share sheet in iOS to jump to Wikipedia or Apple News, neither of which seem to support this simple functionality.

But those are the technical details, which may change at any time. What matters is where we source our information from and why. I recommend that you start cutting out the middleman and go directly to the sources, sites, and services that you trust, because those are the same ones your middleman trusts too. As for the why, well, start doing this and you’ll see a change in how you receive information and perceive the news. Search is good, but search algorithms may very well not be.

A note about people taking the time on the Internet

I read a very interesting post through one of the linkblogs I follow. This link, through the blog kateva.org talks about how Facebook is experimenting with linking Groups and Pages, the two ‘community’ offerings by Facebook with the use of Saas affiliate marketing software. I’m part of a few groups and a few pages (I’ve cut down on the latter a lot in recent years because it’s mostly noise) and I see real value in merging the two and creating a single entity that simplifies group interactions on FB.

But what was interesting to me was John Gordon’s comment on the link – “I miss blogs that used to explain things like this.” Of course, he’s not talking about the change FB is bringing but the blog he’s linked to. The comment resonated with me because there’s something along those lines that I’ve been thinking about since some time now.

When the Internet began, people started filling out blogs and sites talking about the most mundane of things – small changes in their favorite newspapers, versions of textbooks and differences between them, software and the differences between versions, events of their days, to name a few. These discussions were then swept up by sites who collected these minutae, stripped out all ownership information, and presented the collected works as their own. This has been acceptable practice and what certain sites are borne out of (cough cough). This practice both helps grow the Internet at an exponential rate and harms the original authors as their work and name gets lost along the way.

So people on the Internet slowed down. Content creation moved from everywhere on the Internet to either large syndications or small blogs or forums. The large swathe of users on the Internet became consumers. This is part of the problem for most social networks – when a majority of people are consumers, only a choice few are creating value. Thus is born the consumer’s content creation – likes and shares and retweets and reposts. These became the content of today. I don’t have a problem with this.

My problem is with the loss of the minutae. That value that was once created on blogs and static pages is now created on reddit and stackoverflow and obscure forums, if at all. That often means that the type of value creation that I (and Gordon) am looking for has just about disappeared. If no one asks the question on Quora or Stackoverflow, no one answers what the changes FB is making look like.

What I’m looking for is even more specific. I am often faced with a very difficult choice – whether or not to update a particular software. With apps, it’s much more difficult because we notice those changes quickly and it is almost a split second decision whether to update or not (click that button!). Further, there are so many apps that we use and so many updates that get pushed through that it would be draining to discuss what each update brings to the table and whether it is destructive in any way for any specific scenario. For updates on a computer, there’s still some open discussion one can find. People take these a little more seriously and often it’s easy to find information about version changes and impact on systems similar to one’s own.

Let’s take a few examples –

I recently updated to the latest version of the WordPress app on iOS. It was on a whim and I paid dearly for that. The new update brings the ability to manage plugins on your WP blog. But the update is borked. One of my blogs has well over fifty installed updates (not all are enabled) and when I go into that blog, the app crashes and then keeps crashing. I’ve seen no update for this issue in the last two days and haven’t bothered to write up a report to WP for it. I learnt after a few tries that if I don’t open that blog’s admin page from my app, the app doesn’t crash (letting me use the app for other blogs). Presumably this has something to do with not loading the plugins list from that site. I wouldn’t know, I’ve not explored the issue further. Funny thing is, if I’d have waited a little and read a few reviews, I still wouldn’t have come across this issue because people usually don’t blog about specific versions of an app and I’d have to trawl through a bunch of issues pages on GitHub to find some mention of the issue.

The other example I have is of a BIOS update. I have the option of pulling in this update and I know that if I want to go exploring issues around it, I’ll find at least a few pages talking about people’s positive or negative experiences around it. Why the difference? Apps affect our lives just as much as BIOS updates do, because they take up more of our time now than computers do. The only thing is that BIOS updates are infrequent and cause system-wide failure. Plus, the BIOS update has been out there for a while and if it had been problematic, I would be able to find more information about it, and is a big problem since people love to use computers, for work, game or even listen to music using the 5.1 computer speakers 2017 that give the best audio quality to any computer.

There’s a hundred other things associated with these scenarios which I’ve ignored to simplify them – iOS is a closed garden, so the number of users who get affected by an individual app are much fewer than from any BIOS update; app updates are now automated so people don’t even have this dilemma that I have; there is a lot of software out there no one talks about and I’ve not included in my examples.

(By the way, I feel Apple should go the WordPress.org way. It should allow people to report back on app versions with respect to iOS versions, to say that, e.g. “2000 people report 100% compatiblity with iOS 11.2 for version 1.3 of this app”. This will give both us and them so much more information about how stables apps and updates are.)

People have stopped taking the time to talk on the open Internet about changes that affect us all. That’s because the return on investment of time and effort is all but enough to warrant this approach to life – documenting every small change.

That’s somewhat sad, frankly.

Everything wrong with Google Play Music’s iOS app

I’m not much for introductions on such topics. The following is part kvetch and part bug list about the Google Play Music iOS app. It’s a crappy app with a lot of problems.

 

1. Oh Playlist, where art thou?

I like listening to reading music while I’m reading. So one day, I searched for such a playlist (aptly named “reading music”) and added it to my library, marked it for download and started listening to it. After that, I didn’t listen to it for a while and moved on to some other music.

The playlist effectively disappeared. The My Library tab has these options – “Recent Playlists”, “Auto Playlists”, and “All Playlists”. Once the reading playlist was no longer a ‘recent’ playlist, I assumed I’d find it in the All section. Nope, not there.

It’s not my own playlist. It’s a playlist that I’ve effectively subscribed to, downloaded, added to My Library (three separate actions they made me do to ensure I have easy access to the playlist). But if I don’t have ownership, does that mean it’ll not even show up in my library? That’s more horrible a design than Google AMP!

To this day, I don’t know where most of my downloaded playlists are. They’re consuming space on my phone but I don’t even know which ones they are, let alone have a way to play them.

In the end, I had to add the entire ‘reading music’ playlist I like to another playlist I created. That’s the only way to get it to show up in my own collection.

2. When everything is Search, nothing is Search

Google Play Music wants you to Search for everything. The Search button is prominent everywhere but it only does Universal Search. When I’m inside a playlist, it doesn’t search the contents of that playlist. I have a playlist called ‘all’. I dump all my favorite songs in there and then when I’m bathing and listening to music, I know I’m listening to stuff that I like. But every once in a while (once a day) I don’t want to start my music with the first song in the playlist (Taylor’s Look What You Made Me Do). So I go searching for some other song. I have to scroll through the entire playlist and hope to hit on the song I want at random.

Mind you, it took Spotify forever to add in-playlist Search. But isn’t Google supposed to be all about design and iterations and learning quickly? Oh wait, maybe I’m thinking about Facebook (hey google, look what you made me do)!

It’s a simple ask – add Search to your app in a meaningful way. Maybe since they don’t actually listen to anybody, they don’t know that’s an ask.

3. Integration, Integration, Integration!

At the bottom of my ‘all’ playlist is a section that invites me to watch YouTube videos of some of the songs on my list. It’s not a very smart offering – it doesn’t take into account my favorite music, just whatever they want me to watch videos of (I can hit the more button to see loads of videos that might interest me).

So I’m thinking, if they’re so tightly coupled with YouTube, that’s awesome! No. It’s not.

Search for a song that they don’t have and they’re point you to the YouTube video for it. Maybe. This service has gotten better over time but it still doesn’t point me to the right video for a lot of songs. Besides, what’s even the point of this? Do I want to watch the song on YouTube? No, I want to hear it on Google Music. If you don’t have it, just say so and move on! Instead, they show me related musicians, radio stations and then bring up the videos. Also, this brings me to the next one –

4. Competition, Competition, Competition!

Why am I on Google Music? My brother bought YouTube Red’s subscription and liked it so much that he was one of the first people to sign up for the YouTube Red family plan. He got me in and I’ve really started enjoying no-ads YouTube. Experimenting with the options available to us through this, we came to understand that it includes YouTube Music and Google Play Music premium subscriptions too. That’s amazing! But not.

What service would you rather use? Shitty Google Play Music or weird YouTube Music? YT Music is confusing and half-baked. It has nice video/audio modes and background play but it doesn’t support playlists. Google Play Music has tight integration with YT but not YT Music, and it opens the YT app for any video I click on, and starts autoplaying it. Convenient, but irritating. YT Music is essentially Google Music’s own competition and they’re both the worse for it. Neither service is usable. I understand there’s some licensing nonsense behind this, but hey Google, you’re GOOGLE. Getting your way with licensing should be second nature.

5. Tabs. Such useless tabs.

You know what I do when I open the Google Music app? I go to my “Library” tab and look for things manually. If I can’t find them there, I search for them using the universal search. You know what I do not use? Every other tab in the app.

The Browse tab – It has three options – Top charts, New releases, Browse stations. None of these interest me because they’re not suited to my taste. They’re generic.

The Recents tab – what’s the point of this? The Library tab has a Recent playlists section. That’s pretty useless too. So the Recents tab is even more useless. It’s just a list of albums? songs? playlists? I have no idea. There’s no explanation of recent what?

Home – this one is even more weird. It’s got random playlists such as “For fans of blah” and TGIF. No reasoning for these. We’re just supposed to assume that they’re customized to day-of-the-week, listening habits, etc. My top recommendation is Latin Guitar Classics. I don’t know why. I’ve been listening to classical music and I daresay the Guitar is hardly a classical music instrument.

When you look at how good the Google Photos app or the Google Home and Assistant apps (which have some weird overlaps) are, it’s amazing that Google has a division making such a confusing and functionally terrible app.

6. The making of an app

When the Amazon Prime Video Apple TV app came out a few days ago, one of the laments people had was that it looks and acts like a website skinned to work with the Apple TV. It’s not horrible (the Hulu app is horrible) but it’s irksome. The Google Play Music iOS app is a joke. The app regularly forgets state and resets me to Library tab. The settings page is long, confusing and not well sectioned.

The album art is also a joke. Most of the music on there has a YouTube video play button as album art. Is this my personal library scraped over the years or a service run by a multi-trillion dollar enterprise? I wonder.

The display icons for artists under the Library tab are huge and most of the time don’t include any photos of the artists and are either blank or some half screwed up album art. The overall design of the app is not material or bootstrap or anything in between. It’s a monstrosity.

7. Misc.

Can your service one-up other music services? Well, if you can’t sort my playlist by any order (RPM would be nice and Spotify doesn’t have that, but I’d take Alphabetical, frequency, year of release, anything), if you can’t play videos within your app, if your service doesn’t include podcasts (I don’t listen to them. I just know other services have them), if your algorithm can’t predict what kind of listener I am (bollywood music, bhangra, pop instead of OMG this guy is Indian we have no idea what to do) and have music discovery in a meaningful way, what better are you than Spotify or Pandora or, heck, Napster?

Does your service have a landscape view? Most music apps do not. But come on Google. Study your competition and trounce them!

What’s with the name, by the way? Was ‘Google Music’ taken?

Recently, I uncovered another glitch in the app. When the missus tried to airplay a song to the Apple TV, if she airplayed the entire screen, the audio was broken and glitchy, but if she did it from inside the app, it worked fine. You’d think they’d hire a test engineer for these things.

There are so many ways Google Music can be better than Apple Music and Spotify and right now, the only cachet they have for me is that it’s free with YouTube Red. That’s just sad.

 

Epilogue

A few weeks ago, I screamed at Google Music on twitter for these issues. They asked me for feedback but I’d cooled down and didn’t bother to send them the feedback. Now they have the information they requested. Let’s see what they do with it.

You’d be asking me, why are you still using Google Play Music if you hate it so? You know what I hate even more? Paying for redundant services. I’m already slated to get rid of Hulu as soon as this season of Grey’s Anatomy ends. If Google ever gives us the option to drop Google Music from YouTube Red and pay less, I’ll gladly go back to Spotify. Till then, I can kvetch.

Photo by The Logo Smith

Running Compass on Vultr

Intro

Recently, I came across a tweet by Aaron Parecki, where he talked about a lifelogging app he built (and recently released) which tracks our location constantly.

I’ve been using Moves on-and-off over the years and partly due to it being now owned by Facebook, and partly because it’s a very crashy app (first time works fine, doesn’t open ever after that and stops tracking properly soon after; I assume the developer is now working on some darker features for the Facebook apps and so doesn’t spend as much time on his own creation), I’ve never been satisfied with Moves.

So, I downloaded Aaron’s Overland GPS Tracker app (free!) and set it up. The app is rather bare and the functionality is not well explained (within it). But it’s free, open source, a one-man job, and in line with the vision for indie dev, so it’s up to us to figure things out. I asked a few questions, got pointed to the settings explainer here. Well worth a read if you download the app.

The next step of the app was to install a remote server which ingests the data and makes it human readable and useful. As Aaron explains, the quest is to answer the question – “where was I at blah date at blah time?” The app’s official homepage recommends one of two servers to send the data to – a service called Icecondor and a server Aaron wrote called Compass. Compass looks nicer than Icecondor, is self-hosted, and I’ve been itching to play with Vultr.com‘s SSD Cloud, which competes with DigitalOcean in pricing and resources. So, here’s a walk-through for getting yourself setup with Vultr, installing Compass, and setting it up with Overland GPS to start tracking your location as creepily as Facebook and Google do it! 🙂

Vultr

Vultr is a nice competitor to DigitalOcean. At $2.50/mo for their cheapest VPS, it’s half the price of what DigitalOcean offers ($5/mo for the same RAM, storage, and CPU, but DO offers twice the bandwidth and, well, is trusted more). There had to be a caveat, right?

I signed up and the first thing I was told to do was to add money to the account. I had the option of not adding any cash and just attaching my credit card, but I’m going to end up using Vultr for something or the other, so I threw $10 at them (shut-up-and-take-my-money style!).

Then, they told me I can deploy a new server! I picked Seattle as my server location, Ubuntu 17.10 as my poison (which was probably a bad idea; more on that later), and scrolled down to the Server pricing. The $10/mo server was pre-selected for me and the $2.50 option was grayed out! (Seriously though, they should give names to these tiers. It’s silly to keep referring to the price.)

I googled around a bit and found out that they keep disabling the cheapest tier (they call it “Temporarily Sold Out”) as a sort of bait-and-switch model to drive new users to the more expensive options. But that sounds somewhat bullshit. If this was truly the behavior, I’d like my money back. But, and I’m glad I did this, I went back and started clicking around to look for solutions. It came in the form of New York! Turns out, they try to drive users to lesser used data centers while everyone who’s trying to set things up actually tries to use the “Silicon Valley” data center (seriously? Who the heck put a data center there???)

New York and Miami currently have open $2.50/mo tiers (ugh, that naming is so needed! I guess I’ll call it the Micro tier and the next one Mini), and networking is not a problem for me (who cares if a little more bandwidth is needed to get this non-time-sensitive data to New York and back), so I picked New York and threw my hat in the ring.

The server came up within… minutes? (Seriously, it was fast!) and I had an IP address to point to! Yay! But, what’s the password? The usual Ubuntu password didn’t work and I looked around at their docs and there wasn’t much to go by (Vultr’s docs aren’t as awesome as DigitalOcean’s. They’re good, just not there yet. They have a documentation bounty program if you’re interested, dear reader.) Then I checked the email which I would have received on server activation. It said that the password is on the dashboard (silly me!).

As I said before, Vultr’s documentation isn’t great, so I followed a mix of Vultr’s LEMP install here and DO’s LEMP stack installation instructions here. I installed PHP 7.1 with FPM (which, I must admit, was a little leap-of-faith because I wasn’t sure Aaron’s code would work without throwing up legacy issues, which it didn’t) and skipped most of the tweaking that Vultr recommends (YMMV).

Compass

Then, I copied over the Compass files (from here) and started following the Setup. The first issue was the .env file. There’s a few settings in there which are confusing, so here’s what I did –

BASE_URL -> This is your website. It uses HTTPS. More on that below.

STORAGE_DIR -> This is the data directory which is supposed to store your incoming data. Oddly enough, it doesn’t. When you use the application, the GUI prompts you to make a ‘database’ (it should be called a ‘project’ Aaron). This database makes its own folder in the Compass directory, so this variable invariably doesn’t get used. Set it anyways.

APP_KEY -> This confused me a bit. I don’t think this is a password. But I set it to something like a password. It’s a 32 char string, so have fun setting it up.

DB_CONNECTION -> Set this all up as you would any other MySQL application. Use the WordPress tutorial by DigitalOcean as a hint of what to do.

DEFAULT_AUTH_ENDPOINT -> This was one of the more confusing things I saw. Was the idea that this was some generic authorization? To figure out, I found Aaron’s own Compass website and tried to login. Turns out Aaron uses a very neat authorization process. There’s no password. All you do is tell which Indie authorization website you want to use to authenticate who you are and it’ll allow you to login. Specifying this URL will mean that if you can login to that other website, you can login to this website. The default is set to ‘https://indieauth.com/auth’. If you let this remain, it’ll mean that anyone who has an indie auth login anywhere will be able to create an account on your Compass server and potentially use it for their own data. So, I authenticated myself into Aaron’s server and now I have an account there! Of course, I don’t recommend this. I changed this Endpoint to my withKnown.com site. That way, only people who can login to my withKnown site can login to my Compass server. Who can login to my withKnown server? Only me. 🙂

There’s a piece of the puzzle which needs addressing. APP_DEBUG is set to true right now. So whenever there’s an error, Compass spits out the entire MySQL connection string, including password, as well as very important system information out to anyone to see. I suspect that once you’re done setting up this server and you trust it, you should follow the Laravel process of ‘migrating’ the application from dev mode to production. This will help secure your application.

 

After this, I moved on to running Composer to install all the dependencies which I needed for Compass. Here’s all the issues I faced there –

“Composer not installed” – Install using

"apt install composer"

“danielstjules/stringy 1.10.0 requires ext-mbstring” –

"apt install php7.1-mbstring"

“phpunit/phpunit 4.8.21 requires ext-dom” –

"apt install phpunit"

“zip extension and unzip command are both missing” –

"apt install zip unzip"

Now, you can run ‘composer install’ and it’ll work.

 

nginx

I recommend using nginx. You’ve got a small server and you don’t want Apache to drown the memory, so just use nginx.

Aaron’s config for nginx were clear, but not helpful, because it doesn’t go with the usual nginx config floating around tutorials. So here’s mine (relevant portions only) –

index index.php index.html index.htm;
root /var/www/html/compass/public;

location / { 
    try_files $uri /index.php?$args; 
}
location /index.php { 
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/var/run/php/php7.1-fpm.sock;    
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
location ~ \.php$ {
    include snippets/fastcgi-php.conf; 
    fastcgi_pass unix:/var/run/php/php7.1-fpm.sock;
}

At this point, I thought I was done. But then, when I tried to open the site, I ran into some very nice errors in the application. First of all, notice the root. The root of the application is not the compass folder itself, but the public folder inside it. This is not mentioned anywhere in the documentation and was well worth twenty minutes of “what the heck?” and then some. But you have it on good authority that this is what you’re supposed to do.

Secondly, the application wasn’t done making me install stuff. So I also had to install curl –

apt install php-curl

Then, I wanted to digress a little and make my life a little more difficult (or easy, depending on who you ask). Aaron’s own Compass server uses Let’s Encrypt based SSL. I’ve always wanted to secure my own sites using SSL, but I’m lazy. For this, I thought, why not!

I found the CertBot instructions for installing with nginx and Ubuntu here. They’re pretty straightforward, with a small error that I ran into – Cloudflare. I use Cloudflare as my DNS, security, loadbalancer, God of Small Things. Cloudflare provides SSL. It’s literally a one click. When you add a new A record to your domain (such as compass.p3k.io), it adds DNS and security itself by routing traffic through Cloudflare’s network. CertBot doesn’t work with that. CertBot needs direct access to the server. So, I had to disable Cloudflare’s lovely protection for my subdomain and let certbot do it’s job. It did so. It automatically modified the nginx config to accept HTTPS-only connections and to route all traffic to HTTPS. I was even able to setup crontab to auto-renew certs –

43 6 * * * certbot renew --post-hook "service nginx restart"

After this, you run the job queue commands as listed by Aaron and you should technically have a running website. But there’s a catch, as there always is. This server that I’ve got is not a ‘mini’. It’s a ‘micro’. 512 MB RAM is not enough to run MySQL, Ubuntu 17.10, nginx, php-fpm, and actually run an application on top of that. So, I ran into a very cryptic error –

[PDOException]                                    
SQLSTATE[HY000] [2002] No such file or directory 

At this point, I had the application running and I was able to visit the site and all, but try to login and it threw this error. The php artisan command also started throwing this error (by the way, you’re supposed to run the ‘php artisan queue:listen’ command in the background for this server. Follow the instructions here to set up supervisord to do so). Most people on StackOverflow seemed to think that if you replace ‘localhost’ with ‘127.0.0.1’ in the app’s settings, it’ll start working again. But that didn’t help. Finally, someone recommended (not in real-time. I’ve only once ever in my life used StackOverflow in real-time to get answers to a question) restarting MySQL. Well duh.

Oh? MySQL won’t restart. Why???

It was this community question on DigitalOcean that gave me the answer I was looking for – I had run out of RAM. Turns out, 512 MB is just enough to play with a server, but not enough to run it for reals. Nonsense. Let’s just add a swap!

I used this excellent and very easy DO tutorial to add swap to my VPS. Notice the shade it throws at you for trying to use swap on SSDs. They specifically say that it doesn’t recommend using swap for DO “or any other provider that utilizes SSD storage” and that this degrades hardware performance for you and “your neighbors”. DO recommends upgrading your instance so it has more RAM instead of using swap. We don’t listen.

Added swap and voila! It’s working! MySQL fires up and the app stops throwing silly errors! I ran htop all night on the instance to monitor for Memory and Swap use and it works just fine! At last, we can login!

 

Overland

OK, we logged in using our designated Indie Auth website! Now what? You’re staring at the blank screen that recommends you create a database. Do it. You give it a fancy name and it spits out a bunch of configuration. Now what? First of all, change the Timezone in the settings to where you are. It’s set to UTC right now, but for me, it’s PST. Also, use

dpkg-reconfigure tzdata

in your Ubuntu command line to change the timezone of your server to where you are. Remember, my server is in New York. But I told it that its timezone is America/Los Angeles. Because.

OK! You’re good to go! You can throw some data at this server! Head over to the Overland GPS app and add this endpoint to it. Only, what’s the endpoint? I added just my compass server’s URL and that didn’t seem to work. Then I looked at the app screenshots and there it was –

https://compass.p3k.io/api/input?token=E6ncEYWxT...

That’s your Receiver endpoint! But, where should I find this? In your Compass ‘database’ settings, You’ve got a read token and a write token. Next to the write token is a link which says “show API endpoint”. Click it and out pops another line which shows you the above. Simply copy this and magically move it to your phone (I WhatsApp myself these things) and you can plug it into the app and start sending data! The first time you plug it in, the app will collect all the data you’ve accumulated till then (I had some 25000 points of data to transmit) and smoothly move everything to the server (Aaron really has done a great job with the app). After that, it’ll move the data in batches the size of which you can specify (God knows why).

But. You’ll see some odd things. For example, in the afternoon, the server’s map changed the date over to the next data (I suspect this is because my server was still on UTC time. Running the tzdata command above should solve this). Also, whenever there’s no data (or the data hasn’t loaded yet), the map points to Portland. I get that Aaron is from there, but I think we should be able to configure this (Seattle, woooo!) because it’s a little jarring. Finally, this will teach you how bad your GPS data is anyways. Most of the time, the map has me squarely in the water, or swimming out and coming back, or has me cross the I-90 bridge by, well, not crossing the bridge but swimming along it). But, that’s just the world we live in.

 

Questions/Issues
  1. Why does this server need MySQL? The Compass documentation says that the data is stored in flat files. Then is the MySQL database only used for temporary storage of data before it’s processed and saved to flat files?
  2. Is HTTPS a requirement of the server or a nice-to-have? I am not sure about this and I just took the safer route.
  3. The app, in debug mode, spits out way too much information which it shouldn’t. I’d like clear instructions on migrating it off debug mode.
  4. Did I decipher the meaning of DEFAULT_AUTH_ENDPOINT correctly? Not sure. Also, Aaron, if you’re reading this – what do I do with my login on your Compass server? Can you allow people to store their data on there, just for visualization (and wiped every night so as not to flood your server).
  5. I still don’t know what the best configuration is for the app (battery-use to tracking). If you’ve got pointers, throw them in the comments below!

No updates please

I was an avid software updater. I would read the updates list, hit the update button and see the download happen. I enjoyed doing this manually because it’s a fun process to acknowledge all the work someone has put into this update that I’m downloading. In that sense, websites are no fun – they change suddenly and have no changelist to describe what all has changed and what new features are available.

But then I got bitten. First, on my iPad Mini (Series 1). iOS 9 slowed everything to a crawl. I still have use for the iPad, but it’s limited to two apps – Scrivener and Kindle. Everything else is basically unusable. I don’t even browse the web on it. It’s just easier to bring out my iPhone 7 Plus for that.

Then, went my Macbook Pro. The main reason is under-use. When I’m developing something, I’ll update the packages, update Xcode, get the latest and greatest of iTunes. But when I’m browsing or reading on it, Safari suffices. Chrome is a crybaby on OSX, so I dumped it and never looked back. Perhaps the lack of Chrome Sync is what drove my usage down? Not sure. All I know is that my Mac cries for updates and I deny it. I don’t even know what version of OS I have. It’s a pain to find out and keep track. I don’t have Siri on it. APFS, you ask? Not gonna do it.

Finally, the iPhone. Oh, the iPhone. I still enjoyed downloading and updating apps on it for the longest time. It’s the most used device I have (and I have the Apple Watch strapped to my wrist most of the day. It’s just not used in the same way). I have truly enjoyed watching app updates change the way I use my iPhone and what I keep on my main home screen.

Then, the inevitable happened. I got bitten. The app update didn’t mention that Terminology 3 was going to change one of the main features of the app – opening on the search view. I thought the cries of a thousand users would make the developer reconsider. I don’t even know where that debate went.

Then, I updated an app I was just trying and the developer put an ad at the beginning of the app, destroying the experience completely. I gave my first ever App Store review – a 1 star with a few choice bad words. I calmed down after a day and updated the 1 to 4 stars. But I made the developer notice. I made sure they understood that not mentioning the ads in the app update is the reason why they got the bad review. They changed the update text to include mention of the ads.

I don’t mind change. I’d just like to have it mentioned to me. Today, browsing the app updates page, I saw that Delta Dental had updated their app. I opened the details and all it said was “bug fixes”. There’s more effort made to inform users of what’s changing in SnapChat than what’s changing in an insurance company’s app. There’s technology for you.

Twitter changed. Instagram changed. Facebook changed. I see more ads and more crap ‘features’ in these apps that anyone around me. Maybe they’ve labelled me guinea pig?

One day, I updated Google Search’s app. There was a time I used it as my main search app. The app team had added Cards to the app. The feature destroyed the app. It had slowed down to a crawl, it was not even loading the cards properly and wouldn’t let me jump right into a search. Google eventually fixed the cards and made the thing faster, but the app’s main focus is still ‘showing information’ instead of letting me ‘search for information’. My main ‘search’ now happens through Safari – it’s got adblocking, it’s got session retention (Google Search app is crap for that), and it’s just nicer to use.

I’d like to remember what exactly it was that broke the camel’s back, but there’s just a very long list to look through. One day, I was just not updating apps with the same zeal and the same frequency. I realized that the release notes were a joke, and features were going to keep changing at whatever terrible pace the developers decided was right. I’m a developer, I know that it’s very easy to decide to change something (and very difficult to implement it). So I respect the devs who put hours into these updates. But I’m just not going to update apps (and OS versions) as frequently as they come out with them.

Since the last few days, we’ve been talking about iOS 11. My wife has been asking me to backup her phone and update it. She’s never been this excited about an OS update. But I couldn’t be farther away from it. I’m not excited about HEIF/HEVC. I’m not interested in iOS 11 ‘degrading’ my phone. I’m not even excited about all the bugs they’ll eventually iron out with a point release in a month or two.

But, I’ve readied my phone for it. I’ve deleted about thirteen thousand photos from my phone, primarily because I was tired of keeping them around (is it true that less storage used translated to better battery life?). I’ve taken a backup or two. Maybe I’ll update my phone today. Maybe I’ll update my wife’s phone first and see how that goes.

But app updates? No, thank you.

Unraveling the future of Day One Sync

Thursday was an important milestone for Day One and its users. The launch of the Day One browser extensions marks a time when the Day One team is ready to launch API based products outside of their default apps, a somewhat return to the time when Day One 1.0 was a beautiful, open garden of apps and services that could plug-in (and out) without much trouble. Day One 2.0 robbed a lot of people of those options and these browser extensions allow us to come back into the fold.

I’m not under the impression that this means that Day One will suddenly be as open and accepting as it once was. No, the walled garden that the team has created will remain. Their promises to end-to-end encrypt all data (while allowing complete access through their API), their wish to remain free of third-party sync services such as Dropbox, and their interest in keeping their company growing, mean that Day One is never headed back to the old days.

But that doesn’t mean things can’t move forward to a good place. Of course, with the launch of Day One Premium, what that good place is, is a little unclear. Yesterday, while launching extensions on Instagram, the Day One team answered a few questions and that gives us a hint of how things are going to work from now on.

Let’s first summarize what we understand of the customer ‘levels’ for the Day One service –

  1. Basic – This is a new tier. If you download the Day One app today (or are a Day One Classic user updating to Day One 2.0 today), on iOS or Mac, you’ll be a free user. All your data will be saved locally on the device which you use and any time you want to a. Create multiple journals, or b. Sync your data to the Day One Sync service, you’ll be prompted to pony up and become a Premium member.
  2. Premium – This too is a new tier. If you want to sync your data across devices, get access to the encrypted journals feature, support Day One in their awesome venture, and get 25% off print book orders), you get to buy into the Day One subscription service. It’s currently $35/year for new users and $25/year for older users, as explained in the FAQ.
  3. Plus – This is the new name for the old tier. If you downloaded Day One 2.0 on any platform before the Premium tier was introduced, this is where you stand. You get access to Day One Sync, get to make up to 10 journals, get access to data encryption, use cloud services such as IFTTT, etc.

Here’s what you don’t get with the Plus subscription –

  1. If you bought Day One on one platform (iOS or Mac) before Premium was launched, and bought it (for free) on the other platform after, you don’t get Sync between devices. You can still export your Day One journal and import it at the other end, but that’s just too cumbersome.
  2. Similarly, you don’t get access to more than 10 journals, and can have no more than 10 images per post.

But yesterday’s release taught me something interesting –

Day One is still a company that cares for its users. So, it seems that if you’re a Plus member, many future features and launches will work for you. Day One browser extensions currently work only with unencrypted journals. However, since Plus members do get access to encryption and Sync, in the future, it’s possible that support for end-to-end encryption will be added to the extensions and as a Plus member, you’ll still be able to use them.

Similarly, right now IFTTT is the only third-party sync service allowed to plug into Day One. You can use it in a lot of ways – saving your Instagram posts to Day One, emailing an entry into Day One, stashing away your tweets, your weight (using Withings), the day’s weather, your Instapaper Likes, and your Evernote entries.

But I suspect that when Day One launches their API, Plus members will definitely get access to it. They’ll get access to it for both encrypted and unencrypted journals, and will be able to use a lot of the tools and services they were using with Day One 1.0, updated to work with the API, of course. This seems not only likely, it seems definitive with the way the Day One team launched the browser extension.

Why am I even talking about Plus? It would seem that most future users of Day One will be either Basic or Premium members, right? But most of their current users are Plus members. On top of that, I believe that a large percentage of Day One users fall in one of two categories – they either have only on Apple device (iOS or Mac) and so don’t care for a lot of Premium features, or they went ahead and bought both Mac and iOS apps, and so they didn’t get affected by being pushed into the Plus tier either.

However, I am part of a significant majority which either want Day One access on Windows or Android. This is why understanding the Day One team’s motives behind every move they make is important to me. From what I’ve understood, they’ve got nothing but good intentions when it comes to treating Plus users with fairness, even if it comes at the cost of Premium subscriptions in the short-term.

Future Day One apps (for Windows and Android) will be free and siloed the way the new versions of the iOS and Mac apps are. You’ll have to be a Premium user in order to sync between these devices. But the devices on which you’re a Plus member right now will give you a pretty premium experience, and any third party tie-ins and API based features should be available to Plus members without having to move to the subscription model.

Of course, with the launch of the browser extensions, the Day One team has solved a very big problem – getting journal entries in, on Windows (and Mac for iOS-only) for Plus users. That saves people like me from a lot of time and effort!

p.s. According to the Day One team, Day One Classic is still around, just not under active development. Most of us (specially if you read till the end), have moved on to Day One 2.0, but if you download the Day One Classic app (or still have it installed on your system), Day One Sync is still working on it and syncing to it. So if you have that, keep using it!