Chief Burrito Officer

Reflection on an unsuccessful personal project

Just over 18 months ago I embarked upon a personal experiment (not-for-profit) to build a mobile application marketplace that intelligently ‘pulled’ tweets from twitter**. I thought the idea was straightforward — automatically curate the real-time web to find out what is relevant at that instance in time for the iPhone and iPad. You can check it out, though its slightly defunct, at www.apptwits.com

The site took quite awhile to build, in part because it was one of many side projects I had been experimenting with. However, when v1 launched, the app was something that I felt proud of given the time invested.

Three Key Accomplishments:

First, the search worked relatively well. There were 500k apps and tens of millions of tweets that were used to build the index. I’ve had to turn off features due to limited machine resources but nonetheless I think it was one of the best ways to search the app marketplace.

Second, there were numerous applications that it picked up as “hot” and “upcoming” before many other reputable iOS websites were able to do so. On a number of occasions, it was likely in the first dozen or so sites to report a very hot app. 

Given the millions of people that download from the application marketplace, and the thousands of blogs devoted to all things Apple, this was an impressive feat. This was all automated, so I’d often come to the homepage of the website and learn of an app I should download. For example, AppTwits was the first to break news of the AOL’s edition app and blogger’s iOS app, to name a  few 

Third, the site was built with scale in mind. Since it was built on appengine, it is very much hands-off. AppEngine’s prices ultimately increased, which did pose problems on the model and left me to scramble to figure out how to optimize my site for the new billing model imposed. 

Areas of improvement:

The site is mobile-based and would need an iPhone app, in order to take adoption to the next level. I think that would require elevating this from an ‘experimental’ project into something more serious. Frankly I don’t have the time to do that. 

It needs a layout that is elegant yet simple. My design skills are decent but the finishing touches need to be put in. Additionally, I designed for the power user and need to strip out things that may add clutter. 

Finally, the concept as a whole might need be ready for the mainstream. Twitter, and other real-time platforms, will grow and thus this may be easier to explain to people in the future.

If you have thoughts, please leave them in the comments.

**sidenote: I pursue personal “experiments” because I believe they help keep me on top of the latest technologies, programming languages, and trends. I think every good software engineer should do so as well. 

Blogger’s iOS App on AppTwits

The tweets have started to pour in at AppTwits for Blogger.com’s new iOS review. The reviews are mixed but if you are really in to the blogger platform (note: I’m a tumblr guy myself) than you should download it. Its worth a test drive at the very least. 

Stepping back, this shows google’s continued commitment to building iOS apps when appropriate.

AppTwits Link to the Blogger App: http://www.apptwits.com/application/Blogger/

AppEngine Offers Data Retrieval Service For Unapplied Writes

Warning: If you don’t care about Appengine, move on to the next blog post.

When an AppEngine outage does occur, Google goes to great lengths to help those of that have not migrated to the high-replication datastore on appengine. I haven’t done so because there is no easy mechanism to transfer terrabytes of data (yet — please let me know in the comments if I am mistaken).

For some background, it is possible that users may  not apply writes that happen during the same moments a datacenter fails over. Because google has logs of these transaction, they can still let developers know after the fact which writes were unapplied. The developer can then decide how to proceed.  

Dear Developer,

During the day of Aug 19th, 2011, Google App Engine experienced a
serving outage that lasted approximately 2 hours affecting Master
Slave application. HRD applications were not impacted. In the minutes
leading up to the outage, we received a number of writes from your
application that we were unable to replicate to the secondary
Datastore. This caused the secondary Datastore to be slightly out of
sync when we began serving traffic as part of the outage recovery
process. We call these type of writes “unapplied writes.” Rest
assured, unapplied writes do not impact the transactional consistency
and have not corrupted your application data.

During the recovery of the failed data center, we were able to produce
a log of the unapplied writes that occurred during the outage
period. As a service to you, these unapplied writes are now available
from within your application’s to make it easy for you to re-integrate
them. We have also developed a number of tools and example on how to
do so. If you are interested in better understanding this logged data
and how you can work with it, please visit the Unapplied Writes FAQ:
http://code.google.com/appengine/kb/unappliedwrites.html

The following application you own has unapplied writes:
[REDACTED]

You can determine which entities are impacted by unapplied writes by
looking at the Admin Console Datastore Viewer. If you see any Kinds in
the Kind dropdown box with the prefix __unapplied_write__, this Entity
Kind has unapplied writes that can be re-integrated.

No action is required on your part; you can choose to ignore these
writes if you choose. We’re making these writes available to you
because we want you to know that whenever possible, we take every step
we can to recover any data, complete or incomplete, that you attempt
to store on App Engine.

Additionally, if you have any follow up questions, feel free to fill a
production ticket on the public issue tracker:
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

Note that application using the High Replication Datastore for Storage
are not impacted by datacenter failovers, because App Engine commits
data synchronously to multiple datacenters before returning to the
applications.

We strongly suggest you to migrate to HRD to prevent future datacenter
failovers to impact your application.

We are about to release an improved HRD migration tool, if you haven’t
already we strongly encourage you to signup for early testing here:
[REDACTED]

We apologize for any inconvenience this may have caused you and thank
you for your continued support of Google App Engine.

The App Engine team

© 2011 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043

You have received this mandatory email service announcement to notify
you about important maintenance for your Google App Engine
application.

f8 Conference available at facebook.com/f8

I just finished watching the keynote at the f8 website. I’m still trying to juggle the ideas they’ve presented today on making the web more social. I was fairly impressed when I went to CNN and was able to see exactly what my friends liked. And I’m looking forward to getting information in my facebook stream pertaining to yelp (i.e. what sushi restaurants are good etc.,). 

Some huge dependencies:

- will other companies accept this as an open standard? 

- privacy; is over-sharing culture the norm? Or will the web become social with strict privacy levers.

On a side note, the streaming facebook player needs work!