The Aboutbox

We build Web, Search and Video applications.

The Database Is Your Friend

Some lessons we are doomed to learn over and over again. In Rails, ActiveRecord has a number of methods to help ensure the integrity of your data, such as validates_presence_of, validates_uniqueness_of, and dependent: :destroy on relations.

Those are useful for providing error messages when users submit forms, but they are no guarantee that your data will stay consistent. In order to do that, you need to define constraints on the database. Without fail, every time I have ever forgotten to define not-null constraints, unique indexes and foreign key relations, I have been bitten.

The issues can be very subtle, but here’s an example

1
2
3
def delete_removed_items(source_ids)
  Item.where.not(source_id: source_ids).delete_all
end

This code is part of a background process that synchronizes data with a back-end service, and we are trying to delete items that no longer exist. There is a problem, however. According to the Rails documentation for delete_all:

Be careful with relations though, in particular :dependent rules defined on associations are not honored. Returns the number of rows affected.

Oops! I guess the synchronization code doesn’t delete any associations. Now there are a bunch of orphaned records laying around. This is exactly the kind of issue that would have been caught right away if the database were enforcing a foreign key relation. Yeah, I picked the wrong method in this case, but look, it gets worse. Here is what the Rails documentation says for validates_uniqueness_of:

Using this validation method in conjunction with ActiveRecord::Base#save does not guarantee the absence of duplicate record insertions, because uniqueness checks on the application level are inherently prone to race conditions

So, lesson learned, again. By the way, the best way I know to make foreign key relationships easy to deal with in Rails is to use the excellent Foreigner plugin.

Two Factor Authentication

A short note: if you have not turned on two factor authentication for all of your favorite services (Facebook, Twitter, LinkedIn, Gmail, etc.), stop what you are doing, and set it up RIGHT NOW.

iPhone Popovers With RubyMotion

For an app that I am developing, I needed a simple Popover view that appears when the user taps a button. Apple does provide a UIPopoverController class that shows popovers at a specific point, but it only works on the iPad. On the iPhone, your application will crash if you try to use it. To work around this issue, developers have created their own controls.

The simplest one that I have found is called PopoverView. It is not the most powerful, but it is very easy to use. Its interface is basically a set of static methods that display a popover view anchored at a specific point. You can supply a UIView, or if you just want to allow the user to select things, you can keep it really simple and pass in an array of strings:

For example, here is a view controller that displays a button with a popup to change its title:

(popover_controller.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class PopoverController < UIViewController
  def viewDidLoad
    super
    self.view.backgroundColor = UIColor.whiteColor

    @choices = ['Choice 1', 'Choice 2', 'Choice 3']

    @button = UIButton.buttonWithType(UIButtonTypeRoundedRect)
    @button.frame = [[10,10],[80,40]]
    @button.setTitle(@choices[0], forState: UIControlStateNormal)
    @button.setTitleColor(UIColor.blueColor, forState: UIControlStateNormal)
    @button.addTarget(self, action: 'show_popover', forControlEvents: UIControlEventTouchUpInside)
    self.view.addSubview @button
  end

  def show_popover
    PopoverView.showPopoverAtPoint([50,50], inView: self.view, withStringArray: @choices, delegate: self)
  end

  def popoverView(popoverView, didSelectItemAtIndex: index)
    @button.setTitle(@choices[index], forState: UIControlStateNormal)
    popoverView.dismiss
  end

end

See the full example on Github.

Putting Twitter Back in Octopress

I’m using Octopress for blogging and with the latest update the Twitter support went away. Why that happened is a longer conversation, but the short of it is, that Twitter needs to make money, so they are putting up additional roadblocks to grabbing their data and putting it other places.

So now the only option is to embed a Twitter widget on the blog sidebar, which is okay, but you don’t get as much control. This post got me most of the way there, but it’s best to keep customizations separate so when you update Octopress they do not get stomped on.

Without further ado, here’s what you need to do to get your tweets back into your Octopress blog:

Create a new Twitter timeline widget and copy the resulting html snippet to your clipboard.

Create source/_includes/custom/asides/twitter.html:

1
2
3
4
<section>
  <h1>Latest Tweets</h1>
  <!-- Paste your widget HTML snippet here -->
</section>

Add custom/asides/twitter.html to your config.yml file:

1
default_asides: [asides/recent_posts.html, custom/asides/twitter.html, asides/github.html, asides/delicious.html, asides/pinboard.html, asides/googleplus.html]

(Hat Tip to jmac for doing most of the legwork. I just made a few tweaks to make it a little more update-friendly)

Devops Lite

I’ve been hearing some great things about Gitlab to handle source control in those cases where clients want to run Git, but are not in a position to have a Github account or a hosted version of Github, or perhaps for a huge number of private side-projects. Interestingly, it also as a CI server.

I just ran across Dokku, a “mini-heroku” that accepts git push requests and deploys them.

I think one could do something very, very interesting with Dokku plus Knife Solo to spin up servers and provision them.

RubyConf Wrap-Up

Since RubyConf was in Denver this year, it would have been crazy not to go. Last time I went to RubyConf, it was 2009, and it’s amazing how much the community has changed since then.

I remember listening to a presentation on some exotic Erlang-based message-passing system, and the speaker, from Github, referred to Java programmers as ‘Drones’. Perhaps the Ruby community was still in phase where most of the members where ‘enterprise refugees’ and we were determined to burn down everything Enterprise, and remake it in a more Agile, more Flexible image.

Three years passes. There have been a few major Rails releases. There are a lot of people now who have these giant ‘Monorail’ applications, and now they are looking for solutions for some of these scaling problems, and if there is one thing the Enterprise knows, it’s how to operate at scale. It’s as if all of us hippies cut our hair, went to college, started families and are getting some real jobs. We need to look to the Enterprise for wisdom, while hopefully avoiding some of the mistakes the previous generation made. Almost every talk gushed on the awesomeness of JRuby and the JVM, about which I totally agree. Many talks talked about breaking apart large apps into services, and ways of structuring multi-machine architectures, so it’s still possible to test quickly and deploy quickly.

There was also a focus on monitoring, gathering data and using it to track usage patterns and find problems.

Here is a round-up of some of the awesome things that I discovered.

  • T, the command-line twitter client – A twitter client with truly deep karate searching through twitter streams.

  • Gitnesse, running Cucumber Features from a Git Wiki Part of the “sell” for Cucumber is that you write feature descriptions English, so stakeholders can understand or write the stories. If that’s the goal, why are we keeping the stories in the same place as the source code? Wouldn’t a wiki be better?

  • Activesupport notifications The most important feature of Rails that I didn’t know anything about. A really powerful way to log lots of events, which can then be captured, aggregated and presented by other tools.

  • Metriks – A lot of people sung the praises of Metriks as a means of capturing metrics from an application for publication to another piece of software, such as graphite or statsd.

  • Metriksd – A server that can capture and aggregate metrics

  • Graphite – Front-end and back-end system to store time-series data

  • d3.js – Javascript library to visualize data

  • Cube Cubism – Projects from Square that collect and present time-series data.

  • System-metrics – A lo-fi system for collecting metrics

  • Issue Triage – A system Heroku put together to help triage bugs and issues in open source projects

  • mRuby – Minimalistic subset of ruby designed to run on a very low size and memory footprint

  • MobiRuby – Runtime for iOS devices based on MobiRuby

  • @iDoHaiku – Neural network that creates a new haiku every day.

  • metro – Taking the Rails approach to developing 2d games with Gosu

Migrating Part of a Subversion Repository

Recently I needed to take part of a monolithic Subversion repository, and create a new, smaller repository, while keeping all of the history intact. For all intents and purposes, I needed to move:

https://svn.example.com/lots_of_projects/project_a

to its own repository rooted at

https://svn.example.com/project_a

Oh, and to make matters more fun, I don’t have admin access to any of these repos.

Import/Export

The easy way to copy things from one repository to another is do an svn export followed by an svn import to the new repository. This works, but you lose all of the history, which is a bummer.

Git-Svn

Another possibility to keep all of the revisions intact is to use git-svn to clone the subversion repository with history and then use it as a basis to populate a new subversion repository. I haven’t tried it, but something like this would probably work:

git svn clone [args] [subversion repo] temp-repo

Follow the instructions here on Google Code but when you fetch, fetch from the temp-repo you just created.

One nice thing about this approach is that it requires nothing on the subversion server side to work. The downside is that you will probably lose dates and user information in the history.

Sync + Filter

Probably the most robust way to do this is without physical access to the svn server is to create a local repository, mirror the source repository, filter out everything that is NOT what you want to copy, and then sync that with the destination repo. There are a few steps involved and it’s better to actually get access to both repositories (then you can just use Subversion’s admin tools) to get everything, but in the absence of access to the servers, then svnsync is the next best thing to get a repository’s contents, with history.

svnadmin create temp-repo
cd temp-repo/hooks
echo "exit 0" > pre-revprop-hook
chmod a+x pre-revprop-hook
cd ../..
svnsync initialize file://[path-to-temp-repo] https://svn.example.com/lots_of_projects
svnsync synchronize file://[path-to-temp-repo]
(Time passes)
svnadmin dump ./temp-repo > everything.dump
cat everything-dump | svndumpfilter include project_a --drop-empty-revs --renumber-revs > project-a-dump
svnadmin create project_a; svnadmin load project_a < project_a_dump
svnsync initialize https://svn.example.com/project_a file://[path-to-project_a-repo]
svnsync synchronize https://svn.example.com/project_a

In essence, what we are doing is creating a local mirror copy of the big repo that we can play with. Once we have a mirror that we can manipulate, we can dump its history and filter out only the project that we are interested in. Once we have that, we create a new local repo with only those commits in it, and synchronize it with its new home.

The only problem with this approach is that there needs to be a pre-revprop-hook installed on the target repository, which will require some cooperation from the administrator of that repository.

If anyone else knows of a way to copy part of a subversion repository with all of its history, I would love to hear it.

Newly Discovered Things That Rock

In the past few days I’ve run across a few tools that have been very, very handy.

First is virtualhost.sh. This is a shell script that sets up virtual hosts using the Mac OS X built-in Web server. I’ve been working on a Wordpress project for one of my clients, where they have a bunch of plugins that implements part of their Web site, but they do not provide a seamless user experience. Basically they want their customers, when they sign up for a membership, they enter all of their information once and it creates a user with the right access, it puts the user’s information in their back-end database system, and it charges their credit card. The current experience has the user entering information on a bunch of different pages and a lot of stuff needs to be typed in over and over again. Not the best impression.

One of the problems developing sites locally is that a lot of sites really like to be at the root of their own domain/subdomain. Some parts of this wordpress instance’s configuration assume that. So I needed to define a virtual host for this site, and thanks to this script, it took about five minutes to download, install the script and create the virtual host.

For Rails development, that brings me to the second awesome tool: Passenger + PassengerPane.

I cannot believe it has taken me so long to discover passenger for development on the mac. It’s nice not to have to run script/server all the time and worry about more then one app being on the same port. With Passenger on the mac, you can just set up hostfile aliases for all the apps that are under development and just hit them in the browser. It’s especially nice for some of the Flex+Rails work that I’ve been doing because I can set a domain that is allowed under the production servers’ crossdomain.xml file.

PassengerPane makes it super easy to define vhosts and hostfile entries for as many rails applications as you want.

I love it when the things that seem to be hard turn out easy!

Guacamole

I am really good at making guacamole. It’s one of my favorite foods in the world, so I consider myself a little bit of a connoisseur. So I am going to let you in on a little secret to making really awesome guacamole: most people over-complicate things. In fact, I am going to share my recipe with you right now:

  • Two ripe avacados
  • Juice from one lime
  • About one teaspoon salt

Start by mushing the avacados with a little lime juice and salt. Add lime juice until you can taste a hint of lime. Add salt to taste. It’s that simple. When people say that it’s really good and I tell them what’s in it, they suggest lots of ingredients that I could put in it. I could, but sometimes perfection is acheived when there is nothing left to take away.

I am sure there are some lessons to be learned here with writing software. One thing that I’ve learned is to think very carefully before adding a feature to a product. Every feature is a promise and a commitment, and not to be entered into lightly. When cooking, when in doubt, omitting ingredients is better than random substitutions.

I heard somewhere something along the lines of: if you’re not ashamed of your first release, then you’re shipping too late.

On Wireframes

I reached my goal yesterday for my iPhone app: have a set of wireframes complete. I am a fan of using wireframes or mockups to determine scope and requirements when working with clients. It is a good way to talk about functionality in a way that is easy to visualize.

In this case, I was able to figure out what screens there should be, and what should be on them. I was also able to show them to my brilliant wife and she was able to give me some feedback as a ‘normal user’.

So what are wireframes? Wireframes are mockups of how an application should look and feel. There is really not a right or wrong way of doing wireframes. A lot of people like graph paper and a ruler. Lots of people like PowerPoint, and there are many other tools that people use. Here are a few off the top of my head:

Needless to say there are lots of tools that one can use to create wireframes, the only advice I can give you on how to choose:

The best wireframe tool is the one that you have, you know well, and you can use to create wireframes quickly

I am not an expert on design or wireframes, but as someone who finds them useful, there are some tips that I’ve found useful:

  • Don’t be a perfectionist The goal is not to create a realistic mockup. That comes later. The point is to figure out what information you need to display and what functionality needs to be there.
  • Don’t worry about every little button If your app needs a video player, it’s ok to put a big box labeled “Video Player” on the screen
  • Get feedback quickly As soon as you have something to show, show it to some people to get some feedback and suggestions. See if people can tell what the heck you’re talking about.

This is just the tip of the iceberg. One UX Designer offers some good tools and tips on uxbooth.com.