20 Years, Part I - Keep It Simple

As I was updating my resume, I realized that I’ve been writing software either as my full-time job, or for a significant part of my day for 20 years now. I recently ran across a wonderful Landon Noss writes in Medium about what he’s learned after writing software for 12 years. It’s a long read, but worth it.

I suppose that two decades affords me some perspective, so I figured I would take a look back and see if there are any lessons I’ve learned over the years. This is the first in a four-part series.

Build the simplest thing that could possbly work

Prefer the simple to the high-falutin’ - Walt Whitman

Out of all of the projects that I’ve done in my career, only one of them was a dismal failure. I had just graduated college and started a job at Quark. They brought in a teacher to teach a class in Object Oriented Design, and I ate it up.

Soon thereafter, my first client ever, the owner of a chain of clubs in Colorado and Texas wanted some updates to the desktop Windows application I created for them. Mainly they wanted some minor UI updates to make it easier for them to enter data.

I figured that was a great opportunity to put my new Object-oriented skills to work. The original application I wrote as I was learning C++ was sloppy, and the way I wrote out the data felt like an ugly hack to me, so I rewrote the entire thing. It worked great on my machine, but it kept on losing data for my poor client, and I could never figure out why. They eventually got sick of testing my broken apps, so they just canned the whole project and stuck with the old version.

Thankfully I only lost a few months worth of weekends and a few thousand dollars. It was a small price to pay to learn that the only valid strategy for developing applications and evolving them is to work in small steps, and to keep things as simple as possible.

In the real world, people never know what they want ahead of time, but they have plenty of critique after the fact. The only way to mitigate this is by doing the simplest thing possible and iterating.

The ZeroMQ Guide (one of the best pieces of technical documentation anywhere) states this principle eloquently:

This process starts with a realization: we do not know what we have to make until after we start making it. Coming up with ideas or large-scale designs isn’t just wasteful, it’s a direct hindrance to designing the truly accurate solutions. The really juicy problems are hidden like far valleys, and any activity except active scouting creates a fog that hides those distant valleys. You need to keep mobile, pack light, and move fast.:

Stay tuned for Part II.

Calibrating Our Estimates

The book I’m reading right now: How to Measure Anything suggests that estimates about uncertain quantities be expressed as confidence intervals, for example:

There’s a 90 percent chance the market for widgets is between $100 million and $300 million annually.

Not only does the range convey a lot more information that a single number, people can ‘calibrate’ themselves to become better estimators of probability. Interestingly, pracicing on trivia questions will also translate, across domains, to a better overall ability to estimate probability ranges.

My sense is it would be straightforward to apply these principles in product development to get a better handle on the size of the market a product will serve and how long it will take to build a product or feature.

The Database Is Your Friend

Some lessons we are doomed to learn over and over again. In Rails, ActiveRecord has a number of methods to help ensure the integrity of your data, such as validates_presence_of, validates_uniqueness_of, and dependent: :destroy on relations.

Those are useful for providing error messages when users submit forms, but they are no guarantee that your data will stay consistent. In order to do that, you need to define constraints on the database. Without fail, every time I have ever forgotten to define not-null constraints, unique indexes and foreign key relations, I have been bitten.

The issues can be very subtle, but here’s an example

def delete_removed_items(source_ids)
  Item.where.not(source_id: source_ids).delete_all
end

This code is part of a background process that synchronizes data with a back-end service, and we are trying to delete items that no longer exist. There is a problem, however. According to the Rails documentation for delete_all:

Be careful with relations though, in particular :dependent rules defined on associations are not honored. Returns the number of rows affected.

Oops! I guess the synchronization code doesn’t delete any associations. Now there are a bunch of orphaned records laying around. This is exactly the kind of issue that would have been caught right away if the database were enforcing a foreign key relation. Yeah, I picked the wrong method in this case, but look, it gets worse. Here is what the Rails documentation says for validates_uniqueness_of:

Using this validation method in conjunction with ActiveRecord::Base#save does not guarantee the absence of duplicate record insertions, because uniqueness checks on the application level are inherently prone to race conditions

So, lesson learned, again. By the way, the best way I know to make foreign key relationships easy to deal with in Rails is to use the excellent Foreigner plugin.

Two Factor Authentication

A short note: if you have not turned on two factor authentication for all of your favorite services (Facebook, Twitter, LinkedIn, Gmail, etc.), stop what you are doing, and set it up RIGHT NOW.

iPhone Popovers With RubyMotion

For an app that I am developing, I needed a simple Popover view that appears when the user taps a button. Apple does provide a UIPopoverController class that shows popovers at a specific point, but it only works on the iPad. On the iPhone, your application will crash if you try to use it. To work around this issue, developers have created their own controls.

The simplest one that I have found is called PopoverView. It is not the most powerful, but it is very easy to use. Its interface is basically a set of static methods that display a popover view anchored at a specific point. You can supply a UIView, or if you just want to allow the user to select things, you can keep it really simple and pass in an array of strings:

Example

For example, here is a view controller that displays a button with a popup to change its title:

(popover_controller.rb) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class PopoverController < UIViewController
  def viewDidLoad
    super
    self.view.backgroundColor = UIColor.whiteColor

    @choices = ['Choice 1', 'Choice 2', 'Choice 3']

    @button = UIButton.buttonWithType(UIButtonTypeRoundedRect)
    @button.frame = [[10,10],[80,40]]
    @button.setTitle(@choices[0], forState: UIControlStateNormal)
    @button.setTitleColor(UIColor.blueColor, forState: UIControlStateNormal)
    @button.addTarget(self, action: 'show_popover', forControlEvents: UIControlEventTouchUpInside)
    self.view.addSubview @button
  end

  def show_popover
    PopoverView.showPopoverAtPoint([50,50], inView: self.view, withStringArray: @choices, delegate: self)
  end

  def popoverView(popoverView, didSelectItemAtIndex: index)
    @button.setTitle(@choices[index], forState: UIControlStateNormal)
    popoverView.dismiss
  end

end

See the full example on Github.

Putting Twitter Back in Octopress

I’m using Octopress for blogging and with the latest update the Twitter support went away. Why that happened is a longer conversation, but the short of it is, that Twitter needs to make money, so they are putting up additional roadblocks to grabbing their data and putting it other places.

So now the only option is to embed a Twitter widget on the blog sidebar, which is okay, but you don’t get as much control. This post got me most of the way there, but it’s best to keep customizations separate so when you update Octopress they do not get stomped on.

Without further ado, here’s what you need to do to get your tweets back into your Octopress blog:

Create a new Twitter timeline widget and copy the resulting html snippet to your clipboard.

Create source/_includes/custom/asides/twitter.html:

<section>
  <h1>Latest Tweets</h1>
  <!-- Paste your widget HTML snippet here -->
</section>

Add custom/asides/twitter.html to your config.yml file:

default_asides: [asides/recent_posts.html, custom/asides/twitter.html, asides/github.html, asides/delicious.html, asides/pinboard.html, asides/googleplus.html]

(Hat Tip to jmac for doing most of the legwork. I just made a few tweaks to make it a little more update-friendly)

Devops Lite

I’ve been hearing some great things about Gitlab to handle source control in those cases where clients want to run Git, but are not in a position to have a Github account or a hosted version of Github, or perhaps for a huge number of private side-projects. Interestingly, it also as a CI server.

I just ran across Dokku, a “mini-heroku” that accepts git push requests and deploys them.

I think one could do something very, very interesting with Dokku plus Knife Solo to spin up servers and provision them.

RubyConf Wrap-Up

Since RubyConf was in Denver this year, it would have been crazy not to go. Last time I went to RubyConf, it was 2009, and it’s amazing how much the community has changed since then.

I remember listening to a presentation on some exotic Erlang-based message-passing system, and the speaker, from Github, referred to Java programmers as ‘Drones’. Perhaps the Ruby community was still in phase where most of the members where ‘enterprise refugees’ and we were determined to burn down everything Enterprise, and remake it in a more Agile, more Flexible image.

Three years passes. There have been a few major Rails releases. There are a lot of people now who have these giant ‘Monorail’ applications, and now they are looking for solutions for some of these scaling problems, and if there is one thing the Enterprise knows, it’s how to operate at scale. It’s as if all of us hippies cut our hair, went to college, started families and are getting some real jobs. We need to look to the Enterprise for wisdom, while hopefully avoiding some of the mistakes the previous generation made. Almost every talk gushed on the awesomeness of JRuby and the JVM, about which I totally agree. Many talks talked about breaking apart large apps into services, and ways of structuring multi-machine architectures, so it’s still possible to test quickly and deploy quickly.

There was also a focus on monitoring, gathering data and using it to track usage patterns and find problems.

Here is a round-up of some of the awesome things that I discovered.

  • T, the command-line twitter client - A twitter client with truly deep karate searching through twitter streams.

  • Gitnesse, running Cucumber Features from a Git Wiki Part of the “sell” for Cucumber is that you write feature descriptions English, so stakeholders can understand or write the stories. If that’s the goal, why are we keeping the stories in the same place as the source code? Wouldn’t a wiki be better?

  • Activesupport notifications The most important feature of Rails that I didn’t know anything about. A really powerful way to log lots of events, which can then be captured, aggregated and presented by other tools.

  • Metriks - A lot of people sung the praises of Metriks as a means of capturing metrics from an application for publication to another piece of software, such as graphite or statsd.

  • Metriksd - A server that can capture and aggregate metrics

  • Graphite - Front-end and back-end system to store time-series data

  • d3.js - Javascript library to visualize data

  • Cube Cubism - Projects from Square that collect and present time-series data.

  • System-metrics - A lo-fi system for collecting metrics

  • Issue Triage - A system Heroku put together to help triage bugs and issues in open source projects

  • mRuby - Minimalistic subset of ruby designed to run on a very low size and memory footprint

  • MobiRuby - Runtime for iOS devices based on MobiRuby

  • @iDoHaiku - Neural network that creates a new haiku every day.

  • metro - Taking the Rails approach to developing 2d games with Gosu

Migrating Part of a Subversion Repository

Recently I needed to take part of a monolithic Subversion repository, and create a new, smaller repository, while keeping all of the history intact. For all intents and purposes, I needed to move:

https://svn.example.com/lots_of_projects/project_a

to its own repository rooted at

https://svn.example.com/project_a

Oh, and to make matters more fun, I don’t have admin access to any of these repos.

Import/Export

The easy way to copy things from one repository to another is do an svn export followed by an svn import to the new repository. This works, but you lose all of the history, which is a bummer.

Git-Svn

Another possibility to keep all of the revisions intact is to use git-svn to clone the subversion repository with history and then use it as a basis to populate a new subversion repository. I haven’t tried it, but something like this would probably work:

git svn clone [args] [subversion repo] temp-repo

Follow the instructions here on Google Code but when you fetch, fetch from the temp-repo you just created.

One nice thing about this approach is that it requires nothing on the subversion server side to work. The downside is that you will probably lose dates and user information in the history.

Sync + Filter

Probably the most robust way to do this is without physical access to the svn server is to create a local repository, mirror the source repository, filter out everything that is NOT what you want to copy, and then sync that with the destination repo. There are a few steps involved and it’s better to actually get access to both repositories (then you can just use Subversion’s admin tools) to get everything, but in the absence of access to the servers, then svnsync is the next best thing to get a repository’s contents, with history.

svnadmin create temp-repo
cd temp-repo/hooks
echo "exit 0" > pre-revprop-hook
chmod a+x pre-revprop-hook
cd ../..
svnsync initialize file://[path-to-temp-repo] https://svn.example.com/lots_of_projects
svnsync synchronize file://[path-to-temp-repo]
(Time passes)
svnadmin dump ./temp-repo > everything.dump
cat everything-dump | svndumpfilter include project_a --drop-empty-revs --renumber-revs > project-a-dump
svnadmin create project_a; svnadmin load project_a < project_a_dump
svnsync initialize https://svn.example.com/project_a file://[path-to-project_a-repo]
svnsync synchronize https://svn.example.com/project_a

In essence, what we are doing is creating a local mirror copy of the big repo that we can play with. Once we have a mirror that we can manipulate, we can dump its history and filter out only the project that we are interested in. Once we have that, we create a new local repo with only those commits in it, and synchronize it with its new home.

The only problem with this approach is that there needs to be a pre-revprop-hook installed on the target repository, which will require some cooperation from the administrator of that repository.

If anyone else knows of a way to copy part of a subversion repository with all of its history, I would love to hear it.