8th Light Blog

What is a Bug?

13 May 2013

A while back I wrote a little about bugs. I got a few questions and objections and the most common one goes something like this:

"Fixing all the bugs is silly. You're telling me that when somebody reports that IE4 and Netscape Navigator don't properly render your CSS3 transitions, you go in and fix it? Many bugs aren't worth the cost to fix them."

I'm exaggerating, but only slightly. Generally people point out obscure miniscule errors as a way to say my argument is invalid. My answer to the above is simple: that's not a bug. Nobody would reasonably expect that scenario to work, and if it is really necessary it's a user story and should be treated as such. The follow up question to this is naturally...

What Is A Bug?

My comprehensive definition of a a bug is:

A defect is a part of the system that does not work as specified, or a part of the system that while unspecified behaves egregiously (i.e. it crashes).

That sounds professional enough but doesn't really help the situation. I prefer this definition:

A defect is when a developer makes a mistake.

That's a lot better, even though it's less comprehensive and would make a lawyer puke. The important terms to remember here are "developer" and "mistake." Business Analyst specifies the wrong feature? Write a new User Story to change the feature. Nobody likes the new feature? New User Story. If the system doesn't work on IE4 and it needs to, well then you should probably write a story. Let's go through a few examples of potential bug reports, and see if they are really bugs or not.

When I generate a PDF it takes 40 seconds to generate. Fix.

When it comes to the bug or story question, performance and scaling is one of the trickiest areas to make that decision. When determining whether or not a performance issue is a defect you have to ask a few questions:

Was a minimum performance specified?

If we have a system that generates PDFs, like the bug report above, did we originally specify a maximum time limit? If we did then this is probably a bug. Even then we may need to evaluate if the time limit is still reasonable, as we may have added a lot of features to the PDF generation causing it to slow down. We may even be able to throw hardware at it. Still, if we specified a minimum performance and are not meeting it, then it's probably a bug.

Is the performance embarassingly slow?

I keep using emotional terms instead of precise ones, because I usually know in my gut when I've screwed up. Imagine you changed the PDF system above and instead of the bug report saying 40 seconds, it said 2 minutes, or 2 hours. At some point it becomes clear, regardless of specification, that the system is way too slow. Think of links that take 10 seconds, video games that run at 4 frames a second, or iPhone apps that take 2 minutes to start. Context matters, but we don't want to force our customer to write on each feature "and it should be fast enough that the user doesn't think it crashed." Obvious bugs are obvious.

Did it drastically regress?

If the performance was unspecified and is within reasonable parameters, then did it suddenly and drastically regress? Forty seconds may have seemed fine in isolation, but not if it used to take 3. If it did, then we need to look into the reasons why. Perhaps this slowdown was expected, or perhaps it was an accident. Did you add new features and cause a natural slow down? Or did you screw up? That's the difference between a feature and a bug.

On Mobile Safari the popup dialogs don't work

This is a variation on the IE4 bug I described earlier, but one that somebody might reasonably report. The first question to ask is, "Was it specified?" If not, then we ask "Is it reasonable for it to work?" For instance many of the JQuery-UI widgets don't work on Mobile Safari. If you've got an application that's existed for a long time and has many of these dialogs, then it might be a series of stories to optimize the behavior on mobile browsers.

On the other hand if you started the application three months ago and it's been made clear that mobile browsers are a target platform, then hell yes this is a bug. As the developer you should have ensured your code would work across browsers. You made a mistake.

The New File wizard takes too many clicks to get through.

In an Agile approach we refine our solutions over time. We get the first version of an application shipped quickly and get customer feedback as soon as possible, then use that feedback to improve the product. This is closer to what Ward Cunningham meant when he coined the term Technical Debt. We ship code with limited knowledge, and frequently that limited knowledge manifests itself in phrases like: "Well it does the job, but I don't like it."

It is to be expected that no matter how much effort and preparation you put into your user story, even after writing passing acceptance tests, sometimes the completed story will be what the customer said but not what the customer wanted.

This is not a bug. You're refining an existing story based on new information, and should do so in the context of your existing process. Frequently these types of discussions get contentious because the customer, and even other developers in the room, feel like it's a bug. Under pressure to get the stories done right the first time we forget that iterations provide a mechanism for improving existing features, and refining previously played user stories.

There are two ways to help defuse this situation. The first is to remind the customer that they can create more User Stories to improve a feature when there isn't pressure. Sometimes a customer will see something demoed and accept it, but you can just tell it's less than thrilling. Start asking questions about what's wrong, encourage them to write more stories. Remind them that software is never really finished.

The second is to do little things without stories. If the color is wrong, or an icon in the wrong place, just change it right after the meeting. This reminds the customer that your relationship isn't adversarial, and doesn't really hurt you as a developer. They'll see that when you argue against something being a bug you aren't standing on ceremony, but trying build the best software you know how.

All bugs must die, all the time

Pete McBreen wrote: "A Software Craftsman ships with no known defects." This scares a lot of developers, who are concerned that they'll spend all their time gold-plating and never ship, but I believe it's a fundamental idea. It is immoral to know something is broken and ship it anyway. That doesn't mean my software is perfect. It means that I do not knowingly cause problems for users.

We don't accept anything less from other industries. Why should software be any different?

We are Principled: 8th Edition

24 Apr 2013

Steadily Adding Value

We estimate with diligence.
We do not let fear or pressure make us promise what we can’t deliver.

Estimation is Difficult

As craftsmen, we pay close attention to the pain points in our environments and processes. To help produce better results, we create a variety of technologies and practices, including programming principles, organizational frameworks, even standing desks and ergonomic chairs. In our industry, when we perceive a pain, we strive to cure it. How then does the practice of estimation fit in with pain points?

Some people may estimate low for fear of making the client uncomfortable. A low estimate might come from feeling that others would think less of one’s capabilities. Some people may estimate high because the team lead is constantly pushing hard deadlines. Then again, some estimate high from the feeling that they are not as skilled as others. Unprepared estimates bring a high degree of uncertainty, they are no more than predictions of the future, where we will be more wrong than we are right. These assumptions and behaviors, among others, contribute to estimates that lead us down the dangerous path of promising what we can not provide.

Estimates are ripe with pain. The pain exists for all parties involved.

Estimation is Important

When one takes on a task and establishes a timeframe for when it will be completed, everyone involved looks at it differently.

Stakeholders look at how much the work will cost them compared to the functionality delivered.
Product owners look at the current status of the project and use estimates for prioritization of upcoming work.
Teams look at the estimate and uses it for coordinating other work that is needed for the task.
Craftsmen look at the estimate as a commitment and a testament of their capabilities.

Estimates that over-promise degrades the trust others have in our word. Under-promising is unsatisfying when it is just minimizing the gap. When one loses confidence in oneself, their day to day work life becomes taxed and stressed. When the team lead loses confidence in one’s ability to estimate, their chances for growth and success in the organization suffers. When the client loses confidence, the organization risks losing the client.

Due to the ownership, pressure, competing tensions, and difficulty, we often become afraid of estimating. We begin to dread estimation, start avoid it as best as we can. We hear others saying “Developer’s are horrible at estimation, you can’t trust what they say about deadlines.” We start saying it ourselves.

Principled Estimations

At 8th Light we take the challenge of estimations seriously. We know our clients will lose faith if we don’t have strict processes for estimation. As such we take pride in our efforts to improve our estimation processes and practices.

Our estimation process is a group effort. Several craftsmen contribute what they know about the system under consideration. This brings us to a better understanding of what it will take us to complete the work. We collaborate with our clients on the estimates, so they can see the internal discussion and understanding we have of the work. The timeframe for the estimates is not a hard date, but rather a range of possible outcomes.

We have adapted an estimation technique called PERT (Program Evaluation and Review Technique) to augment our process. We estimate an optimistic amount of time, a realistic amount of time, and a pessimistic or worse case timeline. From those three estimates we determine how to charge for the work and then make a commitment. Whenever we find that our estimates were either too high or too low, we work through what caused it.

Diligence

Our diligence in estimation has led to many changes in our process. The methods we use now are not what we used at first, and we will continue to improved them. At the very core of our process is the determination not to let the fears or pains of estimation take over. We will not promise what we cannot deliver, even if that means we have to tell our client “No”. When we as craftsmen make promises we cannot keep. we give others reasons not to trust our abilities.

Estimating work is important. Estimating should be treated no differently than any of our other practices. Estimates are commitments. They affect our image as professionals. Estimates must be honest, whether we are saying “Yes” or “No”. And above all else, we cannot let our fears allow us to promise what we cannot deliver.

More on Estimation

Why is Estimating so Hard? by Uncle Bob

From Estimate to Commitment, by Paul Pagel

We are Principled: 7th Edition

16 Apr 2013

We work at a sustainable pace.
We do not burn out. —from the 8th Light Principles of Steadily Adding Value

At 8th Light, we devote ourselves to crafting durable software for our clients. It is important that we give our full, undivided attention to our work and focus on the needs of the client during the entire life of a project.

We work at a sustainable pace

Software is a craft of unique problems. As practitioners, we are constantly inventing solutions which require us to tread into the unknown. Constantly operating in this uncertainty is difficult and requires constant engagement to build up the contexts necessary.

When we expend our attentional resources^[1] by working long hours, we can no longer think clearly about the problem in front of us. Our brains switch from peripheral to tunnel. We make bad decisions and implement wrong, corner-cutting solutions. By setting a sustainable pace, we commit ourselves to using the hours of the day efficiently and effectively. Working at a pace within our capacity for attention allows us to deliver quality features iteration after iteration.

We do not burn out

We live by the code of endurance. We do not sprint to the finish line^[2]. We know projects are like marathons. If we overwork ourselves, our minds become fatigued and we write sloppy code. It takes more time and effort the next day to rewrite or refactor the mess we created from the night before.

I remember being on a project where we were asked to work overtime to meet an impossible deadline. At the time, however, I thought to myself, “OK, this shouldn’t be too bad. It’s only for one month. I can do this.” For the entire month, I put in anywhere from 10 to 16 hours each day. Not surprisingly, we didn’t meet the deadline and I was completely exhausted. The following months, my body and mind were in recovery mode. I was writing terrible code. I skipped refactoring opportunities, skipped on some tests and adopted an I-don’t-care attitude. I put in my regular hours and left for the day not ever thinking about work.

Marathon runners know the importance of staying within their pace. There is high risk when sprinting at any moment during the race. It can jeopardize results and do bodily harm. In much the same way, we understand sprinting can produce negative effects. It’s a short-term reward with long-term negative consequences. Slow and steady wins the race.

We deliver value to our clients while maintaining a 36 hour work week. We set aside Friday afternoons for fun, open source projects. This gives us an opportunity to learn something new outside of client projects. We also get to pair with others that we do not normally pair with.

This is not to say we are perfect. There are situations where we need to work on Friday afternoons, even nights and weekends. But this is not the norm. We strive to maintain a sustainable pace.

As craftsmen, we uphold our belief that overworking leads to bad code and wasted time. We make the most of our eight hour burns^[3]. We are modern-day endurance runners like Pheidippides. We’re in it to succeed. But we realize that to succeed, we must craft our code at a sustainable pace.

We at 8th Light are principled; and this is one of the principles we follow.

[1] From the book, Pragmatic Thinking and Learning: Refactor Your Wetware. We're not out of time, but out of attention.

[2] Scrum uses the word "sprint" to represent duration. But how can we keep sprinting all the time, week after week? We like to use the term “iteration” instead of “sprint” for this very reason.

[3] Uncle Bob renamed Extreme Programming's 40-hour week to 8-hour burn.

When is a character not a Character?

11 Apr 2013

Characters seem like an easy concept to grasp at first glance. Java has a primitive type char and an object wrapper Character, and these are pretty close to what we mean when we talk about characters. However, these data types aren't fully inclusive of the set of characters we might actually encounter in programs. There are good historical / compatibility reasons for this inconsistency, detailed in the Character javadocs, but I'd like to share a debugging story to highlight the difference between the Java types and what we usually mean when we say "character".

Recently I received a bug report for REPLy, where the character represented by "\ud800\udf30" ^[1], when pasted as input, would immediately exit the program (a nasty bug, to be sure). After a bit of yak shaving, Li-Hsuan and I traced the problem down to a class in Jline responsible for converting bytes from an input stream into characters. Java has a class java.io.InputStreamReader that's responsible for doing exactly this, but it wasn't being used, possibly for historical reasons. The class had been taken from the Apache Harmony codebase, and it correctly handled many non-ASCII UTF-8 characters. All of the characters that worked, however, were a single char in length. The broken ones were non-BMP characters: this was the crux of the problem.

BMP here isn't the image file format, as I learned, but rather the Basic Multilingual Plane. The BMP characters are up to 2 bytes in length (values 0x0000-0xffff), so there's a good bit of coverage there. Characters from the Chinese, Thai, even Mongolian alphabets are there, so if you're not an encoding expert, you might be forgiven if your code only handles BMP characters. But all the same, characters like the one in question won't be correctly handled by code that assumes it'll fit into two bytes.

Back to Java: a char is defined to hold 16 bits of data, and the Character type is just a wrapper around that. And as we've just learned, that's precisely enough space to hold the full range of BMP characters. But when it comes to non-BMP characters, which have values greater than 0xffff, we're out of luck.

The right approach is to bridge the language-level chars to semantic characters by using an API like Character.codePointAt(char[], int) to get one real character starting at a given index in the input array. The output is an int, which is what we need to have enough room for these non-BMP characters. Impressively, the class with the bug was already using an array of chars; it just turned out to only be using one element (of a 1-char array) instead of allowing for multi-char characters. This ended up causing an error case where the alternate InputStreamReader would return -1 to indicate the end of the stream. The actual fix was easy, but as usual, understanding the context surrounding the issue was less so.

As I write this, there are currently 417,000 Google results for non-bmp character bug. This kind of confusion is definitely not limited to Java or even the JVM. If you're interested in learning more about how Unicode works, the wikipedia article on Unicode character planes is as good a place to start as any. There's much more complexity here than I've described in this short post, but if this prevents at least one future mistaken assumption about the size of a character, I'm happy.

[1] I'll settle for linking the description here since something in my toolchain, likely either my browser or my font set, doesn't support it within this blog post.

We are Principled: 6th Edition

08 Apr 2013

We take responsibility for the correctness of our code by testing it thoroughly.
We do not tolerate preventable defects. —from the 8th Light Principles of Well-Crafted Software

Bugs. Defects. Errors, whatever you want to call them they are the scourge of our industry, and they will continue to be so until we take them seriously.

"But Eric!" you say, "We already take our defects seriously! We categorize them, prioritize them, describe them. We have a huuuge database and a gigantic QA department. How can you say we don't take our defects seriously?"

Simple: If you took your defects seriously you would fix them!

Think of all the things I just mentioned. Huge QA departments running manual test scripts, bug tracking databases, bug prioritization meetings, etc. all exist because as an industry we have accepted the premise that software will have a huge number of defects. So many that we can't possibly fix all of them. Instead we track them, and fix only the "important" ones.

At 8th Light we reject this premise. It is not that we believe we don't make mistakes, but we work as hard as we can to prevent them and take responsibility for fixing them.

We Test Thoroughly

We practice Test Driven Development and it's not negotiable. If you want to "save time" by eliminating this, you'll get a straightforward answer.

No.

Any 8th Light team will immediately begin writing tests first, and will write tests for any untested code they encounter. But TDD alone will not produce a quality product.

Each team will add higher-level testing suitable to the context of the project. We may use Fitnesse, Cucumber, Selenium and others. We'll write integration tests as necessary. When we find a defect we will write a test to make sure it never happens again.

Finally we will manually test, and in a special way. In the third edition of this series Paul detailed our demo process. He mentioned that the demo is practiced, which is something we developed on one of our earliest projects. There are many reasons to practice the demo but one of the biggest benefit is catching missed requirements. I couldn't list the number of times one team member would demonstrate a feature only to realize they had left something out or made a simple mistake, usually before the person they were demonstrating to could even catch it. The simple act of walking through the feature slowly proved immensely beneficial.

We Take Responsibility

Have you ever been in a meeting, or meetings, trying to determine the root cause of an expensive defect? Development, QA and management are all represented, and then somebody will ask QA:

"So how come YOU didn't catch this bug?"

It's said with an accusatory emphasis on YOU because after all it's QA's job to catch bugs and if they aren't catching them, then what are they doing? The poor QA lead fumbles around, mentions they'll add a test for it so it won't happen next time, and development goes on doing the same thing they've always done.

Of course the question should be:

"Why did you write that bug?" or better "Why did you ALL miss this bug?"

And the one and only correct answer to this question is really quite simple:

"We screwed up."

You may substitute your own colorful phrasing if you like, but the point is the same. If a mistake was made it's because development made it, not because QA didn't see it, and ideally it's more than one person on the development team. 8th Light backs this up this responsibility in the most clear way possible, financially. On the craftsmanship page of 8th Light's website you'll find the following quote:

If we make a mistake, we'll fix it quickly—for free. It's that simple.

I've told other people we do this. I've told them stories about us working for free for days or weeks at a time while we correct large errors. Usually they claim we're idiots, that we'll lose all our money fixing defects for free, because they accept the premise that large amounts of defects are inevitable.

As evidence to the contrary I point out that we've had this policy for as long as I've been an employee, and in that time we've grown from 5 employees to 40. It seems to be working out just fine.

We Do Not Tolerate Preventable Defects

Why do we "manage" defects? The Wikipedia entry comparing issue-tracking systems has 53 entries in it. Do you know what the best issue tracking system is? A piece of paper, with the defects written on it. Cross them off as you fix them. The end. Why the complications of classification, prioritization, assignment and all that?

It only makes sense if we aren't going to fix them all. We're going to to tolerate many of them, but not "too" many, so we have complicated systems to track them all. I find the notion offensive both to my client and to the end user.

A simple todo list works just fine if you actually fix the defects. On my projects I've found the best system is to stick defects in the next iteration. Sometimes you can fix them all in a given week, sometimes you can't, and if you find you don't regularly get to 0 known defects then you've got to bite the bullet and and do nothing but fix defects for a while. Yes this will delay features shipping occasionally, but working in nearly bug-free code will mean you work faster the overwhelming majority of the time.

I've worked on products with hundreds, even thousands of open defects. These defects were usually tolerated because individually they were small, but collectively they made for extremely slow development and angry customers. When you've reached that number of defects you really only have one option, a complete rewrite. I assure that's far more expensive than occasionally moving out a ship date by a week or two to fix bugs.

Real World

Whenever I get on my soapbox about defects people start giving me reasons why it can't work.

"You'd never ship!"
"What about performance?" or "memory?" or "a feature they didn't like but works?"
"Your customers will take advantage of you!"

The truth is most of the time these problems take care of themselves. We can argue all day on the edges of it, in particular what it means for something to be a bug, and someday I may write about it, but my answer now is to say I work for a company that fixes every defect. And it works.

Stable Sorting in Ruby

26 Mar 2013

The process of learning Ruby has brought several surprises. Some of which are more pleasant than others. The latest I have encountered is that Array#sort is not stable. For those readers who may not understand what I mean by a this, this post starts with a description of what stable sorting is. From there it continues to show why sorting in Ruby is not stable. Lastly I will show a solution to this problem and explain why it works.

What is a stable sort?

Given a list of compound elements, viz. elements which contain elements, we often need different sorts on different elements. We may even need to sort a list without knowing specifically what other sorts were done. A stable sort preserves the order of a previous sort.

Let’s consider an example. Say we have a list of compound elements, each of which contains the elements “First Name” and “Last Name”:

First Name	Last Name
Alex	Jones
Pat	Smith
Alex	Smith
Shawn	Jones
Dana	Smith
Terry	Jones

Suppose we wanted to sort this list using the last name as a primary key and the first name as a secondary key. In other words, all the entries with one last name would come before those with another last name, and for those last names which are the same, the entries would be sorted by first name.

To accomplish this, we would first sort by the secondary key:

First Name	Last Name
Alex	Jones
Alex	Smith
Dana	Smith
Pat	Smith
Shawn	Jones
Terry	Jones

Then we would sort by the primary key. If the second sort is stable, we get the following result:

First Name	Last Name
Alex	Jones
Shawn	Jones
Terry	Jones
Alex	Smith
Dana	Smith
Pat	Smith

However, if the second sort were not stable, the order of the first names would be disturbed, and we might get a result more like this:

First Name	Last Name
Shawn	Jones
Terry	Jones
Alex	Jones
Alex	Smith
Dana	Smith
Pat	Smith

Sorting in Ruby

The Array#sort method in Ruby uses the venerable Quicksort algorithm. In its best case, Quicksort has time complexity O(n log n), but in cases where the data to be sorted is already ordered, the complexity can grow to O(n²). What we gain in efficiency by using Quicksort, we lose in stability. To make matters worse, some implementations randomize the data before beginning the sort to avoid this worst case for time complexity.

Other sorting algorithms, such as merge sort and bubble sort, are stable. However they are not as efficient. Wikipedia has a good comparison of sorting algorithms.

To see an example of how Array#sort is unstable, let’s fire up the Ruby REPL and create some data:

1.9.3-p385 :001 > data = (1..6).map { |a| [rand(3),a] }.shuffle
 => [[2, 1], [1, 6], [0, 5], [2, 3], [1, 2], [1, 4]]

We now have a randomized list of six items, each of which contains two elements. If we treat the second element as a secondary key and sort on it, we get:

1.9.3-p385 :002 > data.sort { |a, b| a[1] <=> b[1] }
 => [[2, 1], [1, 2], [2, 3], [1, 4], [0, 5], [1, 6]]

But then if we take the result of this sort, and sort again on the first element as a primary key, the results may not be what we want:

1.9.3-p385 :003 > data.sort { |a, b| a[1] <=> b[1] }.sort { |a, b| a[0] <=> b[0] }
 => [[0, 5], [1, 2], [1, 6], [1, 4], [2, 3], [2, 1]]

Stabilizing Ruby Sorts

After a few minutes of Google searching, I ran across a blog post with an extension to the Ruby Array class:

class Array
  def stable_sort
      n = 0
      sort_by {|x| n+= 1; [x, n]}
  end
end

If, like myself, you are not content with accepting this as more Ruby magic, allow me to explain my understanding of how it works. This code is based on the Enumerable#sort_by method, and the default comparison operator for Array.

Enumerable#sort_by starts by creating a new array. Each element of the array has two elements. The first is the key given to sort the elements of the original array. The second is the element from the original array corresponding to that key. Returning to our earlier array of data, if we wanted to sort by the second element, sort_by starts by creating an intermediate array:

1.9.3-p385 :004 > intermediate = data.map { |a| [a[1], a] }
 => [[1, [2, 1]], [6, [1, 6]], [5, [0, 5]], [3, [2, 3]], [2, [1, 2]], [4, [1, 4]]]

It then sorts this intermediate array on the first element of each of the new elements:

1.9.3-p385 :005 > intermediate2 = intermediate.sort { |a, b| a[0] <=> b[0] }
 => [[1, [2, 1]], [2, [1, 2]], [3, [2, 3]], [4, [1, 4]], [5, [0, 5]], [6, [1, 6]]]

And extracts the second element of each of the new elements to get the result:

1.9.3-p385 :006 > intermediate2.collect { |a| a[1] }
 => [[2, 1], [1, 2], [2, 3], [1, 4], [0, 5], [1, 6]]

To see how the stable_sort method works, let’s walk through how it would operate on our array. First, we sort on the second element:

1.9.3-p385 :007 > data1 = data.sort { |a, b| a[1] <=> b[1] }
 > [[2, 1], [1, 2], [2, 3], [1, 4], [0, 5], [1, 6]]

Next we create a new array using the first element of the partially sorted array and our monotonically increasing variable:

1.9.3-p385 :008 > n = 0
 => 0 
1.9.3-p385 :009 > data2 = data1.map { |a| n += 1; [[a[0], n], a] }
 => [[[2, 1], [2, 1]], [[1, 2], [1, 2]], [[2, 3], [2, 3]], [[1, 4], [1, 4]], [[0, 5], [0, 5]], [[1, 6], [1, 6]]]

Then we sort on the first element of our new array:

1.9.3-p385 :010 > data3 = data2.sort { |a, b| a[0] <=> b[0] }
 => [[[0, 5], [0, 5]], [[1, 2], [1, 2]], [[1, 4], [1, 4]], [[1, 6], [1, 6]], [[2, 1], [2, 1]], [[2, 3], [2, 3]]]

The reason this works is because the default comparison operator for Array uses the first element of the array as the primary key, the second as the secondary key, and so on. When we created this sort, we built the sort key using a monotonically increasing variable, preserving the order of the elements in the original array whose first elements are equal. In other words, we made the sort stable.

The final step is of course to extract the elements in which we are interested:

1.9.3-p385 :011 > data3.collect { |a| a[1] }
 => [[0, 5], [1, 2], [1, 4], [1, 6], [2, 1], [2, 3]]

This leaves us with the order from the first sort preserved. By understanding that Array#sort may not give the results we expect, how Enumerable#sort_by works, and how the default Array comparison operator works, we can get stable sorts out of Ruby.

We are Principled: 5th Edition

25 Mar 2013

We teach anyone with a willingness to learn.
We do not hoard our knowledge or practices. —from the 8th Light Principles of A Community of Professionals

European scholars often cite the inventions of the compass, gunpowder, papermaking, and printing as the major scientific and technological achievements of Ancient China. Yet there was another invention that perhaps influenced the Chinese civilization in a more profound way.

In the 1st century BC, Ancient China completed its first unification. This transformed a land divided by vassal states into an empire under a single regime. However, even after conquering the feudal lords who relinquished their reigns over the vast territory of China, the imperial family struggled to find enough educated men to fill the government overseeing the enormous population of their empire.

The Han dynasty (206 BC – 220 AD) first appointed administrative officials based on recommendations from aristocrats, who had every interest to keep the power within their own clans. A Han emperor eventually devised an imperial examination, based on Confucian classics, to evaluate the quality of his candidates. The emperor later offered the examination to the general public, and granted any aspiring individual a chance to climb the social ladder and join the ranks of the bureaucracy.

The imperial examination system was the first set of standardized tests based on merit in human history. This, subsequently gave birth to the Chinese Dream, the story of an underprivileged boy spending his entire youth studying Confucius' writings and other Chinese classics, supported by his family and spouse, in the hope of one day becoming a well-respected elite civil servant appointed by the emperor.

By giving its people the motivation to send their children to schools and acquaint themselves with Confucianism, a preferable set of ideals for sustaining a society in harmony, the imperial family of China was able to scale up their administration for one of the largest populations on earth.

As a consultancy, we are fortunate to be living in a time when our service is in great demand. However, we are also faced with the same predicament as those Ancient Chinese rulers. We need more people to expand our business, and we only want to work with the best software professionals to uphold the standard of quality we demand for our service.

For our recruiting, we draw talent from as wide a pool as we can find. We accept applications regardless of their academic background or experience level. We have had apprentices with a Master's degree in Computer Science, a Bachelor's degree in Fine Arts, dropped out of college, or never went to college. We have had apprentices with more than twenty years of industry experience, and with no experience. We host 8th Light University every Friday at noon in our offices and welcome all to come for lunch and lessons, completely free of charge. We also organize the annual Software Craftsmanship North America conference to spark the interest of the public in what we do. We pride ourselves in our dedication to teach anyone with a willingness to learn. We work to enable the dreams of those who want to be software professionals despite their natural advantages or disadvantages. We reward those who stick to their dreams and never give up in the face of all the challenges along their journey.

Just as the Ancient Chinese rulers who had to look past the privileged class to find the right people to run their government, the 8th Light apprenticeship program is a crack in the ivory tower of software engineering that will shatter boundaries and invite a new generation of programmers to build a more diverse future.

The Repository Pattern

23 Mar 2013

A recurring theme on the 8th Light blog is decoupling your applications from the implementation of details. A detail can be anything from UI elements to a database or even a framework. Decoupling your application from the details provides you with the ability to defer implementation decisions until you have a clearer picture of what you really need. In addition, decoupling has the added benefit of making it easier to test your system. In this post I want to take a closer look the Repository Pattern as a way that we can decouple ourselves from one of those details: the database.

Right now you might be having the same reaction I did when I first heard the database referred to as a detail. It was something along the lines of, "WHAT? but every application needs a database! That's where all the stuff goes- how can it be a detail?!?"

Of course we do need data, but the decision on how we're going to store that data is most certainly a detail, i.e. do we use MySQL, PosgreSQL, Redis, flat-files- or even a combination of different options. And there is a huge benefit in delaying that decision as long as possible because it allows us more opportunities to learn how we're going to use the data and the ways that we'll need to access it. Then we can make informed decisions about the details of our database(s).

It's Just An Interface

At its core the Repository pattern is a simple interface. It exists as a layer between your application and your data source so that your application doesn't need to be concerned with the implementation of data storage. As illustrated in the diagram below, instead of talking directly to the database your application uses the interface created by the Repository. It doesn't matter whether I'm using PostgreSQL or a simple in-memory data structure.

Repository, meet Sinatra

Following is a scaled-down example of how I implemented a Repository in a Sinatra application. It was a small internal application and we implemented the Repository pattern for a couple of reasons: 1) I wasn't sure what database I was going to use, and 2) I wanted to keep my tests fast by using in-memory objects.

The first thing I needed for my app was a User, so I created a MemoryRepository::UserRepository that allowed me to store and retrieve a new user for the purposes of testing and development. (Since I'm storing everything in memory all of the records are just User objects in a hash with unique ids.)

 1 module MemoryRepository
 2   class UserRepository
 3     def initialize
 4       @records = {}
 5       @id = 1
 6     end
 7  
 8     def model_class
 9       MemoryRepository::User
10     end
11  
12     def new(attributes = {})
13       model_class.new(attributes)
14     end
15  
16     def save(object)
17       object.id = @id
18       @records[@id] = object
19       @id += 1
20       return object
21     end
22  
23     def find_by_id(n)
24       @records[n.to_i]
25     end
26   end
27 end

There shouldn't be any surprises in these methods, except for maybe model_class, which is used again in the new method. My repository's primary responsibility is the storage of objects so it shouldn't really need to be concerned with the creation of those objects. However, there was a good chance we were going to end up using DataMapper or ActiveRecord so it wouldn't be unreasonable for someone to ask the datastore class to create a new object of the type it holds. So in order to make sure that there weren't any surprises for future devs the new and model_class methods delegate that request to the User object.

Next up is the Repository class. This is the class that my app talks to and allows it to be blissfully unaware of any datastore on the other side. Again, I'm just using a simple hash table and I can register any repository I want to use and then call it with for.

 1 class Repository
 2   def self.register(type, repo)
 3     repositories[type] = repo
 4   end
 5  
 6   def self.repositories
 7     @repositories ||= {}
 8   end
 9  
10   def self.for(type)      
11     repositories[type]
12   end
13 end

In order to tie the Repository and my UserRepository together I add the following lines to an environment.rb file (or a config file or just at the top of the main Sinatra app):

1 configure :test, :development do
2   Repository.register(:user, MemoryRepository::UserRepository.new)
3 end

Now to put it to use. Let's say I have a Sinatra action that takes a user to their home page and I want to grab that user from the database so I can say "Hello, #{@user.name}". Here's how I use my Repository to do that:

1 get "/user/:id" do
2   @user = Repository.for(:user).find_by_id(params[:id])
3   erb '/users/show'.to_sum
4 end

Exciting, right? Well, not really, at least not yet. The real action happens as soon as we decide to implement another data store for Users.

For production I ended up using DataMapper wired to PostgreSQL. Using RSpec's 'shared_examples' (which I've written about here) I was able to quickly build out my new DatamapperRepository::UserRepository based on the methods I'd already implemented in my in-memory UserRepository class.

 1 module DataMapperRepository
 2   class UserRepository
 3     def model_class
 4       DataMapperRepository::User
 5     end
 6  
 7     def new(attributes = {})
 8       model_class.new(attributes)
 9     end
10  
11     def save(object)
12       object.save
13       return object
14     end
15  
16     def find_by_id(n)
17       model_class.get(n)
18     end
19   end
20 end

As you can see, a few of the methods changed, but my main application code knows nothing about the data stores and won't change at all. It will continue to just call Repository.for(:user), and as long as every new UserRepository class implements the same methods then it doesn't matter what's on the other side.

The only lines of code I need to add to start using the new UserRepository in are back in my environment file:

1 configure :production do
2   Repository.register(:user, DatamapperRepository::UserRepository.new)
3 end

Mix & Match Databases

I won't go into detail, but the other real advantage of the Repository pattern is if I decided to use different types of databases. For example, consider if I used PostgreSQL for Users and then Redis for "Tasks". Would my main application care? Nope, it would just continue to call Repository.for(:user) and Repository.for(:task).

Conclusion and Other Resources

This was of course a very simple example of the pattern, but you can see more complex (and real-world) samples in a pair of 8th Light's open source projects. The repositories that we use for an internal project were split out into gems and you can look through those on Gihtub at the artisan-repository, artisan-memory-repository and artisan-ar-repository (for ActiveRecord). Several 8th Light craftsmen have also contributed to Hyperion, which provides a uniform API to multiple datasources using simple key-value pairs. (Hyperion even takes most of the leg work out of setting up the Repository pattern.)

Like any design pattern, it really only makes sense to implement the Repository pattern if you have a good reason. However, the ability to defer your database decision until you've had the chance to learn how you're going to use your data, and then painlessly change it if your needs change, seems like a very good reason.

There are Ladies Present

22 Mar 2013

Oh! I think I get it!

I'm likely to take some heat for what I'm about to write; but I think I finally understand something that's been bothering me.

Three times in the last few years I've been confronted by women who have been offended by something I said during a talk, or a class.

The first time was at a Rails conference a few years ago. I gave a talk entitled "What killed Smalltalk could kill Ruby". In that talk I referred to C++ as a testosterone language, and I referred to Java as an estrogen language. I meant no harm by this statement. I was mostly making a joke about how seriously C++ programmers take themselves compared to Java programmers. However, some of the women in the audience were offended, and tweeted about their concern. I quickly, and publicly, apologized because I do not want the women in our industry to feel unwelcome. I think we need them. More on that later...

Though I apologized, I was also rather puzzled. My "Tim - the tool man - Taylor" persona was going: "eeeuuuhhh?" What was it that was offensive about that testosterone/estrogen remark? I didn't know it, but I had already answered that question. I just didn't get it -- then.

A few years later I was giving a talk at ACCU in Oxford. The name of the talk was "Requiem for C". I started talking about COBOL. And that caused the standard photo image of Admiral Grace Hopper, the inventor of COBOL, to pop into my mind. In the photo she's standing tall in full dress uniform; including a navy cap. Out of my mouth came the words "in that cute little cap". I wanted to grab the words back, but...

After the talk Emily Bache came up to me and gently suggested that the statement was not helpful to all the women in the audience. She reminded me that it's difficult enough to be a woman programmer without someone like me making jokes about other women. I completely agreed, and published yet another apology. I published it because I want women in this industry. I think we need them. More on this later...

Just this week I was teaching a Clean Code course in London. I had 30 students, four of whom were women. I was very pleased with that because it's among the highest ratio of women I've had in class. The class went very well, and after three days I dismissed them all and read their evaluations.

One of the women said she really enjoyed the class and learned a lot, but then wrote "somewhat male oriented, which made me feel out of place". Tim Taylor: "Eeeuuuhhh?" What had I done? Even now I don't know what remark I may have made, or what off hand statement might have slipped. I don't know what her trigger was. But, because of something else that happened this week, I think I get it.

The event I'm referring to is the fiasco at PyCon involving Adria Richards, Alex Reid, Playhaven, and SendGrid. If you aren't aware of it, there's a reasonable history in Amanda Blum's blog. I'm not going to weigh in on the debate about who was right or wrong in this mess, other than to say that the real villains are those jerkwads on the net executing DDOS attacks, and tweeting all manner of obscenities, personal attacks, and illegal threats.

So what is it that I get now? Well, let me tell you a few stories:

When I was in 8th grade, a new kid came to our school. He had been kicked out of his last school in a "tougher" part of the district, so he was an instant hero. A real "hood". He would greet you for the first time by giving you a big smile with sparkling eyes while enticingly pointing his thumbs to his crotch. As you inevitably looked down to follow his pointing gesture he’s say: "Bagged ya!". Ha, ha, ha. Hee, hee, hee. Ho, ho, ho. Nothing brightens the day like teenage boy humor.

Twenty years ago I worked as a chief software architect at a network management startup. I was in the mens room one day and two guys in business suits sidled up to the urinals. As they finished one said to the other: "Careful how you shake the dew off that lilly!" As if on cue the other responded: "Any more that two shakes and you’re playing with it!". Ha, ha, ha. Hee, hee, hee. Ho, ho, ho. Nothing brightens the day like thirty-something urinal humor.

I was watching ER one night a few years back and saw one beautiful young female doctor tell another beautiful young female doctor that she had dropped her recent boyfriend because he was "orally challenged — he wouldn’t go south of the 32nd parallel — he didn’t want to eat at the Y." Ha, ha, ha. Hee, hee, hee. Ho, ho, ho. Nothing brightens the day like the writers of a once great show succumbing to the temptation to use teenage urinal humor.

My wife has become a fan of Grey’s Anatomy. I’ve watched a few of the shows and don't get the attraction. It's all about farting, cheating, screwing on every accommodating surface, pinning panties to bulletin boards, and having your ex-boyfriend teach you how to please your new lesbian lover. Ha, ha, ha. Hee, hee, hee. Ho, ho, ho. Nothing brightens the day like 8 year-old potty humor.

All these things have one thing in common. Locker room humor. I found it all offensive to my intelligence, offensive to my values, and just plain offensive -- because of the context. These things did not belong in the school, in the public men's room, or on TV in my living room. If they belonged anywhere, they belonged in a locker room.

Have we created a locker room environment in the software industry? Has it been male dominated for so long that we've turned it into a place where men relax and tell fart and dick jokes amongst themselves to tickle their pre-pubescent personas? When we male programmers are together, do we feel like we're in a private place where we can drop the rules, pretenses, and manners?

What if the roles were reversed? What if women had dominated the software industry for years, and we men were the ones who were struggling to break into the industry? Men, can you imagine how hard it would be if all the women were constantly, and openly, talking about tampons, cramps, yeast infections, cheating, being cheated on, Trichomoniasis, faking-it, etc? I don't know about you, but It would make me feel out of place. And there'd be no place to escape it, because the women would be everywhere. I'd want them to save that kind of talk for the ladies room. I'd want them to remember that men were present.

Men, perhaps we need to remember that there are ladies present.

Now that last phrase is going to get me into trouble because too many injustices have been enacted under the excuse that ladies were present. I learned about this from a movie called Dangerous Beauty. The story was of a beautiful young courtesan in 16th century Venice. The important men of the city would take her on all their political and business outings, allowing her to know what their ventures were all about. These men kept their wives entirely in the dark, because you didn't discuss things like that when ladies were present. So, to find out what was really happening to their husbands and their city, they consulted the courtesan; and they used a phrase that has stuck with me ever since. They said they were consigned to a life of perpetual inconsequence.

Perpetual Inconsequence. That sounds like prison to me. That sounds like hell to me. If that's what being treated like a lady means, then to hell with that.

I think this explains another experience I had at the Midwestern Ruby conference a couple of years ago. Before I gave the closing keynote on Architecture, I was in the audience listening to a young man give a very interesting talk about GitHub. But the talk bothered me greatly because there was an f-bomb in every sentence. It was effing this and effing that and what the ef here and there and everywhere. Then to top it off he talked about all the code he had written while being "hammered".

Then someone got up and gave another talk and mentioned that he was happy there were so many women in the audience, and that we needed more. (He's right... More on that later...).

Then it was my turn. I started the talk by saying: "If we want more women to come to these events, and to participate in our industry, we need to be careful about dropping F-bombs." Several of the men in the room applauded. But one young woman stood up and yelled: "I totally fucking disagree!"

Tim Taylor: "Eeeuuuhh?"

But now I think I get it. If I may paraphrase:

"Don't treat me differently in a professional context, because I won't stand for perpetual inconsequence. If you drop f bombs in a professional context, don't stop just because I'm a girl. But do remember that I'm a woman and there are things that men do that I don't care to listen to or be part of. Save that for the men's room because it's making me crazy when you do it in front of me."

Now, to pay off my debt.

I had the pleasure of spending an hour with Desi McAdam (@desi) one evening at a function where she happened to be working. I told her that, to me, getting a program to work was like a conquest -- killing the beast and brining home the meat. She replied that to her, it was more of a growing, nurturing experience of helping the software come into the world. Tim Taylor: "Eeeuuuhhh?"

But I've had some time to think about this. When I think about code, I think about it in terms of killing bugs, molding the code into the shape I demand. I am the master. It is my slave. I have the power to make it be what I want. It's a huge power trip for me.

I wonder if that isn't part of the craftsmanship and professionalism problem that our industry has. Why is so much of our software so bad? Is it because we treat it as a conquest? Is it because we don't nurture it enough?

Don't get me wrong, for me it'll always be a conquest. I'll always be the master of my code. That's who I am. But perhaps that's not enough to turn our programming industry into a real profession. Maybe we need the other attitude as well.

After all, in my personal life, I know that the worst decisions I have ever made are the ones I've made without my wife. My wife completes me in many different ways. Without her I'd be a shadow of a man walking through a two-dimensional space. My relationship with her transformed me from a boy into a man. She helped me to grow up.

Our industry needs to grow up too. Perhaps the reason we've had so much trouble maturing is that we don't have enough women to help us make the transition from child to adult. Their attitudes, their ways of thinking, different from ours, may be the very things we need to complete us as an industry, and a profession, and as a craft. It seems to me we need these women in order for our profession to become a profession.

So we need to make the women feel welcome. We need to stop considering that the whole of the software industry is our private locker room. We need to treat women programmers as professional peers, while also remembering that there are ladies present.

Of course I'm just Tim Taylor, talking over the fence to Wilson. Women, do I have this right?

We are Principled: 4th Edition

18 Mar 2013

We embrace differences of opinion and personality.
We do not allow our current practice to impede improvement. —from the 8th Light Principles of A Community of Professionals

Strong opinions

Having a strong opinion about something means that you believe very deeply that you are right. For instance, I have a strong opinion that black coffee is better than coffee with cream and sugar. After many years of experimenting, the idea has become solidified in my mind. When pressed to defend myself, I will argue passionately in favor of black coffee. Does this mean that I am right? Of course not. It simply means that through my experience, I have learned that I like black coffee best. On the other hand, weak opinions are just the opposite. Our experiences have not led us empirically down the same path.

At 8th Light, we often describe ourselves as "opinionated". This means that we have strong opinions about how software should be developed. We work in short iterations because it enables more feedback and transparency. We work in pairs because it provides more accountability. We test drive our code because it gives us more confidence in our solutions. We know and practice SOLID design principles because they lead to cleaner, more maintainable code.

When I say that we feel strongly about these practices, it does not mean that we think all competing practices are wrong. It simply means that our current practices are the most useful we have found so far. Yet we also believe they can be improved.

Weakly held

Strong opinions are often held with a firm grip. This is natural. When we believe strongly that something is right, it can be hard to change our minds. With enough time however, this firm grip can turn into stubbornness. No amount of reasoning will change our minds. We no longer learn and grow from the experiences of others.

At 8th Light, we hold our strong opinions weakly. They are strong opinions because they are strengthened by experience. They are weakly held because we continue to expiriment with an open mind. We do not maintain our current practices for the sake of our pride. Above all, we are most interested in what practices produce the best results for our clients. Sometimes this can be hard. It takes humility to admit that our practices could be wrong or lacking in places. But opening up to others and embracing their feedback is crucial to improvement.

We hold our strong opinions weakly. We embrace diversity because it allows us to learn and grow from the experience of others and not just our own. Otto von Bismarck once said, "A fool learns only from his own mistakes. The wise man learns from the mistakes of others."