8th Light Blog

Deliberate Practice

15 May 2012

“Education comes from within; you get it by struggle and effort and thought.”
— Napoleon Hill

We all know the feeling of flow while coding. It's that moment when you hit a consistent stride and leave the surrounding world. It becomes effortless to glide over problems in an autopilot state and obstacles shrink from the size of mountains to the size of pebbles. Of course, being able to hit this flow means that you have practiced the problems you're facing to the point where it's of second nature. But, how did you get here? You practiced in two ways. First, you practiced moving forward to a place outside of your boundaries. Second, you also practiced the skills acquired from this action. Typically, practice is associated with the latter, but what is missing from this view of practice is the need to feel uncomfortable again. For me, when I begin to see problems as pebbles, it becomes time to deliberately seek a new mountain to climb. Both the acquisition and the polish of skills are important, but I want to shed some light on the practice of pushing boundaries, better known as deliberate practice.

As programmers there are many ways to do this. Here are a few that I find the most helpful.

Impose Artificial Constraints

In order to challenge myself I like to constrain the way I write solutions to problems. I once attended a code retreat where we were not allowed to use if or case statements in writing Conway's Game Of Life. This constraint forced a movement from a comfortable solution to one that left my pair and I feeling uncomfortable with the proceedings. We were forced to think in new ways that expanded our ideas of what constituted a solution. The solution we eventually came up with was a challenge to write.

When the event was over I thought back on my experience and realized that not only did I challenge myself, but I had a lot of fun doing it! It's interesting to see the creatively that flows from constraints. Now, when I approach problems that would have forced me into a solution involving if statements, I can feel comfortable moving outside of that solution and benefit from the understanding my new perspective has brought.

Learn New Languages & Paradigms

Steve Kim has a great post about learning a new language yearly. I want to echo his sentiment as an important part of deliberate practice. When I begin to feel comfortable with a language I know it's time to pick up a new one. The first language I felt comfortable with was Java. While in school I was forced into taking my first jump to another language when, as a class, we dove head first into C++. It was extremely intimidating at first and I started to feel discouraged (I have to manage memory?!?). Thankfully, I didn't back down and when I got the hang of it I was able to write some fun code for an embedded project. If I had stayed inside of my Java comfort zone I would have surely missed the chance to work on the embedded project.

The next big jump for me was to Clojure. This endeavor was another tough challenge because it introduced new paradigms. After some time and many struggles it started to click. Now, Clojure is one of my favorite languages to work with.

When you're feeling comfortable in a language pick up a new one. If you're a Java programmer, like I was, try out C++. If you're a Ruby programmer try out Python. When you feel comfortable in a certain paradigm switch to a new family of languages. There are many paradigms to explore including:

Switch Domains

The development community has many subcommunities such as web developers, front end developers, embedded developers, game developers, and so forth. One way to deliberately practice my craft is to extend myself into other communities. There are new ideas to be shared in the areas of design patterns, system structure, and much more when developers begin to blur the lines between each of these subcommunities. As I said earlier I took on an embedded project. While it was hard at first, the practice of the embedded design craft is important to the learning process and the movement towards greatness. I encourage everyone to take a look at the opportunities to deliberately practice in new domains.

Challenge Yourself

Practice is a process which will ultimately move your skills into a better state. Part of treating software as a craft is to continually raise the bar and engage the surrounding community to raise the bar. When programming begins to feel a little bit mundane seek out an exciting new mountain to climb. I challenge everyone to take a portion of practice time to deliberately practice.

NO DB

15 May 2012

In the United States, in 1920, the manufacture, sale, and importation of alcoholic beverages was prohibited by a constitutional amendment. That amendment was repealed thirteen years later. During that period of prohibition, the beer industry died.

In 1933, when prohibition was lifted, a few giant grain companies started brewing beer. They completely cornered the market. And for nearly 50 years, we in the United State drank this fizzy bodily effluent and called it “beer”. The only way to tolerate the flavor was to drink it very cold.

As a teenager in the ‘60s, I never understood the attraction. Why beer? It was a pale, yellow, distasteful fluid derived from the urine of sick boars, and had no redeeming qualities that I could see.

In 1984, I went to England; and the scales dropped from my eyes. At last I understood. I had tasted beer for the first time; and I found it to be good.

Since those days the beer situation in the United States has improved dramatically. New beer companies are springing up all over the country; and in many cases the beer they make is actually quite good. We don’t have anything quite so nice as a good english bitter; but we’re getting close.

In the ‘80s a few giant database companies cornered the market. They did this by promulgating fear, uncertainty, and doubt amongst managers and marketing people. The word “relational” became synonymous with “good”; and any other kind of data storage mechanism was prohibited.

I was the lead developer in a startup in those days. Our product measured the quality of T1 communications lines. Our data model was relatively simple, and we kept the data in flat files. It worked fine.

But our marketing guy kept on telling us that we had to have a relational database. He said that customers would demand it. I found that to be a strange claim since we hadn’t sold even one system at that time, and no customer had ever mentioned our data storage technology. But the marketing guy was adamant. We just had to have a relational database. Flat files were prohibited.

As the lead developer, responsible for the quality of the software, my view of a relational database was that it would be a big, stogy, slow, expensive pain in the rear. We didn’t have complex queries. We didn’t need massive reporting capabilities. We certainly didn’t need a process with a multi-megabyte footprint sitting in memory and burning cycles. (Remember, this was the ‘80s). So I fought against this idea with everything I had; because it was the wrong technical solution.

This was not a politically astute move for me. Over a period of several months, a hardware engineer who managed to write a few lines of code, was moved into the software group. He was gradually given more and more responsibility, and was eventually named my co-manager. He and I would “share” the responsibility for leading the software team.

Uh huh. Sure. Right. A hardware guy with no real software experience was going to “help” me lead the team. And what do you think his first issue was? Why it was to get a relational database into our system!

I left a month later and started my consulting career. It was best career move I have ever made. The company I left no longer exists. I don’t think they ever made a dime.

I watched the relational database market grow during the ‘90s. I watched as all other data storage technologies, like the object databases, and the B-tree databases dwindled and died; like the beer companies in the 20s. By the end of the ‘90s, only the giants were left.

Those giants were marketing up a storm. They were gods. They were rulers. During the dot com bubble, one of them actually had the audacity to buy television ads that claimed that their product was “the power that drove the internet”. That reminded me of a beer slogan from the ‘70s “Ya gotta grab for all the gusto in life ya can.” Oh brother.

During this time I watched in horror as team after team put the database at the center of their system. They had been convinced by the endless marketing hype that the data model was the most important aspect of the architecture, and that the database was the heart and soul of the design.

I witnessed the rise of a new job function. The DBA! Mere programmers could not be entrusted with the data — so the marketing hype told us. The data is too precious, too fragile, too easily corrupted by those undisciplined louts. We need special people to manage the data. People trained by the database companies. People who would safeguard and promulgate the giant database companies’ marketing message: that the database belongs in the center. The center of the system, the enterprise, the world, the very universe. MUAHAHAHAHAHAHA!

I watched as SQL slipped through every crack and crevice in the system. I ran screaming from systems in which SQL had leaked into the UI. I railed endlessly against the practice of moving all business rules into stored procedures. I quailed and quaked and ranted and raved as I read through entire mail-merge programs written in SQL.

I hammered and hammered as I saw tables and rows permeating the source code of system after system. I hammered out danger. I hammered out a warning. I hammered out that the schema had become “The Blob”, consuming everything in sight. But I knew all my hammering was just slinging pebbles at a behemoth.

And then, in the first decade of the 21st century, the prohibition was lifted, and the NOSQL movement was born. I considered it a kind of miracle, a light shining forth in the wilderness. Finally, someone realized that there might just be some systems in the world that did not require a big, fat, horky, slow, expensive, bodily effluent, memory hog of a relational database!

I watched in glee as I saw BigTable, Mongo, CouchDB, and all the other cute little data storage systems begin to spring up; like little micro-breweries in the ‘80s. The beer was back! And it was starting to taste good.

But then I noticed something. Some of the systems using these nice, simple, tasty, non-relational databases were being designed around those databases. The database, wrapped in shiny new frameworks, was still sitting at the center of the design! That poisonous old relational marketing hype was still echoing through the minds of the designers. They were still making the fatal mistake.

“Stop!” I yelled. “ Stop! You don’t understand. You don’t understand.” But the momentum was too great. An enormous wave of frameworks rose up and smashed down on our industry, washing over the land. Those frameworks wrapped up the databases and fought to grab and hold the center of our applications. They claimed to master and tame the databases. They even claimed to be able to turn a relational database into a NoSQL database. And the frameworks cried out with a great voice heard all over the land: “Depend on me, and I’ll set you free!”

The name of this article is “No DB”. Perhaps after that rant you are getting an inkling of why I named it that.

The center of your application is not the database. Nor is it one or more of the frameworks you may be using. The center of your application are the use cases of your application.

It makes me crazy when I hear a software developer describe his system as a “Tomcat system using Spring and Hibernate using Oracle”. The very wording puts the frameworks and the database at the center.

What do you think the architecture of that system would look like? Do you think you’d find the use cases at the center of the design? Or would you find the source code arranged to fit nicely into the pattern of the frameworks? Would you find business objects that looked suspiciously like database rows? Would the schema and the frameworks pollute everything?

Here’s what an application should look like. The use cases should be the highest level and most visible architectural entities. The use cases are at the center. Always! Databases and frameworks are details! You don’t have to decide upon them up front. You can push them off until later, once you’ve got all the use cases and business rules figured out, written, and tested.

What is the best time to determine your data model? When you know what the data entities are, how they are related, and how they are used. When do you know that? When you’ve gotten all the use cases and business rules written and tested. By that time you will have identified all the queries, all the relationships, all the data elements, and you’ll be able to construct a data model that fits nicely into a database.

Does this change if you are using a NoSql database? Of course not! You still focus on getting the use cases working and tested before you even think about the database; no matter what kind of database it ends up being.

If you get the database involved early, then it will warp your design. It’ll fight to gain control of the center, and once there it will hold onto the center like a scruffy terrier. You have to work hard to keep the database out of the center of your systems. You have to continuously say “No” to the temptation to get the database working early.

We are heading into an interesting time. A time when the prohibition against different data storage mechanisms has been lifted, and we are free to experiment with many novel new approaches. But as we play with our CouchDBs and our Mongos and BigTables, remember this: The database is just a detail that you don’t need to figure out right away.

Questions and Developers

14 May 2012

Have you ever had a question you wanted to ask, but didn't?

Have you ever asked a question then immediately regretted asking it?

Have you ever been asked a question that made you rethink the problem you were trying to solve?

Have you ever struggled along a problem, broke down, asked somebody, and got an immediate answer?

Questions have the power to expose problems, gather information, refine solutions, and save time. Questions can also waste time, make people feel stupid, and shift discussion to the wrong topic. Knowing what questions to ask can be the difference between an average developer and a great developer. Questions cause learning but can also be scary to ask. However, there are times when developers must be brave and ask difficult questions.

Questions Induce Learning

If you don't think critically and ask questions, you won't learn anything. Since learning is a huge part of a developer's job, questions need to be asked on a frequent basis. Three major types of questions asked by developers are about details, concepts, and opinions.

When adding code to a large codebase, it often saves time to ask another developer how the details of some code works. When a developer is asking a question about code, there is a good chance that there is some opportunity to refactor. Like a dog, code is usually only clean if you just got done cleaning it. So make sure to ask questions and refactor accordingly whenever possible.

There are too many concepts for one developer to know in the software industry, so they specialize. Since they can't specialize in everything, questions are a necessity. I have found that passionate developers will never be bothered by questions about their specialty. They love their field and they love sharing knowledge.

Another question you won't see a passionate developer turn down is one that is asked to investigate an opinion. The code in an application is a reflection of the developer's knowledge and opinions. Understanding these opinions is crucial to writing code that matches the rest of the application and forming your own opinions. It is important to understand the intent behind the code before you change it, even if the style is harmful or messy.

Questions Are Terrifying

Nothing gets the nerves firing more than asking a question when you are uncomfortable. When I was an apprentice asking questions was very intimidating at times. The major fear is that I would be exposed as being a complete n00b. That fear was pretty valid because I was a complete n00b. I eventually realized that asking good questions earned myself respect as a developer.

During a code retreat I got to pair with somebody new to developing software. Our pairing session was pretty ineffective because my pair was afraid to ask me questions. He knew conceptually how to solve the problem, but wasn't familiar with Ruby's syntax. When it was his turn to code, he could've walked me through his idea and then asked how to represent that in Ruby. Instead he would spend time writing code that was incorrect syntax and confusing.

In successful companies, employees help each other out for the greater good of the company. If you are in a work environment where questions are seen as an annoyance, get a new job. How can you expect to learn if you get the cold shoulder from a colleague every time you try to learn from him/her? If you get some resistance getting an answer out of somebody, calm them down and persist. Help them understand why it is important that you know something and ask again.

Asking Questions Isn't Always Easy

There are times when asking a question can be extremely difficult. Those are typically the times when it is critical that they are brought up. If you call software your craft, it is your responsibility to ask the difficult questions.

Playing the devil's advocate for somebody with strong opinions can be a very difficult job. Asking questions that go against somebody's beliefs can go wrong very easily. It is hard for the person being questioned to not feel attacked. In this type of situation framing is extremely important. Start off your questions with "I don't understand" or "I think I'm missing something." The person being questioned will likely be more explanatory rather than defensive.

Asking questions in an apathetic environment is also a difficult task. When nobody cares about the topic, it is hard to have a discussion about it. If this is an important topic, you must ask thought provoking or scary implication questions to get their attention. That might sound like a politician's strategy, but if the topic is truly important that is an extremely effective strategy.

Conclusion

Developers rely on asking questions effectively every day. Since questions are so important, developers should be consciously aware of the roles they play in their job. Questions cause learning, can be scary, and are difficult to ask at times. I hope that after reading this blog, you understand the importance of questions and know a little bit more about the role they play in a developers job.

Advice for Novice Mentors

08 May 2012

In the past year I've started to shift into mentoring more, both at 8th Light and outside of 8th Light. Despite having been a craftsman for three years and having been an apprentice with a great mentor myself, I was woefully unprepared to teach others. Hopefully this blog can serve as advice to help novice mentors with their first apprentices. This article is an overview of the topic, and so presents a large array of ideas in general rather than focusing on any particular issue in depth.

My journey into mentoring started by teaching my friends who were interested in programming. I've also been a mentor at Code Academy for its first three semesters. Some of the knowledge in this blog can be applied to any sort of apprenticeship, while some of it may be specific to a "once a week" sort of mentor/student relationship as found in Code Academy.

The Role of a Mentor

My original image of what a mentor should be was very narrow. I thought that a mentor was there to teach the student everything they needed to know. I've since come to realize that an effective mentor/student relationship occurs when the student is responsible for their own learning, and the mentor acts as a guide along that path. The mentor is still there to help the student learn, but in addition to teaching and pairing with the student, other duties include suggesting books to read, introducing the student to people they should meet, and answering questions.

For example, in meeting with my Code Academy students, there's been a couple weeks where we weren't able to meet, due to one of us being sick or just other stuff coming up, but I was still able to answer questions through email and offer them advice. Looking back on it, I was still able to help them, just not through pairing.

Even thinking back to when I was an apprentice, I can recall a number of occasions where spending just five minutes talking to my mentor saved me hours of work.

Mentoring is Hard

Another mistake that I made was underestimating how hard it is to be a good mentor. I was definitely tricked into thinking it was easy with my first student, Cyrus. As a graduate student eventually pursing his Ph.D., he was the type of person who already knew how to learn and was eager to do so. Cyrus would have picked up programming with or without my help. Some of the other people I've had the pleasure to mentor have required a different level of guidance.

In fact, hands-on versus hands-off has been one of my biggest struggles with mentoring. Since I am a self-admitted keyboard hog, I tend to overcorrect to the side of not enough involvement. On one hand, too much involvement means that they don't get a chance to try out things for themselves. On the other hand, too little involvement means that they might spend more time than they should have struggling through their problems and aren't benefiting from your experience. Finding an appropriate balance has been a challenge, but what has helped me is more communication between myself and the student.

One Size Doesn't Fit All

That said, if you find a good balance or technique with one student, that doesn't necessarily work for every person you teach. Some students may need more assistance or different styles of teaching than others.

The most obvious difference would be students starting with two different skill levels. However, even with two students of similar skill level, they may have completely different goals or ways of thinking. A great example of this is the overall apprenticeship program at 8th Light. We've had apprenticeships lasting from as short as two months to as long as seven or eight months.

Find a Good Student

Given that a student is responsible for his own learning, finding a good student is by far the most important part in being an effective mentor. A good student has two main qualities: a passion for learning, and an ability to learn. Like my friend Cyrus, if a person has a passion for learning, they won't need too much from you.

Some prior experience with programming may be a plus, but depending on the situation, it isn't necessarily a requirement for a good student.

Get to Know the Student

After you've found a good student, the first step is to just talk to the student. Ask them about their background. Ask what they already know. Ask what they hope to learn. Even something as simple as how they prefer to learn can be useful. Most importantly, ask what they expect from you.

After that, find a way to figure out where their skill level is at beyond just talking to them. Hint: The answer is probably just pairing.

I made this mistake when I first started mentoring one of my students. We misgauged his starting skill level and had a month where it felt like we made little progress. Once we talked it over, we decided to go back to square one and had a much more successful time continuing. If we had just spent a bit more time at the beginning, I'm sure the first month of the mentoring would have far more effective. This mistake was not due to either the student or myself alone, but was more indicative of a lack of communication between us.

Have Focus

Once you have gotten to know your student, the next step is to figure out what to do during your meetings. For my Code Academy students, I've only been meeting once a week for about an hour, so before we meet, I make sure to spend some time thinking about what we could discuss.

The best case scenario is when the student comes to you with something to work on. Also, this doesn't mean you can't just pair on random code for the meeting, but you don't want to spend half your time trying to figure out what to work on.

Be Prepared

Continuing the previous thought, if you are going to present yourself as an authority on a subject, you should spend some time brushing up and reviewing it. A little preparation goes a long way.

Examples of some things I might do to prepare range from reviewing one of the SOLID principles complete with examples all the way to making sure my system is set up with whatever code we want to pair on. If you are unprepared for the meeting, it's a waste of both your time and theirs.

Be Consistent

Going along with preparation, consistency is also key. Try to schedule a regular meeting with your student. I know that one of the big hurdles I faced with my first Code Academy student was finding a time to meet each week. I was able to be a much better resource for my second Code Academy student as compared to my first. Part of that was due to having more practice at teaching, but part of it was meeting more consistently and for longer periods.

Learn About Learning

One thing I've done recently is start to read about different learning models. Putting names and stages to a person's level of knowledge can be useful for knowing when to elaborate on a subject or move on and let them internalize it on their own. I've only just started doing this, but have seen a couple small places where it's improved my teaching.

Also, use a variety of different teaching methods. Draw pictures on a whiteboard, ask leading questions, give them coding assignments or even give them reading assignments. Different people prefer different ways of learning, so this helps find the methods that work for them.

Have Fun

Most importantly, you need to have fun. Hopefully both you and your student look forward to your meetings. One thing that I've seen work as both a fun and useful exercise is to have your student take ownership of a slightly larger application that takes multiple iterations to complete. Not only is building something to completion fun, but this gives you something easy to choose to work on and forces the student to deal with code he wrote.

The exact application would depend on the student in question and the technologies he focused on learning, but many of 8th Light's internal applications have been started as apprentice projects.

Closing Notes

Hopefully this helps you to not make some of the same mistakes as I have when you start to teach others. That said, I'm still a novice when it comes to teaching, so I still have a lot to learn myself.

Like with coding, look to people who are good at mentoring for examples on how to effectively teach. Mentoring is an art, and there are a lot of people who are already very skilled at it. Also, if you have a different style of teaching that works for you, then go ahead and use it. Even at 8th Light we have a wide array of mentors with a variety of teaching styles, all of which have contributed to the knowledge of past and present apprentices.

A Scope Heuristic

01 May 2012

One of my first exposures to TDD was through a code kata. I was mortified.

The performer test-drove his way to a simple function for factoring prime numbers. He added tests one by one:

It should factor 1 correctly
It should factor 2 correctly
...

To me, the order in which the tests were added did not suggest any pre-meditated approach. It did not suggest any deep understanding of the problem domain. The performer did not seem to think that writing a factorization function required any forethought at all! "If I were to test drive this problem," I thought, "I would at least start by test-driving the helper function that is necessitated by my Algorithm. When the kata was over, nobody voiced any of the criticisms I had. I was upset and confused.

My feelings on TDD have evolved somewhat since then. I have become significantly more enthusiastic about the practice, but I am still trying to understand how exactly it is helpful. In particular, that day's kata illustrated a tension which I have not resolved for myself.

This tension is between thinking far ahead, and focusing on immediate concerns, when test-driving a change. In some cases, it's clear that you need to leave the computer and think through your options. In others, it's clear that you need to start typing and let the software evolve.

Some of my most frustrating software development experiences have been rooted in this tension. I have been burned by my lack of foresight: it has allowed me to let poor design decisions cement. Conversely, I have spent hours fretting about which approach to take, only to find out that my worries were ultimately irrelevant.

Perhaps you are thinking to yourself that the answer is obvious: always put a lot of thought into your changes. If so, I agree with you, but you are missing the point! One of the benefits of TDD is that it can keep me from thinking too far ahead; it does not cause me to think any less. Sometimes, by shortening my sights, TDD will prevent me from writing complicated things that I don't need, or from worrying about non-issues.

I am slowly learning to distinguish the situations where I can reap this benefit from those where I cannot.

I would like to suggest a rough classification of the situations where it makes sense to embrace short-term thinking.

When a coding decision is tightly coupled to the structure of existing code, and is cheap to revise, you should try to reap the benefits of short-term thinking.
When a coding decision is costly to revise, you should not try to reap the benefits of short-term thinking.

Decisions coupled to the problem domain, for example, often fall into category 2. There is often a high price to pay for misunderstanding the problem domain, because this understanding informs your high-level approach. Moreover, thinking deeply about the entire problem domain can yield high returns.

Refactoring tasks often force you to make decisions which fall into category 1. When you refactor incrementally, it should be cheap to revise any mistakes that you make. Moreover, it is hard to accurately anticipate the outcome of a refactor. You have to interact with the codebase to know what approach to take, and the best way to do this is by test-driving small changes.

Where does this classification hold up?

Suppose, for example, that you would like to write an AI module for a zero-sum two-player board game. This task is tied to the problem domain, and long-term thinking can pay dividends. If you choose the wrong algorithm, you will have to rewrite essentially the entire module; this task therefore falls into category 2.

Without thinking far ahead, you could test-drive your way to a module that works pretty well. Unfortunately, you are liable to write tests haphazardly, producing an inelegant pile of control flow statements. Once you choose to use an algorithm based in game theory, your approach crystallizes. It's easy to start working methodically towards a solution, with tests along the lines of,

The AI looks 0 moves into the future.
The AI looks 1 move into the future.
The AI looks arbitrarily into the future.

Now, suppose you anticipate duplicating code in the next feature that you add to this two-player game. Maybe you think that the "game over" function in your main game loop will resemble the "stop searching into the future" function of your AI module. The anticipated refactor will require you to think about the structure of the existing code. This refactor will also (hopefully) be cheap to revise; it therefore falls into category 1.

It would be a bad idea to jump ahead and spec out a function which eliminates duplication. It makes more sense to focus on immediate concerns, adding the feature so that the duplicated code exposes itself. Even as I write this article, I am creating and eliminating duplication.

Where does this classification fail?

This classification essentially falls apart when it comes to deciding how information should travel through a program. I will refer to this as software "workflow". Decisions about software workflow do not fall cleanly into either side of my classification, and I am not sure how to approach making them. Why don't decisions about workflow fall cleanly into either side of the classification?

First, workflow decisions are very costly to revise. Changing existing workflow can be extremely difficult because it involves updating interfaces; this can break your dependency inversion measures, requiring you to make cascading changes [1]. Workflow, therefore, fits into category 2.

Second, workflow is tightly coupled to the code that you write. Your workflow decisions dictate which modules exist and what their interfaces look like. Consequently, you cannot think very deeply about workflow without getting tripped up by complications in the existing code. So workflow decisions also partially fit category 1.

What can we learn from our inability to classify decisions related to workflow? I think we have gained an understanding of why workflow is hard to get right. Workflow is complicated, so it demands incremental evolution. Existing workflow is also extremely hard to alter, so incremental evolution can be costly and difficult.

Conclusion

It is unfortunate that my heuristic does not apply to every challenge a developer faces. I think that my heuristic will offer some value to developers who, as I once was, are confused about the benefit of test-driven development.

[1] For more information on why workflow is costly to revise, see the discussion of the StableDependencies principle in PPP. In the nomenclature of the text, changes to workflow often result in changes to "stable" pieces of software.

Why I Chose to be an Apprentice

30 Apr 2012

Last year, when I was looking for post-graduation jobs, I had no idea that I would end up at 8th Light. I went to a lot of career fairs and interviewed with a lot of companies. After a while I was offered a job by a company located in downtown Chicago. It was a .NET shop specializing in enterprise eDiscovery software. Even though I had no idea what that meant at the time, they offered me a good salary and I thought that the .NET world would be a fun adventure. So, I accepted the job and moved to Chicago after graduation.

I worked there for 6 months. The company was great and I enjoyed my time there, but, it became apparent to me very quickly that the enterprise .NET world was not for me.

After a few months of searching for a new job, I stumbled across 8th Light. I was immediately attracted to the idea of Craftsmanship. I wanted to be a part of their passionate and skilled group of developers. It seemed like the perfect fit. I submitted a Tic-Tac-Toe and an application without too much hope, but to my surprise I was offered a Resident Apprenticeship.

Choosing whether or not to become an apprentice at 8th Light was hard. There were a few things that made me nervous about joining. First, 8th Light apprentices are not paid as much as the Craftsman, which meant that I would have to take a pay cut to become an apprentice. At the time, my wife was unemployed, so taking on a new job that paid less would be a stretch on the budget. Second, there was no guarantee that I would be hired on as a Craftsman after completing the apprenticeship. This uncertainty was my biggest concern about becoming an apprentice. Coming to 8th Light meant that I would get paid less, work a lot harder, take out the trash, do the dishes, and still have no guarantee of a job in six months. I wasn't sure if I could do it. But here's what made me take the leap.

8th Light is committed to teaching apprentices how to be skilled and passionate software developers. Because of this commitment, there is no better place to learn and grow. I realized that, even if I was not offered a job at the end of my apprenticeship, my time at 8th Light would not have been wasted; spending a few months in an environment geared towards my professional growth would be better for my career than spending a few more months in my old job. And at the end, the experiences and skills that I gained as an apprentice would be invaluable to future employers. The decision was obvious, I had to be an apprentice.

In retrospect, the apprenticeship was very challenging, but I am extremely grateful that I accepted. Now that I am a Craftsman, my experiences as an apprentice mean even more to me. It was a great time of personal and professional growth; I cannot imagine myself without it. If you find yourself in a position like mine, do not fear the uncertainty of apprenticeship. You can only gain from spending some time learning from the Craftsmen at 8th Light.

Ruby Accessors Considered Pernicious

26 Apr 2012

One Ruby feature I fell in love with is the ability to define methods like `foo` and `foo=` which are used like simple variable accessors. This can be a very powerful abstraction. Yet now I'm convinced that putting logic in these accessors is often the wrong tool for the job.

The entire beauty of such methods is that they appear to be simple accessors, emulating a struct, but they can execute arbitrary logic behind the scenes. This means you can define Tweet#text and it can pretend to have its content on hand, while in reality it's actually making an HTTP request!

The problem is, life isn't that simple. Anything more complex than an actual getter or setter, whether reading the contents of a file, making an HTTP or TCP request, or executing some SQL, has the capacity to fail.

And when these actions fail, we typically have some Plan B that's not just "oh no everyone panic!" (which is exactly what letting the exception get to the top level does). Plus, it's unsightly to wrap simple accessor expressions in begin/rescue/end blocks.

There are also times when an API doesn't make it clear that some expression which looks like a simple accessor actually has the potential to throw custom exceptions of some kind. This is especially true when we learn about an API via pry or some other dynamic introspection tool. So even if we wanted to, we might not even realize that we need to wrap such a statement with a rescue block.

A simple and effective alternative is to execute such actions using explicit, non-accessor methods. Instead of every accessor on Tweet making real HTTP requests, they could really be simple accessors. We could then add two new methods, #save and #load, which could do the HTTP interaction and store fetched data on the object. If we designed our API like this, all of Tweet's methods would benefit from having an obvious and predictable behavior.

Custom accessors are great when dealing with calculated or derivative fields, such as deriving full_name when you already have first_name and last_name, or for various mathematical calculations. We should be kind to our fellow developers and design our APIs to be predictable by restricting our usage of custom accessor-like methods to these trivial cases and leaving the heavy lifting to explicit methods.

Polymorphism in Clojure

26 Apr 2012

“polymorphism is a programming language feature that allows values of different data types to be handled using a uniform interface” -Wikipedia

In most Object Oriented programming languages, polymorphism is tied to inheritance. In Clojure however, the concept of concrete inheritance is not built into the language. So, when I was first learning Clojure, it was hard for me to use my previous knowledge of polymorphism in the functional world. No need to fear, Clojure provides great methods for achieving polymorphism without using concrete inheritance.

As an example, this is a simple function that takes basic Clojure data and converts it to JSON.

 1 (defn convert [data]
 2   (cond
 3     (nil? data)
 4       "null"
 5     (string? data)
 6       (str "\"" data "\"")
 7     (keyword? data)
 8       (convert (name data))
 9     :else
10       (str data)))

This works, but it is not polymorphic. So let's take a look at a few ways to improve this.

Multimethods

A Clojure multimethods is a combination of a dispatch function and one or more methods, each defining its own dispatch value. The dispatching function is called first, and returns a dispatch value. This value is then matched to the correct method. Lets take a look at our previous example refactored into a multimethod.

 1 (defmulti convert class)
 2 
 3 (defmethod convert clojure.lang.Keyword [data]
 4   (convert (name data)))
 5 
 6 (defmethod convert java.lang.String [data]
 7   (str "\"" data "\""))
 8 
 9 (defmethod convert nil [data]
10   "null")
11 
12 (defmethod convert :default [data]
13   (str data))

Awesome! We have our first polymorphic solution. Now we can add more data types without altering the existing functions. Let's add a method for vectors as well.

1 (defmethod convert clojure.lang.PersistentVector [data]
2   (str "[" (join ", " (map convert data)) "]"))

Now we can also convert vectors into JSON.

There is another feature of multimethods that we can use to extend this solution further. Multimethods actually use the isa? function instead of the = function to match dispatch values to the correct method. This yields a very important feature, hierarchies. Let's open up the REPL and take a look at the classic Shape example,

1 user=> (derive ::rect ::shape)
2 nil
3 user=> (derive ::circle ::shape)
4 nil
5 user=> (isa? ::circle ::shape)
6 true
7 user=> (isa? ::rect ::shape)
8 true

Now let's see if we can apply this hierarchy system to our previous vectors method.

1 (derive clojure.lang.PersistentVector ::collection)
2 
3 (defmethod convert ::collection [data]
4   (str "[" (join ", " (map convert data)) "]"))

With this hierarchy, any type that matches ::collection will dispatch to the same method as vectors. Since there is no there is no difference between a list and vector in JSON, we can convert them to JSON in the same way, so we simply make PersistentList derive from ::collection as well.

1 (derive clojure.lang.PersistentList ::collection)

We were able to extend the multimethod to handle Lists with one line!

Its also worth noting that we could implement the same functionality without introducing hierarchies here. List and vectors already share many common interfaces, so we could use one of those instead. We simply replace our vector method with this

1 (defmethod convert clojure.lang.Sequential [data]
2   (str "[" (join ", " (map convert data)) "]"))

Now we’re able to convert vectors and lists without using hierarchies.

Another great feature that is hard to demonstrate here is that the methods do not have to be defined in the same file as their dispatch function. This allows us to extend the functionality of multimethods that are defined elsewhere in the system or even in a 3rd party library.

Multimethods can be very useful in situations where you cannot change the clients of a function. For instance, if there are thirty other functions using the convert function, it is hard to change. However, we can refactor into a multimethod without changing any of the clients. This allows us to refactor safely without affecting any clients.

In my experience, multimethods are best used in cases where you only need to define one polymorphic function. When multimethods are used to define a group of polymorphic methods, this solution can get a little messy. However, Clojure provides great facilities for this as well, namely Protocols.

Protocols

Another way to improve our original switch statement is though Protocols. Protocols will be a little more familiar for those coming from Object Oriented languages, as they are very similar to interfaces. Let's look at our original function refactored to use a Protocol.

 1 (defprotocol JSON
 2   (to-json [this]))
 3 
 4 (extend-protocol JSON
 5   java.lang.Boolean
 6     (to-json [this]
 7       (str this))
 8 
 9   java.lang.Long
10     (to-json [this]
11       (str this))
12 
13   java.lang.Double
14     (to-json [this]
15       (str this))
16 
17   java.lang.String
18     (to-json [this]
19       (str "\"" this "\""))
20 
21   clojure.lang.Keyword
22     (to-json [this]
23       (to-json (name this)))
24 
25   nil
26     (to-json [this]
27       "null"))

In this example, we define a Protocol, JSON, which has one method, to-json. Then we use the helper macro extend-protocol to extend all of our types at once. Now, we can simply call the to-json method directly on the data types themselves rather than through a conversion function.

1 user=> (to-json "1")
2 "\"1\""
3 user=> (to-json 1)
4 "1"

In the same way that interfaces are used in conjuction with classes in Java and C#, protocols, in conjunction with deftype or defrecord, can be used to define an explicit API that a type implements. For example,

 1 (defprotocol Dog
 2   (sit [this])
 3   (bark [this])
 4   (eat [this]))
 5 
 6 (deftype Terrier []
 7   Dog
 8   (sit [this]
 9     (prn "sitting"))
10   (bark [this]
11     (prn "woof!"))
12   (eat [this]
13     (prn "nom nom nom!")))
14 
15   (defn new-terrier []
16     (Terrier.))

Using this method, we can kind of mimic objects. Terrier can be instantiated and used just like any other object.

 1 user=> (def terrier (new-terrier))
 2 #'user/terrier
 3 user=> (bark terrier)
 4 "woof!"
 5 nil
 6 user=> (sit terrier)
 7 "sitting"
 8 nil
 9 user=> (eat terrier)
10 "nom nom nom!"
11 nil

Even though I don’t like mimicking objects in Clojure, sometimes it is necessary in order to use Java interfaces and objects, which you can learn more about here.

As with multimethods, type definitions do not have to be defined in the same file as their protocol definition, allowing us to implement protocols defined in other libraries.

As seen in the Dog/Terrier example, Protocols are great for encapsulating a group of related methods into one polymorphic package. However, this method of polymorphism should be used sparingly. Protocols are often abused to mimick an Object Orient approach because it is a familiar solution. However, if you do this, you will often miss out on the simplest solution, pure functions.

Functions as parameters

This is simplest form of polymorphism in Clojure. By accepting a function as a parameter, the given function proide multiple solutions with one interface (the parameters). Instead of refactoring our original function, let's consider a context in which it is commonly used. Web servers often need to convert data structures into JSON before returning them to clients. So let's consider a resource controller, which has an action to create a User given the request parameters.

1 (defn create [params]
2   (let [user (build-user params)
3         user (save user)]
4         (convert-to-json user)))

This function simply takes the request params, builds a user model out of them, saves them to the database and returns the result as JSON. We can make this controller polymorphic by accepting the JSON converter function as a parameter to the create function instead of calling it directly.

1 (defn create [params converter]
2   (let [user (build-user params)
3         user (save user)]
4     (converter user)))

Now the create function doesn’t have to know anything about the structure of the data that is returned to the user. The given converter function is polymorphic, meaning that it can convert the data into whatever format the user requests, instead of just JSON. So if the user requests XML or HTML instead of JSON, we don’t need to alter our create function; we simply need to pass in a different converter.

Conclusion

Using functions as parameters is by far my favorite form of polymorphism in Clojure. It is this simplest and most consise of all three methods; I try to use this method whenever possible. However, Multimethods and Protocols are equally powerful and provide useful ways in which to design polymorphic systems without concrete inheritence. With these methods in your tool belt, you don't have to feel as lost as I did when coming into the world of Clojure.

Apprenticeship Over College

25 Apr 2012

I interviewed a bright young woman today who is graduating from high school this spring among the top of her class. Despite having several generous scholarships at her disposal, she is considering an 8th Light apprenticeship.

"You can't tell her not to go to college!" people would scold me. So I began thinking... and concluded that college is not all it's cracked up to be these days.

1. College is not affordable. It was expensive when I was in school 15 years ago. Today it costs about three times as much. How do students afford that? Student Loans of course. But how discouraging! The majority of young adults get to start their adult lives with a mountain of debt.

TechPaladin - Sourced from NCES

2. A college degree doesn't differentiate you. The good news is that the U.S. is more educated than ever before. The bad news is that your college degree is one in a million. Sure, fifty years ago a 4-year degree made you stand out big-time, Mr. Smarty Pants. Today though, a 4 year degree just makes you an average Jane.

A college degree is not so special these days.

3. A College education doesn't guarantee a good job. Most of us grew up hearing the mantra: "Work Hard. Go to School. And you'll get a good job." Well, this was one of the hot topic of the recent Occupy Movement. People were protesting because they did work hard, they did go to school, and well, they didn't have a good job!

4. College doesn't prepare you for a career in software. This may be true for other industries, but I speak from experience with regard to the software industry. After earning a 4 year degree in Math and Computer Science, I got a job as a software engineer, and the sad truth was I had no clue how to develop the first project assigned to me. Fortunately I had great mentors to help me along.

12 years later I have interviewed countless college grads. Some amaze me with their experience and ability, but the majority are as clueless as I was.

What's a high school grad to do?

The social pressure to go to college is immense. I asked people, "Knowing what you know today, would you still have gone to college? And they told me "Yes! Because my parents would not have allowed otherwise." Even my brother, who dropped out of school to work with me, eventually quit and went back to school because of the pressure our mother put on him.

Yet, this high school grad must consider that although 4 years in college does earn her that conventional degree, it also means a huge debt, no guarantee of a job, and no real-world preparation for the work place.

8th Light Apprenticeship

The "Go to School" mindset is instilled in me to the point where I cannot in good conscience tell people NOT to go to college. Yet I can happily state the advantages of 8th Light's Apprenticeship program and, without any guilt, hire a young apprentice who decides not to go to college. That is, if they get accepted into the program.

What are the advantages?

No debt
We pay you
You gain real-world experience
You will be more skilled and more marketable in the software industry than your college faring peers
Full salary Craftsman position in a year or two
A network of contacts in the software industry

While pondering this whole issue, I came across one of our apprentices blog entry that included an RSA Animate video on education. It's a good video; watch it. In summary it says that our modern education system is broken and that we need a change. 8th Light's Apprenticeship may not be the answer, but it seems to be filling a niche.

Why is Estimating so Hard?

20 Apr 2012

Consider the Gettysburg Address:

“Four score and seven years ago our fathers brought forth upon this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal…”

Let’s ignore the profundity and melody of those remarkable words, and focus instead on the formatting. I’d like to fit the entire address on a bookmark measuring 1.5" X 8". I’ll use a mono-font that fits 10 characters per inch. And I’ll leave 0.1" on the right and left. So I can fit 13 characters across each line. The text contains 237 words. How long do you think it would take you to manually break the address up into 13 character lines, breaking those lines at appropriate spaces?

The calculation isn’t difficult. If you spent one second per word determining whether or not that word is the appropriate break point for a line, it would take you just under five minutes to break the entire address up into lines that are 13 characters or less. And the odds are you’d do it perfectly.

So how long would it take you to write a program to do it?

Now remember, you know this algorithm. You can execute it manually intuitively. During that five minute manual execution there would be no surprises, no stumbling blocks, no backing up and redoing old lines. This is an algorithm that you can execute without even thinking of it as an algorithm. You’d just do it.

So how long would it take you to write a program to do it.

Keep in mind that a program is nothing more than the detailed instructions for following a procedure; and this is a procedure you already know!

I’ll be kind to you. Don’t give me a single estimate. Give me three estimates. Tell me how long it will take in the best case, the worst case, and the nominal case. Go ahead, write these three numbers down. Now.

Got em? OK, now write the program. Make sure it works. I’ll wait here until you are done.

* * *

Done? How’d you do? Most people need 30-45 minutes to get this working. I’ve seen it done in 15, and I’ve seen it done in 90. Did you fall within your estimate range? Or did you blow the range completely?

Of course lots of people blow the range completely. One of the reasons they blow it is that they estimate it based on how easy the manual task appears to be. You think to yourself: “I could split those lines in 5 minutes by hand, so writing the program ought to be trivial.” We are sadly mistaken.

I remember sitting down with Kent Beck one afternoon to write this algorithm just for fun. I figured it would take us 10-15 minutes. He and I paired on it, test first, for 30 minutes, and got nowhere. Eventually we gave up because we were teaching a class together and actually had to spend time with the students.

But the experience stuck with me. Why was that algorithm so hard – for us – at that particular time? Why was it so hard to write down the procedure for doing something so basic and intuitive?

Answer: Because when we do it manually, we don’t follow a procedure. What we do instead it continuously evaluate the output and adjust it until it’s right. A procedure is blind. It doesn’t look at the output to see if it’s right. If the procedure is wrong, the output will be wrong. Period. But we, humans, are goal seekers. The goal is to split the lines up to no greater than 13 characters, and so we evaluate every line. We look it over and adjust it until it meets the goal. And we can do that in 5 minutes.

It turns out that we don’t know the procedure. We haven’t got any clue to just how difficult the procedure is. We aren’t computers. We don’t follow procedures. And so comparing the complexity of the manual task, to the complexity of the procedure is invalid.

This is one of the reasons that estimates are so hard, and why we get them wrong so often. We look at a task that seems easy and estimate it on that basis, only to find that writing down the procedure is actually quite intricate. We blow the estimate because we estimate the wrong thing.

Try this. Break some long string of text up into columns that are 10 characters long. Each time you break a line, record the position of the break, and why you decided to use that position. If you are good at abstracting, you’ll likely come up with three different scenarios for breaking a line. 1. you break it at the 10th character of a word if that word is longer than 10 characters. 2. You break it at the 11th character if that character is a space. 3. You look backwards from the 10th character looking for a space and if there is one, you break it there.

These three scenarios still need to be arranged into a procedure, but at least you now know how many elements that procedure contains. Knowing that makes the procedure much easier to estimate.

The moral of this story is that tasks that appear easy for a human to solve are often described by complex procedures. So when estimating, make sure you aren’t affected by the apparent ease of that task. Look below the surface to try to enumerate the number of procedural elements.

And if anyone tries to tell you that your estimate is bogus because the task is so simple, ask them to write down the procedure for tying their shoes.

8th Light Blog

Deliberate Practice

“Education comes from within; you get it by struggle and effort and thought.” — Napoleon Hill

Impose Artificial Constraints

Learn New Languages & Paradigms

Switch Domains

Challenge Yourself

NO DB

Questions and Developers

Questions Induce Learning

Questions Are Terrifying

Asking Questions Isn't Always Easy

Conclusion

Advice for Novice Mentors

The Role of a Mentor

Mentoring is Hard

One Size Doesn't Fit All

Find a Good Student

Get to Know the Student

Have Focus

Be Prepared

Be Consistent

Learn About Learning

Have Fun

Closing Notes

A Scope Heuristic

One of my first exposures to TDD was through a code kata. I was mortified.

Where does this classification hold up?

Where does this classification fail?

Conclusion

Why I Chose to be an Apprentice

Ruby Accessors Considered Pernicious

One Ruby feature I fell in love with is the ability to define methods like foo and foo= which are used like simple variable accessors. This can be a very powerful abstraction. Yet now I'm convinced that putting logic in these accessors is often the wrong tool for the job.

Polymorphism in Clojure

“polymorphism is a programming language feature that allows values of different data types to be handled using a uniform interface” -Wikipedia

Multimethods

Protocols

Functions as parameters

Conclusion

Apprenticeship Over College

I interviewed a bright young woman today who is graduating from high school this spring among the top of her class. Despite having several generous scholarships at her disposal, she is considering an 8th Light apprenticeship.

What's a high school grad to do?

8th Light Apprenticeship

Why is Estimating so Hard?

“Education comes from within; you get it by struggle and effort and thought.”
— Napoleon Hill

One Ruby feature I fell in love with is the ability to define methods like `foo` and `foo=` which are used like simple variable accessors. This can be a very powerful abstraction. Yet now I'm convinced that putting logic in these accessors is often the wrong tool for the job.