• You are here:
  • Home »
  • Archive: January, 2015

Archive Monthly Archives: January 2015

Telling Stories With Data and D3.js (Video)

If you missed out on my D3.js talk from last year, the fine people at MadJs recorded it.

Psychology for Software Developers: Dunning-Kruger Effect

The more I learn about software development, the more I notice the things I don’t know. Each tactic I pick up or lesson I learn just leads to more questions. It’s like lighting a candle in the dark and realizing that you’re in a room full of doors. Then, as you open each door and turn on the light, you find more rooms to explore. When I started out, I was so sure of myself. Each problem had the “one right way” solution. Now, when I see a problem, I see several imperfect solutions, each with their own trade offs.

This seems like a common problem.

John Sonmez feels the same way:
Simple Programmer: The More I Know, the Less I Know

So does Scott Hanselman:
I’m a Phony. Are You?

They aren’t the only ones either.

Psychologists call this the Dunning–Kruger Effect.

The Dunning-Kruger Effect is when unskilled people overestimate their abilities and skilled people underestimate their abilities. As you become better at something, you realize how little you know. It’s like getting to the top of the mountain only to find that you are at the bottom of a way bigger mountain. It sucks.

This concept has huge implications in software development. Software development is a complex field filled with people who have strong opinions. This creates situations where confident, unskilled people have an advantage when determining technical direction. This can lead to poor choices and technical debt. It’s easy to trust someone who’s confident over someone who isn’t sure of the right answer. Even when not being sure of there is a right answer is the right answer. (This explains most of politics, but I digress…)

What can we do to counter this effect during technical discussions?

If you’re unsure of yourself, don’t be afraid to present the facts and make your case anyway. It’s better to be wrong than to allow bad ideas into your project. You probably know more than you think.

If you’re confident of a particular solution, look for alternative view points. Let the other side make their case. Realize that the person who has a few years of experience on you may know something you don’t. Even if they have trouble articulating that knowledge, it’s your responsibility to listen. You may learn something.

If you’re a manager trying to decide between two viewpoints, don’t let the louder person automatically win. It’s easy to side with people who are confident or articulate, but it’s important to hear out both sides.

What about when you’re just feeling unskilled?

For me, the best way to feel competent is to write code. The code doesn’t care how smart you feel that day. If it works, you win. For me, adding value makes me feel competent. If I can pull off some complex functionality or learn something new, I feel like I know what I’m doing.

OR/M Pain: Maybe Stored Procedures Aren’t So Bad

When I first started using OR/M, I looked forward to not writing boilerplate SQL code. As a lazy developer, I prefer to automate boilerplate code generation. After using OR/M for couple of years, I am beginning to miss stored procedures. OR/Ms like Entity Framework reduce the need for cookie cutter code, but they have some major flaws.

The first flaw is that OR/Ms are a leaky abstraction. The goal of any abstraction is to hide the internals of the thing you are trying to abstract. OR/Ms fail at this. Entity Framework, the OR/M I use the most, falls down on complex queries. For example, if you want to join in a related entity, you can use the Include() statement. The problem with Include() is that it generates horrendous queries that have terrible performance. You can’t do more than three or four includes in a single query. You end up having to either do a bunch of small queries or build a stored procedure. This defeats the purpose of OR/M.

The second flaw is that configuring complex relationships can be painful. I’ve used nHibernate (Fluent API) and Entity Framework (Code First,). Configuration in both OR/Ms is painful. I have spent hours trying to configure a particular database relationship. I’ve done small model changes generate migration scripts that left me scratching my head. One major advantage of Entity Framework is that the documentation is excellent. MSDN does a good job of highlighting common configurations, and there’s lots of information on the web.

The final flaw of OR/M is that you lose the ability to use a lot of features of the database. For example, until recently, you couldn’t create a unique key in Entity Framework. Entity Framework still doesn’t support database triggers. You can live without these features, but it does limit what you can do with OR/M.

This leads us to the obvious question:

Are OR/Ms a waste of time?

The answer, of course, is “umm… maybe”.

If you have a project with a simple data model or low performance requirements, OR/Ms will save you time. If your project is more complex, then you should consider skipping the OR/M. There’s nothing wrong with using stored procedures. Sometimes the old ways are the best ways.

More on OR/M Pain:
Bendyworks: Actually Using the Database

Seldo.com ORM is an anti-pattern

Building Antifragile Software

Software developers spend most of their time taming chaos into orderly systems. Along the way, there’s often lots of random issues that can throw you off track. Heavy processes like waterfall development sought to eliminate as much risk as possible. Unfortunately, this risk avoidance often lead to lots of pain at the end of a project. After several decades of failure, we now use agile practices that expect chaos. Dealing with chaos and risk are important for anyone who builds software for a living.

I recently read Antifragile by Nassim Nicholas Taleb. Antifragile is one of those books that changes how you see the world. Once you understand the concept of antifragility, it becomes a useful analytic tool. As a software developer, I started thinking about how to build less fragile software.


Antifragile is about how some things become better when exposed to volatility. For example, when you lift a heavy weight, your body makes itself stronger. It becomes stronger in response to stress. Additionally, if you never exercise, your body becomes weak. The human body needs some stress to be healthy. The author talks about three different states.


Fragile things suffer from volatility. Think of a teacup. If you drop it, it will break. Fragile things tend to be rigid and over-optimized. Think of a complex code base with lots of dependencies. One small change can cause the whole system to come crashing down. Systems that are fragile tend to suffer from a small number of devastating events.


Resilient things are indifferent to volatility. Think of a rock. If you drop it, nothing happens. Resiliency can come from redundancy or structural integrity. Think of a well structured code base with lots of automated tests. Failures are generally caught and corrected for before they can cause a problem. Many of the principles of software development revolve around building resilient software programs.


Some systems go a step beyond resiliency and become antifragile. Antifragile things actually gain with disorder (up to a point). Changes are generally frequent and have a small impact. Free market economies are a good example of antifragile systems. Individual businesses are fragile, but the system is strong because the best businesses survive. Evolution is another example of an antifragile system. Individuals of a species can die, but the best adapted animals get to reproduce. This causes the species as a whole to get stronger with volatility.

Other summaries of Antifragile

Art of Manliness – Beyond “Sissy” Resilience: On Becoming Antifragile


So how does this help us make better software?

There are lots of software development practices that captialize on chaos. The key is to use frequent failures as learning expiriences. Here are some examples:

Agile Software Development

Agile processes revolve around building software in tight iterations with frequent customer feedback. Frequent exposure to criticism reduces the impact of criticism. It also allows you to gather more accurate information about what the users want. Additionally, your process continually improves, which means your team becomes more effective over time. This is in contrast to waterfall processes that front-load understanding and are fragile. One mistake in understanding can have a large and painful impact in waterfall projects.

Frequent releases / Continuous Integration

If you do hard things frequently, like releasing software, they become easier. A frequent release cycle breeds processes that make deployment easy and reversible. Continuous integration allows companies like Facebook to push several updates a day. This gives them a huge advantage over slower companies.

Chaos Monkey

The chaos monkey is a process build by engineers at Netflix. It is a program that randomly breaks processes. The introduced chaos allows developers handle problems in a controlled way. It also encourages resilient design in the first place. Contrast this to companies that avoid chaos. Random failures occur at inconvenient times and the fix is likely to consist of bubblegum and duct tape.

More Info: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

Minimum Viable Product

A minimum viable product is a small product used to test the market. Because the product is cheap to build, you can run many iterations. Each iteration allows you to learn more about your potential customers. These experiments allow companies to find profitable products quickly. This process allows smaller companies to exploit market situations that bigger companies cannot.


The world of software development is chaotic. We should not fear the chaos, but instead build systems that capitalize on it. Modern software development practices expect change and use it to create better software. Using these processes, we can build software that isn’t just resilient, but is also antifragile.

If you’re interested in learning more, check out Antifragile