OR/M Pain: Maybe Stored Procedures Aren’t So Bad

When I first started using OR/M, I looked forward to not writing boilerplate SQL code. As a lazy developer, I prefer to automate boilerplate code generation. After using OR/M for couple of years, I am beginning to miss stored procedures. OR/Ms like Entity Framework reduce the need for cookie cutter code, but they have some major flaws.

The first flaw is that OR/Ms are a leaky abstraction. The goal of any abstraction is to hide the internals of the thing you are trying to abstract. OR/Ms fail at this. Entity Framework, the OR/M I use the most, falls down on complex queries. For example, if you want to join in a related entity, you can use the Include() statement. The problem with Include() is that it generates horrendous queries that have terrible performance. You can’t do more than three or four includes in a single query. You end up having to either do a bunch of small queries or build a stored procedure. This defeats the purpose of OR/M.

The second flaw is that configuring complex relationships can be painful. I’ve used nHibernate (Fluent API) and Entity Framework (Code First,). Configuration in both OR/Ms is painful. I have spent hours trying to configure a particular database relationship. I’ve done small model changes generate migration scripts that left me scratching my head. One major advantage of Entity Framework is that the documentation is excellent. MSDN does a good job of highlighting common configurations, and there’s lots of information on the web.

The final flaw of OR/M is that you lose the ability to use a lot of features of the database. For example, until recently, you couldn’t create a unique key in Entity Framework. Entity Framework still doesn’t support database triggers. You can live without these features, but it does limit what you can do with OR/M.

This leads us to the obvious question:

Are OR/Ms a waste of time?

The answer, of course, is “umm… maybe”.

If you have a project with a simple data model or low performance requirements, OR/Ms will save you time. If your project is more complex, then you should consider skipping the OR/M. There’s nothing wrong with using stored procedures. Sometimes the old ways are the best ways.

More on OR/M Pain:
Bendyworks: Actually Using the Database

Seldo.com ORM is an anti-pattern

Building Antifragile Software

Software developers spend most of their time taming chaos into orderly systems. Along the way, there’s often lots of random issues that can throw you off track. Heavy processes like waterfall development sought to eliminate as much risk as possible. Unfortunately, this risk avoidance often lead to lots of pain at the end of a project. After several decades of failure, we now use agile practices that expect chaos. Dealing with chaos and risk are important for anyone who builds software for a living.

I recently read Antifragile by Nassim Nicholas Taleb. Antifragile is one of those books that changes how you see the world. Once you understand the concept of antifragility, it becomes a useful analytic tool. As a software developer, I started thinking about how to build less fragile software.

Summary

Antifragile is about how some things become better when exposed to volatility. For example, when you lift a heavy weight, your body makes itself stronger. It becomes stronger in response to stress. Additionally, if you never exercise, your body becomes weak. The human body needs some stress to be healthy. The author talks about three different states.

Fragile

Fragile things suffer from volatility. Think of a teacup. If you drop it, it will break. Fragile things tend to be rigid and over-optimized. Think of a complex code base with lots of dependencies. One small change can cause the whole system to come crashing down. Systems that are fragile tend to suffer from a small number of devastating events.

Resilient

Resilient things are indifferent to volatility. Think of a rock. If you drop it, nothing happens. Resiliency can come from redundancy or structural integrity. Think of a well structured code base with lots of automated tests. Failures are generally caught and corrected for before they can cause a problem. Many of the principles of software development revolve around building resilient software programs.

Antifragile

Some systems go a step beyond resiliency and become antifragile. Antifragile things actually gain with disorder (up to a point). Changes are generally frequent and have a small impact. Free market economies are a good example of antifragile systems. Individual businesses are fragile, but the system is strong because the best businesses survive. Evolution is another example of an antifragile system. Individuals of a species can die, but the best adapted animals get to reproduce. This causes the species as a whole to get stronger with volatility.

Other summaries of Antifragile

Art of Manliness – Beyond “Sissy” Resilience: On Becoming Antifragile

Wikipedia

So how does this help us make better software?

There are lots of software development practices that captialize on chaos. The key is to use frequent failures as learning expiriences. Here are some examples:

Agile Software Development

Agile processes revolve around building software in tight iterations with frequent customer feedback. Frequent exposure to criticism reduces the impact of criticism. It also allows you to gather more accurate information about what the users want. Additionally, your process continually improves, which means your team becomes more effective over time. This is in contrast to waterfall processes that front-load understanding and are fragile. One mistake in understanding can have a large and painful impact in waterfall projects.

Frequent releases / Continuous Integration

If you do hard things frequently, like releasing software, they become easier. A frequent release cycle breeds processes that make deployment easy and reversible. Continuous integration allows companies like Facebook to push several updates a day. This gives them a huge advantage over slower companies.

Chaos Monkey

The chaos monkey is a process build by engineers at Netflix. It is a program that randomly breaks processes. The introduced chaos allows developers handle problems in a controlled way. It also encourages resilient design in the first place. Contrast this to companies that avoid chaos. Random failures occur at inconvenient times and the fix is likely to consist of bubblegum and duct tape.

More Info: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

Minimum Viable Product

A minimum viable product is a small product used to test the market. Because the product is cheap to build, you can run many iterations. Each iteration allows you to learn more about your potential customers. These experiments allow companies to find profitable products quickly. This process allows smaller companies to exploit market situations that bigger companies cannot.

Conclusion

The world of software development is chaotic. We should not fear the chaos, but instead build systems that capitalize on it. Modern software development practices expect change and use it to create better software. Using these processes, we can build software that isn’t just resilient, but is also antifragile.

If you’re interested in learning more, check out Antifragile

Book Review: Soft Skills by John Sonmez

Have you ever wished that you had a manual for how to live the good life? A guidebook that will tell you how to make the best decisions. I certainly do. John Sonmez has taken a crack at this task with his latest book, Soft Skills: The Software Developers Life Manual.

Summary

The core premise of Soft Skills is that being a good software developer starts with being a good person. By “good”, I mean someone who has their life under control, makes good life decisions, and can achieve their goals.

Soft Skills is a book about a lot of things. It covers general career skills, like interviewing, setting career direction, and marketing yourself. It also covers skills that increase your job performance, like learning and productivity. Additionally, it covers general life skills, like managing your finances and fitness (mental and physical). Soft Skills reads like a summary of many of the top business and self help books.

While the book covers many topics, it does so in bite sized pieces. Chapters run around 5-10 pages each, so it’s easy to read and reference. Since most of the topics stand alone, it’s also easy to skip around and focus on the topics that interest you the most.

Highlights

These are a few concepts from the book that I found useful.

Business Mindset

Instead of having an employee mentality, software developers should treat themselves as a business. Their primary asset is their ability to produce working software. Adopting this mindset makes it easier to think clearly about career choices and gives you a sense of autonomy. Like any other business, you must invest in improving your assets (aka. human capital), and market your skills.

Career Direction

Software developers have three routes for advancement, each with advantages and disadvantages. This book dissuaded me from considering freelancing as a future career goal. The routes are:

  • Employment – High stability, but lower income and lower freedom.
  • Freelance – More freedom and money, but also more hassle and risk.
  • Entrepreneur – High risk, but near infinite upside on rewards and lots of freedom.

Personal Marketing

Marketing is multiplier that can catapult your career into the stratosphere. You marketing yourself by helping out your fellow developers. You can do this in a vareity of ways, including writing, speaking at conferences, and making videos.

Finance

The finance information in this book is great. John “retired” at age 33, so you know he’s doing something right. I enjoyed reading about the ups and downs on his journey to financial independence. He covers a variety of topics, from general finance to the different types of investments.

Attitude

One of my favorite things about John Sonmez, whether it’s his books or his videos, is his attitude. John is upbeat and pragmatic. He doesn’t believe in zealotry, which is something that plagues our industry.

Who’s this book good for?

This is a good reference book for any software developer. People who are early in their careers will derive the most benefit, but it’s a good book even you’re a few years into your career. I wish I had this book about eight years ago when I was graduating college. If you are well read in the topics the book covers, you may find the content a little shallow, but I’m happy with my purchase.

Links:
Soft Skills (Amazon) (Manning)
The author’s web site: Simple Programmer

Navigating the Mobile Development Landscape – September 2014

Smartphone and tablet adoption has exploded in the past five years. This trend has encouraged me to start exploring mobile development technologies. I found that there are lots of ways to build apps for mobile devices. Several mobile development technologies leverage my current web development skills. I’m going to cover the four primary ways to build mobile applications.

1. Responsive Design

The easiest way for a web developer to get started in mobile development is to build mobile-friendly web applications. You can build web applications that work on most screens by applying CSS tactics like media queries and relative sizing. Front end frameworks like Bootstrap and Foundation make it easy to build responsive web apps. Responsive design should be the default for any new web application. You never know when someone is going to try to browse your app with their Internet enabled toaster.

Responsive design is great because it has the flattest learning curve for a web developer. The other huge advantage is that you don’t need to build a separate app for each platform.

Unfortunately, building a mobile web app means you have to design for the lowest common denominator. You don’t get access to the platform specific features like a native app. You also have to deal with network issues and browser quirks. Additionally, most people on mobile devices prefer to use apps instead of their web browser.

Links:
http://getbootstrap.com/
http://foundation.zurb.com/

2. Hybrid Application

Hybrid applications are another way to use existing web skills to build mobile apps. You develop hybrid apps using standard web technologies like CSS, JS, and HTML5. They have some access to platform specific features via JavaScript apis. Unlike pure web applications, hybrid apps can be sold in app stores alongside native apps.

At first glance, hybrid development sounds like a great deal. A single code base that can be used on all the major mobile platforms and still have access to platform specific functionality. The people who make hybrid application platforms claim that hybrid apps compose the majority of top apps in the app store.

Unfortunately, there are serious flaws to the hybrid development model. The first is that hybrid apps are much slower than their native counterparts. Performance is better than a few years ago, but hybrid apps are never going to be as performant as native apps. Additionally, hybrid apps don’t behave like other native apps. There are lots of transition effects and other native behaviors that are hard to replicate in a hybrid app.

Links:
http://www.telerik.com/appbuilder
http://cordova.apache.org/
http://phonegap.com/

3. Xamarin

The Xamarin platform offers the ability to build cross platform mobile applications using C#. With Xamarin, you build the user interface for each platform using C# wrappers for native libraries. You can then use shared libraries for all the core business functionality. Xamarin claims that you can have up to 95% code reuse on their platform.

Xamarin is great for .NET developers who want to build mobile apps because using tools they are familiar with. You can get near native performance and significant code reuse between platforms.

Xamarin is not perfect though. Xamarin is expensive and it comes with a steep learning curve. Not only do you need to learn the APIs for each mobile platform, you also need to learn Xamarin.

Links:
http://xamarin.com/
http://www.whitneyland.com/2013/05/why-i-dont-recommend-xamarin-for-mobile-development.html

4. Native Applications

The final, and most obvious option, is to build separate apps for each native platform you wish to support. While its inconvenient to have to build separate apps, going 100% native has some advantages. First, each app will have a native look and feel without having to resort to outside libraries. Second, pure native development offers the best performance. Finally, all the major native app platforms have great references and development resources. There will be better support and more open source code for native developers than any of the hybrid platforms.

Links:
http://developer.windows.com/en-us
https://developer.android.com/index.html
https://developer.apple.com/devcenter/ios/index.action

Conclusion

I’ve spent a lot of time reading about different mobile development options. Additionally, I watched demos of several different mobile development plaforms. (Specifically, native Android, Window, and iOS developement, several Xamarin demos, and Telerik’s App Builder.) Based on that experience, I’m going to focus on mobile web applications and native development. Mobile web applications leverage my current skills as a web developer. I also think the mobile web option will become more popular as mobile web browsers get better. Hybrid development looks promising, but based on what I’ve learned, it’s a “good enough” solution at best. Native development provides the best user experience and is a prerequisite to “combination” platforms like Xamarin. Overall, there’s no shortage of ways to build a mobile app.

4 Steps to Better Non-Fiction Reading

I love to read. I’ve read over 40 non-fiction books in the past year (and all five Game of Thrones books). One of the problems I ran into is that when you read a lot, it becomes hard to recall what’s in specific books. Reading is a waste of time if you can’t recall what you’ve read.

To solve the retention problem, I use a four step process to remember more of what I’ve read. I base this process on what I’ve learned from a speed reading class I took in college and reading about the psychology of learning. I find that following this process allows me to read more books and remember what’s in them.

This method works best on non-fiction books. Recent examples that I’ve read include Drive, So Good They Can’t Ignore You, and The Art of Learning. Most non-fiction books range from 200-500 pages and usually focus around a single primary thesis. Generally, there’s about 50-100 pages of valuable content mixed in with a few hundred pages of filler supporting material.

Core Principles

This process is based on two core principles from psychology.

1. Repetition

Repetition increases retention. Everyone who’s been through school knows this. Often, you get more out of reading something fast several times than one slow reading. Additionally, processing an idea in different ways will help you retain more knowledge. To increase retention, process knowledge in at least two different ways. It’s also important to actively process information. It forces you to recall the information and exposes gaps in your knowledge. For example, read something and talk about it with your friends or listen to a podcast and write a one page summary.

2. Schema

When we learn things, we organize the knowledge in our heads into an organizational pattern called a schema. I like to think of schema as a tree of knowledge. The basic facts are the trunk, the major functional areas are the branches, and details are the leaves. When learning a new topic, it’s important to get a high level view to create a structure on which you can attach details to. Here’s an example from: Psychology – Wikipedia. Notice how the whole field of psychology is structured into branches of increasingly grandular detail.

Process

This process focuses on building up a schema while exposing yourself to the ideas of the book several times. While it seems counter intuitive, this process doesn’t take any more time than reading books from cover to cover.

1. Gather Background Information

Begin by looking at the title, author, summaries, and reviews. I start with Amazon for reviews and Wikipedia for summaries. Google/Bing is your friend. The reason we do this is to get an idea of what the thesis of the book is and to find any useful criticisms of that thesis. This gives you a foundation to hang other facts off of and additionally gives you an idea of the other side of the argument.

For example, I recently began reading the book Grain Brain. Before I began reading, I wanted to get some background information on the book.

I started off with the Amazon page.

Decent summary and reviews:
Grain Brain (Amazon)

Most of the reviews of the book are positive, so lets find some different viewpoints. I did some Googling and found some decent resources:

A news article on the book:
http://www.theatlantic.com/health/archive/2013/12/this-is-your-brain-on-gluten/282550/

A low quality criticism of the book (lots of straw men and cherry picking):
http://www.forksoverknives.com/the-smoke-and-mirrors-behind-wheat-belly-and-grain-brain/

A much better critisism of the book:
http://chriskresser.com/do-carbs-kill-your-brain

Based on those resources, the book has useful information. There are some issues surrounding the extremeness of the authors recommendations. Now that I understand the basic premise of the book, I can focus on the details.

Another way to get a book summary is to listen to a podcast about the book. Many authors like to go on podcasts to market their books and talk about their ideas. I’ve learned about several books by listening to podcasts about them.

2. Get a feel for the book’s structure

Next, open up the book and look at the table of contents. This will give you an idea of how the book is structured and may point you to useful resources. For example, I recently finished reading Drive. When I opened the table of contents, I noticed that the author includes a short summary in the back of the book. I read that part first to get a feel for the book. After examining the table of contents, read the introduction and conclusion of the book. At this point, you should have a good idea of what the book is about and what information the author is going to use to support their thesis.

3. Read the actual book

By the time you get to this point, you should already have a good idea of what the author is going to say and how the book is structured. The advantage of pre-reading is that you can now focus on the details.

This gives you two options. First, you can read the book quickly, grabbing details that jump out at you. Most non-fiction books contain a lot of filler. It’s nice to know you can skip over it without missing anything important. I use this tactic for general business books, like Good to Great. Most of the time, there’s about 50 pages of useful information sandwiched between 300 pages of fluff. Second, you can carefully read the book. Because you’ve already learned the basics, it’s a lot easier to pick up the details. Which path you take is dependent on your interests and the quality of the book.

4. Commonplace and discuss the book

To further increase retention, you need to process the book in different ways. Preferably active ways that require you to recall the information you’ve learned. I like to talk about interesting books that I’ve read with the people around me.

Another way to increase retention is to keep a commonplace book. A commonplace is a set of notes and quotations from books that you’ve read. Many famous historical figures kept these, sometimes copying large swaths of texts into notebooks for future reference. For each book I read, I create a one page summary and file it in OneNote. Forcing yourself to recall and process your book helps to lock in the knowledge.

Conclusion

Follow these four steps to increase your retention and get more value out of the non-fiction books you read. This tactic is a great way to learn new skills and improve your life.

Telling Stories With Data: Building Interactive Data Visualizations With D3.JS

This is a text version of a talk I’ve performed. The best way to use this post is to grab the demo code and follow along.

Demo Code:
https://github.com/DustinEwers/D3-js-Demos

If you’re interested in integrating D3.js, Web API, and ASP.NET MVC, I have a repo for that too:
https://github.com/DustinEwers/D3-DotNetMVC-Demos

Introduction

Ever increasing mountains of data surround us. Finding insights in that data is one of the great challenges of the 21st century. One way to make your data work for you is to visualize it. In this post, I will show you how to create interactive data visualizations using D3.js, a popular JavaScript data visualization library.

Why should I care about data visualization?

Data visualization is important because it gives us powerful tools to make sense of all the data humanity is creating. We have more data available to us then at any time in human history and we need sophisticated tools to store, manage, parse, and learn from it. Data visualization earns a place in our data toolbox because it gives us three advantages.

Understanding

One of the major advantages of data visualization is that it adds meaning to incomprehensible numbers. For example, I can tell you that the Earth has a radius of 3,959 miles and that Jupiter has a radius of 43,441 miles. It’s clear that the Earth is much smaller than Jupiter, but those numbers are too large for humans to comprehend.

I could instead show you this image:
Kepler Planets

It’s clear that Earth is tiny compared to Jupiter. The visual image shows the relationship more clearly than any set of numbers could.

Here’s another example from the creator of XKCD:

Money: A Chart of Almost All of It

Another way to increase undersanding is by telling stories with your data. Humans are hardwired to create and relate to stories. Using data visualization to tell stories can create a powerful understanding, even of complex information.

The best example of telling stories with data I’ve seen is the Health and Wealth of Nations (D3.js Version) by Hans Rosling and Gapminder.

Here’s a video of him describing it:
Hans Rosling’s 200 Countries, 200 Years, 4 Minutes – The Joy of Stats – BBC Four

This visualization tells us a complex and uplifting story of the last 200 years of human civilization. Here are a few of the things that can be learned from Health and Wealth of Nations:

  • 200 years ago: Everyone was poor and most people lived short lives.
  • 1800 – 1900: Major industrial powers like US, UK, Germany and France started gaining wealth, but most of the world was still poor and everyone still lived short lives.
  • 1900-1950: Industrial nations like Japan, the United States, and Western Europe had massive increases in both wealth and life expectancy
  • 1950-1980: Even though many countries were still poor, they had massive gains in life expectancy. The industrialized nations developed in the opposite order.
  • 1980-Present: India, China, and the rest of the developing world begin gaining wealth.

Persuasion

Having the facts is great, but good stories are more persuasive than statistics. Stories are an integral part of the human experience. We naturally tell them from childhood. Stories are so integral that all human cultures tell the same story. Look at any successful marketing campaign and you will see that stories. Data visualization is a great way to tell stories. Notice how effectively the Health and Wealth of Nations visualization combats the constant “if it bleeds it leads” mentality of the news media. The visualization tells us a story about the world while making a persuasive argument that the world is becoming a much better place to live.

Additional Reading:
http://www.copyblogger.com/marketing-stories/
http://www.presentationzen.com/presentationzen/2008/07/robert-mckee-on-the-power-of-story.html

Efficiency

Several public figures have tried to communicate, inaccurately, how much data is being produced. While solid numbers are hard to come by, it’s clear that we have more data available that in any time in human history. There is a pressing need to process that data efficiently. Humans are visual creatures. We can process an image much faster than a table of data points. Data visualization gives us a powerful tool to manage the data we’ve created. There’s a reason Excel comes with a “chart” function.

What is D3.js?

D3.js is a web based data visualization library written in JavaScript. Mike Bostock created D3.js in 2011. “D3” stands for Data Driven Documents. D3.js creates data driven documents by binding data to DOM elements.

Why D3.js?

  • D3.js uses standard web technologies like JavaScript, HTML, CSS, and SVG
  • Free and Open Source (Free as in speech and beer)
  • D3.js includes tools to create almost any standard data visualization
  • No need to install plugins like Flash or Java, which was a limitation of previous web based data visualization libraries.

Why not D3.js?

  • D3.js can be hard to learn, especially if you are not comfortable with JavaScript.
  • D3.js doesn’t include any “pre-canned” visuals, as opposed to tools like Flot
  • D3.js doesn’t support IE8 out of the box. You can jump through some hoops to make D3.js work with IE8, but support doesn’t come out of the box.
  • D3.js doesn’t hide your root data. Any data you use in D3.js is accessible to any savvy web user.

Ideal Use Case

D3.js works best when you want to build complex, web based, interactive data visualizations targeted at modern web browsers.

Getting Started with D3.js

Obtaining D3.js

You can get D3.js from a variety of places including Github, NuGet, NPM, and Bower

I encourage you to follow along with the examples as I explain them. I’ve included links to the relevant project files.

Select – Data – Enter – Append

“Select, Data, Enter, Append” is the core pattern in D3.js. You select an element, bind some data, handle the enter event, and append some DOM elements. This examples shows you how to do that. Consider it “Hello World” for D3.js.

Example Code:
loading_data.html
loading_data.js

This is the relevant code:

  var dataSet = [{
              LiteracyRate: 80,
              PercentGuessed: 22
            },
            {
              LiteracyRate: 60,
              PercentGuessed: 52
            },
            {
              LiteracyRate: 40,
              PercentGuessed: 26
            }];

  d3.select("#tagTableBody")
    .selectAll("tr")
    .data(dataSet)
    .enter()
    .append("tr")
    .html(function (dataPoint) {
        var message = "<td>" + dataPoint.LiteracyRate + "</td>";
        message += "<td>" + dataPoint.PercentGuessed + "</td>";
        return message;
    });

Here’s what’s happening:

select()
Selects a place in the DOM to append elements to. For this example, that’s a tbody element selected by it’s id. D3.js uses CSS3 style selectors, so you have several options for selectors, though I use ID and element selectors most of the time.

selectAll()
Selects the items you’re going to bind data to. Don’t worry about the fact that these elements don’t exist yet. In this case, we’re selecting table rows.

data()
Binds the data to the elements.

enter()
Handles any new data points. Anything you call after this is attached to the enter event.

append()
Adds a table row for each data point.

html()
Adds the content of the row. The html takes a function that passes in the data point. Most generator functions will accept a function parameter.

A Brief Primer on SVG

SVG stands for scalable vector graphics. It’s an xml based way to define vector images. It’s a commonly used web technology for building graphs and is the preferred way of making charts in d3.js.

Note SVG is not supported in IE8. (SVG Support)

The example code shows you how to build a circle, a group of rectangles, and a text element. If you would like to learn more about SVG, check out Mozilla’s SVG Reference.

Example Code:
svg.html

Important things about SVG:

  1. SVG elements can be styled with CSS, though SVG elements use different styles than HTML elements. Preface your svg styles with “svg” so they only apply to svg elements.
  2. SVG elements can trigger events, which is useful for triggering interaction.
  3. Many SVG elements can use relative sizes, like percentages and ems, which makes it easier to build responsive charts.
  4. (0,0) is SVG is the top left corner. This is different from the coordinate systems I’m used to where (0,0) is the bottom left corner.

Scales and Axes

When building charts, you are going to need to scale your data to your image size and build relevant axes to label your data. D3.js has several helper functions to assisting in creating scales and building the SVG for your axes. The demo below uses three different scales and renders an axis for each of them.

Example Code:
axes_scales.html
axes_scales.js

A D3.js scale is a function that maps a group (domain) of inputs to a range of values. There are three different types of scales in D3.js:

Quantitative Scales scales apply a mathematical function to build a scale.

Ordinal Scales are useful for mapping discrete values (like categories) to a scale.

Time Scales map a time range.

If you want to learn more:
D3.js Scales.

A basic scale:

// Linear Scale
var linearScale = d3.scale
    .linear()
    .range([0, height-10])
    .domain([0, d3.max(data)])
    .nice();

Here’s what’s going on:

d3.scale.linear()
Generates a linear scale function.

.range([0, height-10])
Sets the range of outputs. We’re subtracting 10 pixels to give the the scale a little bit of room.

.domain([0, d3.max(data)])
Sets the domain of inputs. d3.max() gets the largest item in the set.

.nice();
Causes D3.js to round off outputs. This prevents D3.js from creating values like “45.8530260724415” for pixel measurements.

D3.js, an axis is a function the draws an axis with the appropriate labels and tick marks. This function creates an axis for the scale above.

//Linear Axis
var linearAxis = d3.svg.axis()
                   .scale(linearScale)
                   .orient("left")
                   .ticks(5);

d3.svg.axis()
Declares an axis function.

.scale(linearScale)
Sets a scale for the axis to draw.

orient(“left”) and ticks(5)
Set up how the scale is rendered.

Handling Data Changes

As I’ve mentioned before, D3.js binds data to DOM elements. As you add, remove, and change your dataset, D3.js has events that are called. You can handle those events to add, change, and remove elements in your chart.

Example Code:
updating_data.html
updating_data.js

D3.js Data Events:

Enter occurs when data enters the set. This is the most common event.

selection.enter()
         .append("circle")
         .attr("cx", function (item, i) { return width })
         .attr("cy", function (item, i) { return height - item.Height })
         .attr("r", function (item) { return item.Radius })
         .on("click", function (item) { alert(generateMessage(item));});

In this example, when enter is triggered, we build an SVG circle and add a click handler to it.

Exit occurs when data leaves the set.

selection.exit()
         .transition()
         .attr("cx", function (item, i) { return width })
         .attr("cy", function (item, i) { return height })
         .attr("r", function (item) { return 0 })
         .attr("fill", function (item) { return "#000000" })
         .duration(1000)
         .remove();

In this example, we move the SVG element to the bottom right corner, fade it out, and then remove the element. transition() and duration() create a smooth animation as the element leaves the chart.

Update occurs when data changes.

selection.transition()
         .duration(1000)
         .attr("cx", function (item, i) { return xScale(i) })
         .attr("cy", function (item, i) { return height - item.Height })
         .attr("r", function (item) { return item.Radius })
         .attr("fill", function (item, i) { return prettyColors[item.Index % (prettyColors.length - 1)] });

To update an element, you just add the code to change the element to your selection. In this case, we’re moving the circle and changing it’s color.

Important Note
In order for the update event to work as expected, you need to set an index function to provide a key to the elements. In this example, we give each item in our data set an index and use that when the data is bound.

var selection = svg.selectAll("circle")
                   .data(dataSet, function (item) { return item.Index });

Advanced Topics

Reuse

Building a chart in D3.js takes a significant amount of code, which we don’t to have to repeat for every chart we make. To make our charts reusable, we can use the pattern that D3.js uses in it’s generator functions.

Example Code:
bar_chart.html
bar_chart.js

Basic Reusable Chart Pattern:

// Add a new object onto the d3 namespace
d3.d3Demos = {};

// Add a generator function
d3.d3Demos.barchart = function(){
  // set default values
  var width = 900;

  // Create a generator function that builds the chart.
  function generator(selection){
    selection.each(function(dataSet) {
        // Chart Building Code            
    });             
  }

  // Create setter functions that add the values to the function object.
  generator.width = function(value) {
    if (!arguments.length) return width;
    width = value;
    return generator;
  };

  // return the generator function
  return generator;
};

To call the generator function:

var data = [{y: 50, x: 11}, {y: 60, x: 36}, {y: 70, x: 53}];
// Creates the generator
var chart1 = d3.d3Demos.barchart();

d3.select("#question1").datum(data).call(chart1);

When we call the generator function, we are using the datum() (not data()) function to add the data to the selection. dataum() adds the data, but doesn’t bind it. The call() function executes a function and passes in the current selection.

If you want to learn more:
Towards Reusable Charts

Layouts

While our previous demos have focused on making charts with simple SVG elements, D3.js is capable of making complex visualizations. It does this through the use of layouts. A layout is a function that transforms your data into a format that makes it easier to build a complex visualization. There are many layouts in D3.js. In this example, we will focus on the pie chart layout.

Example Code:
responsive_pie.html
responsive_pie.js

// Create the layout function
var pie = d3.layout.pie();

//Create SVG element
var viewBox = "0 0 " + size + " " + size;
var svg = d3.select(this)
            .append("svg")
            .attr("class", "svg-content")
            .attr("viewBox", viewBox)
            .attr("preserveAspectRatio", "xMinYMin meet");

// This method generates SVG arcs
var arc = d3.svg.arc()
            .innerRadius(innerRadius)
            .outerRadius(outerRadius);

var arcs = svg.selectAll("g.arc")
            .data(pie(dataSet))
            .enter()
            .append("g")
            .attr("class", "arc")
            .attr("transform", "translate(" + outerRadius + ", " + outerRadius + ")");

arcs.append("path").attr("fill", function(d, i) {
                return colors(i);
            })
            .attr("d", arc);

    });
  }

We are passing our data set to the pie() function we created at the top of the example. The layout calculates the data we need to use the arc() function, which generates the definition of the path elements we’re using for our pie wedges.

Responsive Tactics

Anyone doing modern web development should be concerned with making their websites responsive. Because D3.js uses SVG, which is a vector based format, we have several options for making our charts responsive.

The first option is to use relative sizes like percentages or ems. This approach is useful for simple charts, but can be cumbersome for complex charts.

Another option is to make the SVG element scale to the size of it’s container. This is a flexible method, but this means everything in your chart scales, including your text. This is the method I used in the previous example. To deal with the text size limitation, I moved the legend outside of the SVG element.

If you would like to learn more making SVG responsive:
http://demosthenes.info/blog/744/Make-SVG-Responsive

Resources

Books

Interactive Data Visualization for the Web
Free Online Version |
Amazon

Developing a D3.js Edge
Amazon

Helpful Sites

Mike Bostock’s website
D3.js website
MDN SVG Reference

Repos

https://github.com/DustinEwers/D3-js-Demos
https://github.com/DustinEwers/D3-DotNetMVC-Demos

Stoic Wisdom for Software Developers: Introduction

How can a bunch of philosophers who lived over 2000 years ago help you become a better software developer? The ancient philosophy of Stoicism has tools that can help you become happier and more productive.

This post is the first part in a series where we’ll explore different techniques from Stoic philosophy and how to use them to become a better software developer.

History

Zeno of Citium founded Stoicism around 300 B.C. in Greece. The school continued through the Roman emperor Marcus Aurelius in 180 A.D. The Stoic philosophers came from a variety of backgrounds. Some Stoics, like Marcus Aurelius or Seneca, were important and powerful, while others, like Epictetus were slaves.

The modern usage of the word “stoic” would make you believe that Stoicism is about being a melancholy robot. This is not the case. The primary goal of Stoic philosophy is to focus only on the things you control. The Stoics used reason to temper negative emotions. The Stoics worked to achieve a peaceful state of mind, regardless of adversity. They found happiness and peace in a chaotic world.

To achieve this peace of mind, the stoics developed a set of mental tools. These tools can be used to deal with chaotic situations, like those encountered by software developers.

Why Pursue Stoicism?

Stoicism is useful because software development is an emotional and chaotic field.

Examples include:

Most of these situations occur because we get too emotional over things that are not in our control. Studying stocism won’t make these issues go away, but it will make it much easier to deal with them.

More Information:
http://fourhourworkweek.com/2009/04/13/stoicism-101-a-practical-guide-for-entrepreneurs/

http://fourhourworkweek.com/2011/05/18/philosophy-as-a-personal-operating-system-from-seneca-to-musashi/

http://www.ryanholiday.net/stoicism-a-practical-philosophy-you-can-actually-use/

Building My First Tech Talk

Ever want to give a tech talk to your peers? I recently built and delivered my first tech talk and I’d like to share my experience.

Motivation

There are several factors that have contributed to my desire to to make a tech talk. The first is that one of my career goals is to become more involved with my local developer community. I am fortunate to live in a city that has a vibrant technical community and I wanted to contribute something to that community.

If you would like to know some of the benefits of becoming involved more in your local tech community, watch this video (It’s currently free):
Get Involved

The second reason is that when I was in college, I participated in forensics (speech and debate) and miss publicly speaking. I’ve thought about joining Toastmasters or a similar organization, but doing a technical presentation is more accessible and congruent with my career goals.

The final reason is that the best way to learn something is teach it. Teaching a subject requires in depth knowledge and the fear of looking like an idiot encourages you to thoroughly learn your subject material.

Process

The first step in build a talk is picking a subject to talk about. For this talk, I decided to demonstrate D3.js, a JavaScript data visualization library. After using it at work and seeing how powerful the library is, I really wanted to dig into it more.

I recommend picking something you are personally interested in, and not worry about whether or not you are currently an expert in your topic. Obviously don’t pick something you know nothing about, but don’t be afraid to branch out. Researching the presentation will help you become an expert.

The second step is to research. For my talk, I read Interactive Data Visualization for the Web by Scott Murray and many online tutorials. I also built lots of small demos.

After some research, start building an outline. This outline will probably change with new information, but it’s good to get an idea of what you want to talk about. Having an outline helps focus your research efforts.

While researching, I also built the demonstrations that I used in my presentation. My goal was to show the power of data visualization on the web, so the focus of my presentation was walking through lots of examples.

Tech demos have a tendency to fail, so it’s important to keep things simple and minimize dependencies. I made it a point to make sure my demos didn’t rely on any outside components. No databases, no web services, just local code. I also put my code on Github, so I could easily setup my project on another computer if I needed to.

After building the demo, and the outline, it’s time to put together your talk. If you have slides that you want to use, build those now. I had a few slides, but most of my presentation was examples and code demos.

After that, practice! Try to run through your presentation at least a couple of times. Rehearse in your head and out-loud. Give your talk to the dog or cat. Make sure your timing is good and your transitions are smooth.

Also, don’t write out your talk and memorize it. This will stifle your delivery and ability to go with the flow of your audience. Rehearse your key points and don’t worry about getting everything word for word. Your audience changes with every performance and it’s important to be able to cater to that audience. Memorizing a written speech will prevent that from happening.

Lessons Learned

So far, I’ve given my presentation twice. Once at the local .NET meetup group and once at my current employer. Both situations were different and provided different learning opportunities.

What didn’t go so well?

The one thing that surprised me was the awkwardness of transitioning in and out of PowerPoint presenter mode. My presentation initially consisted of an alternation between slides and demos of visualizations on the web. Unfortunately, PowerPoint made those transitions really awkward. To fix this, I ended up pulling those slides and just showing the demos. Less PowerPoint is not a bad thing.

I ran into another issue when I gave my presentation to my current employer. I didn’t realize that I was also giving my presentation to the non-technical staff as well as the developers. To adapt, I spent more time talking about general data visualization concepts and less time walking through code. I also told my audience that the second part of the talk was more technical. I can’t stress enough about how important it is to cater to your audience.

What went well?

I was surprised at how well the question and answer portion of the talk went. Studying up for the presentation really paid off.

I also felt like I adapted to changing circumstances well. This is probably because my forensics career in college was spent in limited preparation events, where you need to be able to think on your feet.

I put all of my talk materials on Github, so people could follow along and download the code themselves. I had a number of positive comments about that.

Tips

  1. Adapt to your audience and your situation. Don’t regurgitate a canned speech.
  2. Publicly share your demo code and slides.
  3. Beware of potential technical difficulty. Pre-load any web pages and make sure all of your code functions correctly. Make sure all of of the A/V stuff works (like font sizes and slide transitions).

Conclusion

I am glad I made this talk and plan to make more in the future. The whole experience was highly educational and it felt good to make something useful for the community. I can’t recommend it enough.

I’ll be speaking at That Conference in August, so please check out my talk.

Talk Code and Slides

3 Tips To Help You Ace Your Next Technical Interview

I used to be a little nervous about doing technical interviews. Even though I spend much of my free time learning about technology, I’m still afraid that I’ll be perceived as an idiot. To combat this feeling, I have developed several practices to improve my interviewing skills.

1. Review Your Resume

Every interview I’ve been to asks you for examples from your work history. To prepare for these types of questions, review your resume before each interview. For each job on your resume, think about the challenges you faced, what technologies you used, and what you were proud of. Think about stories you can tell from previous jobs to potential employers. Practice telling those stories in your head or in front of a mirror. This exercise primes your brain. When prompted, you will be ready with a story from your work history.

2. Practice Answering Standard Technical Questions

Depending on the technology stack and what job you are interviewing for, most employers ask a list of standard questions. You will get a feel for this list as you do more technical interviews. I’m an ASP.NET Developer, therefore I get questions on the following topics:

  • Basic OOP (Encapsulation, Inheritance, Polymorphism)
  • Abstract Base Classes vs Interfaces
  • The ASP.NET Page Lifecycle
  • String Concatenation using StringBuilders
  • Basic Design Patterns
  • Linq
  • Generics
  • Job Specific C# and ASP.NET Trivia
  • Major features in new releases of ASP.NET (async, await, etc…)

To prepare, you need to research the concept, practice describing the concept, and for more complex questions, come up with relevant examples. For example, when describing OOP concepts, come up with a novel example. Instead of the usual examples of vehicles or animals, try something a little different, like a beer factory or different types of ducks. The key here is to show that you understand the concepts, as opposed to being able to memorize a textbook example.

3. Practice Your Public Speaking Skills

I learned my public speaking skills by competing in college forensics (speech and debate). It was a great experience that has helped me in many ways. I highly recommend competitive speaking if you have the opportunity. If you are no longer in school, you can join Toastmasters or practice with a group of your peers. Meetups are a good way to find groups of people to practice with. You can also give presentations at user groups and conferences. Practicing your public speaking skills on a regular basis will help you eliminate filler words from your vocabulary and help you learn how to handle fears surrounding public speaking. If you can explain a complex technical concept to a group of thirty people, then you should have no problem with three people in a job interview.

Reprogram Your Wetware: Habit Change for the Modern Software Developer

Habits are like cron jobs for the brain. They govern a large portion of our behavior automatically. According to a study by a researcher at Duke University, more than 40 percent of our day to day actions are the result of habits. (1) Fortunately, habits can be reprogrammed.

The Habit Loop

The habit loop is composed of three parts. Once this loop is established, the habit becomes automatic. These components are:

Cue

A cue is what triggers a habit. For example, my cue to eat lunch is that the time is around noon. Cues can take many forms, including specific times (dinner at six PM), mental states (emotional eating when sad), visuals (eating candy on the counter), or preceding actions (brushing your teeth after a meal).

Routine

A routine is the action that triggers the reward. This is the behavior you are trying to change.

Reward

The reward is what you get for completing the routine. Examples include the sugar rush at the end of a candy binge or the runners high you feel after a hard workout. Rewards are not always obvious. For instance, when I was in college, I would work out with my friends on a regular basis. The reward wasn’t the energy boost from the workout, but the time I spent with my friends.

The key to changing a habit is to identify the habit loop. Once you identify the habit loop, you can change the components of the loop and change the behavior. Here are some strategies that you can use to modify your own habits:

Environment Design

The easiest way to change a habit is to modify your environment. Add cues to trigger good habits or remove the cues that trigger bad habits. For example, I like to put my weights in my living room so I’m reminded to workout. The great thing about environment design is that you can take it as far as you want to go. You could even build a Batcave for Habit Change

Swap the Routine

Charles Duhigg, in the book The Power of Habit, advocates changing routines. This is a good strategy when you don’t have control over your environment. It’s easy to hide the M&Ms in your pantry, but hard to get your office to remove a vending machine. To change a routine, learn to identify the cue that triggers the habit and then try different routines until you find one that delivers the same reward. For example, if you want to kick a morning soda habit, try switching to tea or doing 20 squats instead.

Material Decomposition

This technique comes from Stoic philosophy. It works by devaluing the reward for a habit. Take something you desire, like a new gadget, and decompose it. For example, the hot new phone is really just a metal and glass enclosure with some copper and plastic inside. Decomposition shows you that the items you desire are not special.

Keystone Habits

There are certain habits that have the potential to radically alter your life. These are called keystone habits. Keystone habits can start a chain reaction that changes other habits. If you can develop a keystone habit, you can radically alter your life.

Common keystone habits include:

  • Exercise
  • Eating dinner with your family
  • Keeping a Journal

Organizational Habits

Habits don’t just apply to individuals. Organizations also have habits. Some of these habits are documented in policies and others are a result of organizational culture. Most business books highlight organizational habits. The classic business book Good To Great is about habits that are shared by “great” companies. The Joel Test is a list of desirable habits for software development organizations.

Additional Reading

  1. The Power of Habit – Charles Duhigg
  2. See Like a Stoic: An Ancient Technique for Modern Times
  3. Good To Great – Jim Collins

Footnotes

  1. Habits—A Repeat Performance, David Neal