How I Build Tech Talks: Part 1

How I Build Tech Talks: Part 1

Last year, I created my first tech talk and delivered it to several venues. I enjoyed the experience, so I decided to produce a new talk this year. This time though, I want to document the process of building the talk while I’m working on it. I think it’s important to see things unfold while they happen. Often, you see a polished talk and you don’t see the work that has gone into it. This isn’t meant to be a list of recommendations from on high, but a log of how I like to build talks. Feel free to use what works and forget what doesn’t.

Qualifications

Last year, I gave two versions of my D3.js talk in five different venues. I’ve also delivered several talks at work. Additionally, I did speech and debate in college for two years. While I’m no Scott Hanselman, I know my way around a lectern.

Step 1: Topic selection

Last year, I picked a topic I didn’t know much about (D3.js). I picked that topic because I knew building a talk would force me to learn about it. This year, I want to deepen my understanding of a topic I have experience with. I work with ASP.NET MVC in my current job. Additionally, I enjoy building UI components. When I started using ASP.NET MVC a few years ago, I had a hard time finding information. The books on MVC cover the basics well, but the reference material is incomplete. (Example) Doing a talk on building UI in ASP.NET MVC would deepen my own knowledge while helping others.

Building UI in ASP.NET MVC is a broad topic, so I drafted a list of subtopics to help me narrow my scope down.

Here’s my, clustered into similar topics:

HTML Helpers
Editor and Display Templates
Custom Validation

Grunt / JS automation
LESS and SASS
Bundling and Minification

Web Performance

UI Testing (like Selenium)
JavaScript Testing (like QUnit)

That’s way too many things for an hour long talk. Each of the above clusters could be it’s own talk. To narrow the scope, I’m going to remove anything that doesn’t have to do with building UI components. This leaves me with the following list:

HTML Helpers
Editor and Display Templates
Custom Validation

Step 2: Generate a Theme

Now that I have a topic list, I want to create a theme for the talk. A theme gives structure to the talk and provides a narrative for people to latch on to. I believe that in software development, you should bake good defaults into easy to use components. This lowers the cost of development and increases the quality. Good components allow you to create a great user experience without having to drop a ton of code on each page. My theme for this talk is going to be how to build better software in less time.

Another thing I’m trying to do with this talk is build in more stories. Stories are one of the primary ways people communicate. Studies show that stories engage emotions, which provides a stronger connection than raw data.

Step 3: Generate an Abstract

The next step is generating an abstract. An abstract is a summary of you talk that you can submit to conferences and user groups. It’s important to have a strong abstract because that’s how your talk is picked by conference organizers.

Keys to a Good Abstract

  1. A catchy title.
  2. A clear benefit to the audience.
  3. A summary of the topic.

Here’s my abstract:

ASP.NET MVC UI Recipes: Build Better Interfaces With Less Code

Who doesn’t want to get more done in less time?

ASP.NET MVC gives us an excellent toolset for building web applications. Unfortunately, due to it’s rapid evolution, good documentation is hard to find. However, using some simple techniques, you can build user interface components that can speed up development while maintaining a clear separation of concerns. In this presentation, we’ll learn how to build custom controls, templates, and custom validation. Save time while creating cleaner code.

Step 4: Make An Outline

The next step is building an outline. I don’t memorize my speeches, so the outline is what I study when I prepare. I use my outline to put my topics and demos into a logical structure. I tend to capture a lot of detail in my outlines and then make a simpler outline for reference. I haven’t finished my outline yet, so I’m not going to share it.

Conclusion

This is the first in a series of blog posts documenting my process for building tech talks. As I mentioned before, feel free to use what works and forget what doesn’t.

Telling Stories With Data and D3.js (Video)

If you missed out on my D3.js talk from last year, the fine people at MadJs recorded it.

Psychology for Software Developers: Dunning-Kruger Effect

The more I learn about software development, the more I notice the things I don’t know. Each tactic I pick up or lesson I learn just leads to more questions. It’s like lighting a candle in the dark and realizing that you’re in a room full of doors. Then, as you open each door and turn on the light, you find more rooms to explore. When I started out, I was so sure of myself. Each problem had the “one right way” solution. Now, when I see a problem, I see several imperfect solutions, each with their own trade offs.

This seems like a common problem.

John Sonmez feels the same way:
Simple Programmer: The More I Know, the Less I Know

So does Scott Hanselman:
I’m a Phony. Are You?

They aren’t the only ones either.

Psychologists call this the Dunning–Kruger Effect.

The Dunning-Kruger Effect is when unskilled people overestimate their abilities and skilled people underestimate their abilities. As you become better at something, you realize how little you know. It’s like getting to the top of the mountain only to find that you are at the bottom of a way bigger mountain. It sucks.

This concept has huge implications in software development. Software development is a complex field filled with people who have strong opinions. This creates situations where confident, unskilled people have an advantage when determining technical direction. This can lead to poor choices and technical debt. It’s easy to trust someone who’s confident over someone who isn’t sure of the right answer. Even when not being sure of there is a right answer is the right answer. (This explains most of politics, but I digress…)

What can we do to counter this effect during technical discussions?

If you’re unsure of yourself, don’t be afraid to present the facts and make your case anyway. It’s better to be wrong than to allow bad ideas into your project. You probably know more than you think.

If you’re confident of a particular solution, look for alternative view points. Let the other side make their case. Realize that the person who has a few years of experience on you may know something you don’t. Even if they have trouble articulating that knowledge, it’s your responsibility to listen. You may learn something.

If you’re a manager trying to decide between two viewpoints, don’t let the louder person automatically win. It’s easy to side with people who are confident or articulate, but it’s important to hear out both sides.

What about when you’re just feeling unskilled?

For me, the best way to feel competent is to write code. The code doesn’t care how smart you feel that day. If it works, you win. For me, adding value makes me feel competent. If I can pull off some complex functionality or learn something new, I feel like I know what I’m doing.

OR/M Pain: Maybe Stored Procedures Aren’t So Bad

When I first started using OR/M, I looked forward to not writing boilerplate SQL code. As a lazy developer, I prefer to automate boilerplate code generation. After using OR/M for couple of years, I am beginning to miss stored procedures. OR/Ms like Entity Framework reduce the need for cookie cutter code, but they have some major flaws.

The first flaw is that OR/Ms are a leaky abstraction. The goal of any abstraction is to hide the internals of the thing you are trying to abstract. OR/Ms fail at this. Entity Framework, the OR/M I use the most, falls down on complex queries. For example, if you want to join in a related entity, you can use the Include() statement. The problem with Include() is that it generates horrendous queries that have terrible performance. You can’t do more than three or four includes in a single query. You end up having to either do a bunch of small queries or build a stored procedure. This defeats the purpose of OR/M.

The second flaw is that configuring complex relationships can be painful. I’ve used nHibernate (Fluent API) and Entity Framework (Code First,). Configuration in both OR/Ms is painful. I have spent hours trying to configure a particular database relationship. I’ve done small model changes generate migration scripts that left me scratching my head. One major advantage of Entity Framework is that the documentation is excellent. MSDN does a good job of highlighting common configurations, and there’s lots of information on the web.

The final flaw of OR/M is that you lose the ability to use a lot of features of the database. For example, until recently, you couldn’t create a unique key in Entity Framework. Entity Framework still doesn’t support database triggers. You can live without these features, but it does limit what you can do with OR/M.

This leads us to the obvious question:

Are OR/Ms a waste of time?

The answer, of course, is “umm… maybe”.

If you have a project with a simple data model or low performance requirements, OR/Ms will save you time. If your project is more complex, then you should consider skipping the OR/M. There’s nothing wrong with using stored procedures. Sometimes the old ways are the best ways.

More on OR/M Pain:
Bendyworks: Actually Using the Database

Seldo.com ORM is an anti-pattern

Building Antifragile Software

Software developers spend most of their time taming chaos into orderly systems. Along the way, there’s often lots of random issues that can throw you off track. Heavy processes like waterfall development sought to eliminate as much risk as possible. Unfortunately, this risk avoidance often lead to lots of pain at the end of a project. After several decades of failure, we now use agile practices that expect chaos. Dealing with chaos and risk are important for anyone who builds software for a living.

I recently read Antifragile by Nassim Nicholas Taleb. Antifragile is one of those books that changes how you see the world. Once you understand the concept of antifragility, it becomes a useful analytic tool. As a software developer, I started thinking about how to build less fragile software.

Summary

Antifragile is about how some things become better when exposed to volatility. For example, when you lift a heavy weight, your body makes itself stronger. It becomes stronger in response to stress. Additionally, if you never exercise, your body becomes weak. The human body needs some stress to be healthy. The author talks about three different states.

Fragile

Fragile things suffer from volatility. Think of a teacup. If you drop it, it will break. Fragile things tend to be rigid and over-optimized. Think of a complex code base with lots of dependencies. One small change can cause the whole system to come crashing down. Systems that are fragile tend to suffer from a small number of devastating events.

Resilient

Resilient things are indifferent to volatility. Think of a rock. If you drop it, nothing happens. Resiliency can come from redundancy or structural integrity. Think of a well structured code base with lots of automated tests. Failures are generally caught and corrected for before they can cause a problem. Many of the principles of software development revolve around building resilient software programs.

Antifragile

Some systems go a step beyond resiliency and become antifragile. Antifragile things actually gain with disorder (up to a point). Changes are generally frequent and have a small impact. Free market economies are a good example of antifragile systems. Individual businesses are fragile, but the system is strong because the best businesses survive. Evolution is another example of an antifragile system. Individuals of a species can die, but the best adapted animals get to reproduce. This causes the species as a whole to get stronger with volatility.

Other summaries of Antifragile

Art of Manliness – Beyond “Sissy” Resilience: On Becoming Antifragile

Wikipedia

So how does this help us make better software?

There are lots of software development practices that captialize on chaos. The key is to use frequent failures as learning expiriences. Here are some examples:

Agile Software Development

Agile processes revolve around building software in tight iterations with frequent customer feedback. Frequent exposure to criticism reduces the impact of criticism. It also allows you to gather more accurate information about what the users want. Additionally, your process continually improves, which means your team becomes more effective over time. This is in contrast to waterfall processes that front-load understanding and are fragile. One mistake in understanding can have a large and painful impact in waterfall projects.

Frequent releases / Continuous Integration

If you do hard things frequently, like releasing software, they become easier. A frequent release cycle breeds processes that make deployment easy and reversible. Continuous integration allows companies like Facebook to push several updates a day. This gives them a huge advantage over slower companies.

Chaos Monkey

The chaos monkey is a process build by engineers at Netflix. It is a program that randomly breaks processes. The introduced chaos allows developers handle problems in a controlled way. It also encourages resilient design in the first place. Contrast this to companies that avoid chaos. Random failures occur at inconvenient times and the fix is likely to consist of bubblegum and duct tape.

More Info: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

Minimum Viable Product

A minimum viable product is a small product used to test the market. Because the product is cheap to build, you can run many iterations. Each iteration allows you to learn more about your potential customers. These experiments allow companies to find profitable products quickly. This process allows smaller companies to exploit market situations that bigger companies cannot.

Conclusion

The world of software development is chaotic. We should not fear the chaos, but instead build systems that capitalize on it. Modern software development practices expect change and use it to create better software. Using these processes, we can build software that isn’t just resilient, but is also antifragile.

If you’re interested in learning more, check out Antifragile

Book Review: Soft Skills by John Sonmez

Have you ever wished that you had a manual for how to live the good life? A guidebook that will tell you how to make the best decisions. I certainly do. John Sonmez has taken a crack at this task with his latest book, Soft Skills: The Software Developers Life Manual.

Summary

The core premise of Soft Skills is that being a good software developer starts with being a good person. By “good”, I mean someone who has their life under control, makes good life decisions, and can achieve their goals.

Soft Skills is a book about a lot of things. It covers general career skills, like interviewing, setting career direction, and marketing yourself. It also covers skills that increase your job performance, like learning and productivity. Additionally, it covers general life skills, like managing your finances and fitness (mental and physical). Soft Skills reads like a summary of many of the top business and self help books.

While the book covers many topics, it does so in bite sized pieces. Chapters run around 5-10 pages each, so it’s easy to read and reference. Since most of the topics stand alone, it’s also easy to skip around and focus on the topics that interest you the most.

Highlights

These are a few concepts from the book that I found useful.

Business Mindset

Instead of having an employee mentality, software developers should treat themselves as a business. Their primary asset is their ability to produce working software. Adopting this mindset makes it easier to think clearly about career choices and gives you a sense of autonomy. Like any other business, you must invest in improving your assets (aka. human capital), and market your skills.

Career Direction

Software developers have three routes for advancement, each with advantages and disadvantages. This book dissuaded me from considering freelancing as a future career goal. The routes are:

  • Employment – High stability, but lower income and lower freedom.
  • Freelance – More freedom and money, but also more hassle and risk.
  • Entrepreneur – High risk, but near infinite upside on rewards and lots of freedom.

Personal Marketing

Marketing is multiplier that can catapult your career into the stratosphere. You marketing yourself by helping out your fellow developers. You can do this in a vareity of ways, including writing, speaking at conferences, and making videos.

Finance

The finance information in this book is great. John “retired” at age 33, so you know he’s doing something right. I enjoyed reading about the ups and downs on his journey to financial independence. He covers a variety of topics, from general finance to the different types of investments.

Attitude

One of my favorite things about John Sonmez, whether it’s his books or his videos, is his attitude. John is upbeat and pragmatic. He doesn’t believe in zealotry, which is something that plagues our industry.

Who’s this book good for?

This is a good reference book for any software developer. People who are early in their careers will derive the most benefit, but it’s a good book even you’re a few years into your career. I wish I had this book about eight years ago when I was graduating college. If you are well read in the topics the book covers, you may find the content a little shallow, but I’m happy with my purchase.

Links:
Soft Skills (Amazon) (Manning)
The author’s web site: Simple Programmer

Navigating the Mobile Development Landscape – September 2014

Smartphone and tablet adoption has exploded in the past five years. This trend has encouraged me to start exploring mobile development technologies. I found that there are lots of ways to build apps for mobile devices. Several mobile development technologies leverage my current web development skills. I’m going to cover the four primary ways to build mobile applications.

1. Responsive Design

The easiest way for a web developer to get started in mobile development is to build mobile-friendly web applications. You can build web applications that work on most screens by applying CSS tactics like media queries and relative sizing. Front end frameworks like Bootstrap and Foundation make it easy to build responsive web apps. Responsive design should be the default for any new web application. You never know when someone is going to try to browse your app with their Internet enabled toaster.

Responsive design is great because it has the flattest learning curve for a web developer. The other huge advantage is that you don’t need to build a separate app for each platform.

Unfortunately, building a mobile web app means you have to design for the lowest common denominator. You don’t get access to the platform specific features like a native app. You also have to deal with network issues and browser quirks. Additionally, most people on mobile devices prefer to use apps instead of their web browser.

Links:
http://getbootstrap.com/
http://foundation.zurb.com/

2. Hybrid Application

Hybrid applications are another way to use existing web skills to build mobile apps. You develop hybrid apps using standard web technologies like CSS, JS, and HTML5. They have some access to platform specific features via JavaScript apis. Unlike pure web applications, hybrid apps can be sold in app stores alongside native apps.

At first glance, hybrid development sounds like a great deal. A single code base that can be used on all the major mobile platforms and still have access to platform specific functionality. The people who make hybrid application platforms claim that hybrid apps compose the majority of top apps in the app store.

Unfortunately, there are serious flaws to the hybrid development model. The first is that hybrid apps are much slower than their native counterparts. Performance is better than a few years ago, but hybrid apps are never going to be as performant as native apps. Additionally, hybrid apps don’t behave like other native apps. There are lots of transition effects and other native behaviors that are hard to replicate in a hybrid app.

Links:
http://www.telerik.com/appbuilder
http://cordova.apache.org/
http://phonegap.com/

3. Xamarin

The Xamarin platform offers the ability to build cross platform mobile applications using C#. With Xamarin, you build the user interface for each platform using C# wrappers for native libraries. You can then use shared libraries for all the core business functionality. Xamarin claims that you can have up to 95% code reuse on their platform.

Xamarin is great for .NET developers who want to build mobile apps because using tools they are familiar with. You can get near native performance and significant code reuse between platforms.

Xamarin is not perfect though. Xamarin is expensive and it comes with a steep learning curve. Not only do you need to learn the APIs for each mobile platform, you also need to learn Xamarin.

Links:
http://xamarin.com/
http://www.whitneyland.com/2013/05/why-i-dont-recommend-xamarin-for-mobile-development.html

4. Native Applications

The final, and most obvious option, is to build separate apps for each native platform you wish to support. While its inconvenient to have to build separate apps, going 100% native has some advantages. First, each app will have a native look and feel without having to resort to outside libraries. Second, pure native development offers the best performance. Finally, all the major native app platforms have great references and development resources. There will be better support and more open source code for native developers than any of the hybrid platforms.

Links:
http://developer.windows.com/en-us
https://developer.android.com/index.html
https://developer.apple.com/devcenter/ios/index.action

Conclusion

I’ve spent a lot of time reading about different mobile development options. Additionally, I watched demos of several different mobile development plaforms. (Specifically, native Android, Window, and iOS developement, several Xamarin demos, and Telerik’s App Builder.) Based on that experience, I’m going to focus on mobile web applications and native development. Mobile web applications leverage my current skills as a web developer. I also think the mobile web option will become more popular as mobile web browsers get better. Hybrid development looks promising, but based on what I’ve learned, it’s a “good enough” solution at best. Native development provides the best user experience and is a prerequisite to “combination” platforms like Xamarin. Overall, there’s no shortage of ways to build a mobile app.

4 Steps to Better Non-Fiction Reading

I love to read. I’ve read over 40 non-fiction books in the past year (and all five Game of Thrones books). One of the problems I ran into is that when you read a lot, it becomes hard to recall what’s in specific books. Reading is a waste of time if you can’t recall what you’ve read.

To solve the retention problem, I use a four step process to remember more of what I’ve read. I base this process on what I’ve learned from a speed reading class I took in college and reading about the psychology of learning. I find that following this process allows me to read more books and remember what’s in them.

This method works best on non-fiction books. Recent examples that I’ve read include Drive, So Good They Can’t Ignore You, and The Art of Learning. Most non-fiction books range from 200-500 pages and usually focus around a single primary thesis. Generally, there’s about 50-100 pages of valuable content mixed in with a few hundred pages of filler supporting material.

Core Principles

This process is based on two core principles from psychology.

1. Repetition

Repetition increases retention. Everyone who’s been through school knows this. Often, you get more out of reading something fast several times than one slow reading. Additionally, processing an idea in different ways will help you retain more knowledge. To increase retention, process knowledge in at least two different ways. It’s also important to actively process information. It forces you to recall the information and exposes gaps in your knowledge. For example, read something and talk about it with your friends or listen to a podcast and write a one page summary.

2. Schema

When we learn things, we organize the knowledge in our heads into an organizational pattern called a schema. I like to think of schema as a tree of knowledge. The basic facts are the trunk, the major functional areas are the branches, and details are the leaves. When learning a new topic, it’s important to get a high level view to create a structure on which you can attach details to. Here’s an example from: Psychology – Wikipedia. Notice how the whole field of psychology is structured into branches of increasingly grandular detail.

Process

This process focuses on building up a schema while exposing yourself to the ideas of the book several times. While it seems counter intuitive, this process doesn’t take any more time than reading books from cover to cover.

1. Gather Background Information

Begin by looking at the title, author, summaries, and reviews. I start with Amazon for reviews and Wikipedia for summaries. Google/Bing is your friend. The reason we do this is to get an idea of what the thesis of the book is and to find any useful criticisms of that thesis. This gives you a foundation to hang other facts off of and additionally gives you an idea of the other side of the argument.

For example, I recently began reading the book Grain Brain. Before I began reading, I wanted to get some background information on the book.

I started off with the Amazon page.

Decent summary and reviews:
Grain Brain (Amazon)

Most of the reviews of the book are positive, so lets find some different viewpoints. I did some Googling and found some decent resources:

A news article on the book:
http://www.theatlantic.com/health/archive/2013/12/this-is-your-brain-on-gluten/282550/

A low quality criticism of the book (lots of straw men and cherry picking):
http://www.forksoverknives.com/the-smoke-and-mirrors-behind-wheat-belly-and-grain-brain/

A much better critisism of the book:
http://chriskresser.com/do-carbs-kill-your-brain

Based on those resources, the book has useful information. There are some issues surrounding the extremeness of the authors recommendations. Now that I understand the basic premise of the book, I can focus on the details.

Another way to get a book summary is to listen to a podcast about the book. Many authors like to go on podcasts to market their books and talk about their ideas. I’ve learned about several books by listening to podcasts about them.

2. Get a feel for the book’s structure

Next, open up the book and look at the table of contents. This will give you an idea of how the book is structured and may point you to useful resources. For example, I recently finished reading Drive. When I opened the table of contents, I noticed that the author includes a short summary in the back of the book. I read that part first to get a feel for the book. After examining the table of contents, read the introduction and conclusion of the book. At this point, you should have a good idea of what the book is about and what information the author is going to use to support their thesis.

3. Read the actual book

By the time you get to this point, you should already have a good idea of what the author is going to say and how the book is structured. The advantage of pre-reading is that you can now focus on the details.

This gives you two options. First, you can read the book quickly, grabbing details that jump out at you. Most non-fiction books contain a lot of filler. It’s nice to know you can skip over it without missing anything important. I use this tactic for general business books, like Good to Great. Most of the time, there’s about 50 pages of useful information sandwiched between 300 pages of fluff. Second, you can carefully read the book. Because you’ve already learned the basics, it’s a lot easier to pick up the details. Which path you take is dependent on your interests and the quality of the book.

4. Commonplace and discuss the book

To further increase retention, you need to process the book in different ways. Preferably active ways that require you to recall the information you’ve learned. I like to talk about interesting books that I’ve read with the people around me.

Another way to increase retention is to keep a commonplace book. A commonplace is a set of notes and quotations from books that you’ve read. Many famous historical figures kept these, sometimes copying large swaths of texts into notebooks for future reference. For each book I read, I create a one page summary and file it in OneNote. Forcing yourself to recall and process your book helps to lock in the knowledge.

Conclusion

Follow these four steps to increase your retention and get more value out of the non-fiction books you read. This tactic is a great way to learn new skills and improve your life.

Telling Stories With Data: Building Interactive Data Visualizations With D3.JS

This is a text version of a talk I’ve performed. The best way to use this post is to grab the demo code and follow along.

Demo Code:
https://github.com/DustinEwers/D3-js-Demos

If you’re interested in integrating D3.js, Web API, and ASP.NET MVC, I have a repo for that too:
https://github.com/DustinEwers/D3-DotNetMVC-Demos

Introduction

Ever increasing mountains of data surround us. Finding insights in that data is one of the great challenges of the 21st century. One way to make your data work for you is to visualize it. In this post, I will show you how to create interactive data visualizations using D3.js, a popular JavaScript data visualization library.

Why should I care about data visualization?

Data visualization is important because it gives us powerful tools to make sense of all the data humanity is creating. We have more data available to us then at any time in human history and we need sophisticated tools to store, manage, parse, and learn from it. Data visualization earns a place in our data toolbox because it gives us three advantages.

Understanding

One of the major advantages of data visualization is that it adds meaning to incomprehensible numbers. For example, I can tell you that the Earth has a radius of 3,959 miles and that Jupiter has a radius of 43,441 miles. It’s clear that the Earth is much smaller than Jupiter, but those numbers are too large for humans to comprehend.

I could instead show you this image:
Kepler Planets

It’s clear that Earth is tiny compared to Jupiter. The visual image shows the relationship more clearly than any set of numbers could.

Here’s another example from the creator of XKCD:

Money: A Chart of Almost All of It

Another way to increase undersanding is by telling stories with your data. Humans are hardwired to create and relate to stories. Using data visualization to tell stories can create a powerful understanding, even of complex information.

The best example of telling stories with data I’ve seen is the Health and Wealth of Nations (D3.js Version) by Hans Rosling and Gapminder.

Here’s a video of him describing it:
Hans Rosling’s 200 Countries, 200 Years, 4 Minutes – The Joy of Stats – BBC Four

This visualization tells us a complex and uplifting story of the last 200 years of human civilization. Here are a few of the things that can be learned from Health and Wealth of Nations:

  • 200 years ago: Everyone was poor and most people lived short lives.
  • 1800 – 1900: Major industrial powers like US, UK, Germany and France started gaining wealth, but most of the world was still poor and everyone still lived short lives.
  • 1900-1950: Industrial nations like Japan, the United States, and Western Europe had massive increases in both wealth and life expectancy
  • 1950-1980: Even though many countries were still poor, they had massive gains in life expectancy. The industrialized nations developed in the opposite order.
  • 1980-Present: India, China, and the rest of the developing world begin gaining wealth.

Persuasion

Having the facts is great, but good stories are more persuasive than statistics. Stories are an integral part of the human experience. We naturally tell them from childhood. Stories are so integral that all human cultures tell the same story. Look at any successful marketing campaign and you will see that stories. Data visualization is a great way to tell stories. Notice how effectively the Health and Wealth of Nations visualization combats the constant “if it bleeds it leads” mentality of the news media. The visualization tells us a story about the world while making a persuasive argument that the world is becoming a much better place to live.

Additional Reading:
http://www.copyblogger.com/marketing-stories/
http://www.presentationzen.com/presentationzen/2008/07/robert-mckee-on-the-power-of-story.html

Efficiency

Several public figures have tried to communicate, inaccurately, how much data is being produced. While solid numbers are hard to come by, it’s clear that we have more data available that in any time in human history. There is a pressing need to process that data efficiently. Humans are visual creatures. We can process an image much faster than a table of data points. Data visualization gives us a powerful tool to manage the data we’ve created. There’s a reason Excel comes with a “chart” function.

What is D3.js?

D3.js is a web based data visualization library written in JavaScript. Mike Bostock created D3.js in 2011. “D3” stands for Data Driven Documents. D3.js creates data driven documents by binding data to DOM elements.

Why D3.js?

  • D3.js uses standard web technologies like JavaScript, HTML, CSS, and SVG
  • Free and Open Source (Free as in speech and beer)
  • D3.js includes tools to create almost any standard data visualization
  • No need to install plugins like Flash or Java, which was a limitation of previous web based data visualization libraries.

Why not D3.js?

  • D3.js can be hard to learn, especially if you are not comfortable with JavaScript.
  • D3.js doesn’t include any “pre-canned” visuals, as opposed to tools like Flot
  • D3.js doesn’t support IE8 out of the box. You can jump through some hoops to make D3.js work with IE8, but support doesn’t come out of the box.
  • D3.js doesn’t hide your root data. Any data you use in D3.js is accessible to any savvy web user.

Ideal Use Case

D3.js works best when you want to build complex, web based, interactive data visualizations targeted at modern web browsers.

Getting Started with D3.js

Obtaining D3.js

You can get D3.js from a variety of places including Github, NuGet, NPM, and Bower

I encourage you to follow along with the examples as I explain them. I’ve included links to the relevant project files.

Select – Data – Enter – Append

“Select, Data, Enter, Append” is the core pattern in D3.js. You select an element, bind some data, handle the enter event, and append some DOM elements. This examples shows you how to do that. Consider it “Hello World” for D3.js.

Example Code:
loading_data.html
loading_data.js

This is the relevant code:

  var dataSet = [{
              LiteracyRate: 80,
              PercentGuessed: 22
            },
            {
              LiteracyRate: 60,
              PercentGuessed: 52
            },
            {
              LiteracyRate: 40,
              PercentGuessed: 26
            }];

  d3.select("#tagTableBody")
    .selectAll("tr")
    .data(dataSet)
    .enter()
    .append("tr")
    .html(function (dataPoint) {
        var message = "<td>" + dataPoint.LiteracyRate + "</td>";
        message += "<td>" + dataPoint.PercentGuessed + "</td>";
        return message;
    });

Here’s what’s happening:

select()
Selects a place in the DOM to append elements to. For this example, that’s a tbody element selected by it’s id. D3.js uses CSS3 style selectors, so you have several options for selectors, though I use ID and element selectors most of the time.

selectAll()
Selects the items you’re going to bind data to. Don’t worry about the fact that these elements don’t exist yet. In this case, we’re selecting table rows.

data()
Binds the data to the elements.

enter()
Handles any new data points. Anything you call after this is attached to the enter event.

append()
Adds a table row for each data point.

html()
Adds the content of the row. The html takes a function that passes in the data point. Most generator functions will accept a function parameter.

A Brief Primer on SVG

SVG stands for scalable vector graphics. It’s an xml based way to define vector images. It’s a commonly used web technology for building graphs and is the preferred way of making charts in d3.js.

Note SVG is not supported in IE8. (SVG Support)

The example code shows you how to build a circle, a group of rectangles, and a text element. If you would like to learn more about SVG, check out Mozilla’s SVG Reference.

Example Code:
svg.html

Important things about SVG:

  1. SVG elements can be styled with CSS, though SVG elements use different styles than HTML elements. Preface your svg styles with “svg” so they only apply to svg elements.
  2. SVG elements can trigger events, which is useful for triggering interaction.
  3. Many SVG elements can use relative sizes, like percentages and ems, which makes it easier to build responsive charts.
  4. (0,0) is SVG is the top left corner. This is different from the coordinate systems I’m used to where (0,0) is the bottom left corner.

Scales and Axes

When building charts, you are going to need to scale your data to your image size and build relevant axes to label your data. D3.js has several helper functions to assisting in creating scales and building the SVG for your axes. The demo below uses three different scales and renders an axis for each of them.

Example Code:
axes_scales.html
axes_scales.js

A D3.js scale is a function that maps a group (domain) of inputs to a range of values. There are three different types of scales in D3.js:

Quantitative Scales scales apply a mathematical function to build a scale.

Ordinal Scales are useful for mapping discrete values (like categories) to a scale.

Time Scales map a time range.

If you want to learn more:
D3.js Scales.

A basic scale:

// Linear Scale
var linearScale = d3.scale
    .linear()
    .range([0, height-10])
    .domain([0, d3.max(data)])
    .nice();

Here’s what’s going on:

d3.scale.linear()
Generates a linear scale function.

.range([0, height-10])
Sets the range of outputs. We’re subtracting 10 pixels to give the the scale a little bit of room.

.domain([0, d3.max(data)])
Sets the domain of inputs. d3.max() gets the largest item in the set.

.nice();
Causes D3.js to round off outputs. This prevents D3.js from creating values like “45.8530260724415” for pixel measurements.

D3.js, an axis is a function the draws an axis with the appropriate labels and tick marks. This function creates an axis for the scale above.

//Linear Axis
var linearAxis = d3.svg.axis()
                   .scale(linearScale)
                   .orient("left")
                   .ticks(5);

d3.svg.axis()
Declares an axis function.

.scale(linearScale)
Sets a scale for the axis to draw.

orient(“left”) and ticks(5)
Set up how the scale is rendered.

Handling Data Changes

As I’ve mentioned before, D3.js binds data to DOM elements. As you add, remove, and change your dataset, D3.js has events that are called. You can handle those events to add, change, and remove elements in your chart.

Example Code:
updating_data.html
updating_data.js

D3.js Data Events:

Enter occurs when data enters the set. This is the most common event.

selection.enter()
         .append("circle")
         .attr("cx", function (item, i) { return width })
         .attr("cy", function (item, i) { return height - item.Height })
         .attr("r", function (item) { return item.Radius })
         .on("click", function (item) { alert(generateMessage(item));});

In this example, when enter is triggered, we build an SVG circle and add a click handler to it.

Exit occurs when data leaves the set.

selection.exit()
         .transition()
         .attr("cx", function (item, i) { return width })
         .attr("cy", function (item, i) { return height })
         .attr("r", function (item) { return 0 })
         .attr("fill", function (item) { return "#000000" })
         .duration(1000)
         .remove();

In this example, we move the SVG element to the bottom right corner, fade it out, and then remove the element. transition() and duration() create a smooth animation as the element leaves the chart.

Update occurs when data changes.

selection.transition()
         .duration(1000)
         .attr("cx", function (item, i) { return xScale(i) })
         .attr("cy", function (item, i) { return height - item.Height })
         .attr("r", function (item) { return item.Radius })
         .attr("fill", function (item, i) { return prettyColors[item.Index % (prettyColors.length - 1)] });

To update an element, you just add the code to change the element to your selection. In this case, we’re moving the circle and changing it’s color.

Important Note
In order for the update event to work as expected, you need to set an index function to provide a key to the elements. In this example, we give each item in our data set an index and use that when the data is bound.

var selection = svg.selectAll("circle")
                   .data(dataSet, function (item) { return item.Index });

Advanced Topics

Reuse

Building a chart in D3.js takes a significant amount of code, which we don’t to have to repeat for every chart we make. To make our charts reusable, we can use the pattern that D3.js uses in it’s generator functions.

Example Code:
bar_chart.html
bar_chart.js

Basic Reusable Chart Pattern:

// Add a new object onto the d3 namespace
d3.d3Demos = {};

// Add a generator function
d3.d3Demos.barchart = function(){
  // set default values
  var width = 900;

  // Create a generator function that builds the chart.
  function generator(selection){
    selection.each(function(dataSet) {
        // Chart Building Code            
    });             
  }

  // Create setter functions that add the values to the function object.
  generator.width = function(value) {
    if (!arguments.length) return width;
    width = value;
    return generator;
  };

  // return the generator function
  return generator;
};

To call the generator function:

var data = [{y: 50, x: 11}, {y: 60, x: 36}, {y: 70, x: 53}];
// Creates the generator
var chart1 = d3.d3Demos.barchart();

d3.select("#question1").datum(data).call(chart1);

When we call the generator function, we are using the datum() (not data()) function to add the data to the selection. dataum() adds the data, but doesn’t bind it. The call() function executes a function and passes in the current selection.

If you want to learn more:
Towards Reusable Charts

Layouts

While our previous demos have focused on making charts with simple SVG elements, D3.js is capable of making complex visualizations. It does this through the use of layouts. A layout is a function that transforms your data into a format that makes it easier to build a complex visualization. There are many layouts in D3.js. In this example, we will focus on the pie chart layout.

Example Code:
responsive_pie.html
responsive_pie.js

// Create the layout function
var pie = d3.layout.pie();

//Create SVG element
var viewBox = "0 0 " + size + " " + size;
var svg = d3.select(this)
            .append("svg")
            .attr("class", "svg-content")
            .attr("viewBox", viewBox)
            .attr("preserveAspectRatio", "xMinYMin meet");

// This method generates SVG arcs
var arc = d3.svg.arc()
            .innerRadius(innerRadius)
            .outerRadius(outerRadius);

var arcs = svg.selectAll("g.arc")
            .data(pie(dataSet))
            .enter()
            .append("g")
            .attr("class", "arc")
            .attr("transform", "translate(" + outerRadius + ", " + outerRadius + ")");

arcs.append("path").attr("fill", function(d, i) {
                return colors(i);
            })
            .attr("d", arc);

    });
  }

We are passing our data set to the pie() function we created at the top of the example. The layout calculates the data we need to use the arc() function, which generates the definition of the path elements we’re using for our pie wedges.

Responsive Tactics

Anyone doing modern web development should be concerned with making their websites responsive. Because D3.js uses SVG, which is a vector based format, we have several options for making our charts responsive.

The first option is to use relative sizes like percentages or ems. This approach is useful for simple charts, but can be cumbersome for complex charts.

Another option is to make the SVG element scale to the size of it’s container. This is a flexible method, but this means everything in your chart scales, including your text. This is the method I used in the previous example. To deal with the text size limitation, I moved the legend outside of the SVG element.

If you would like to learn more making SVG responsive:
http://demosthenes.info/blog/744/Make-SVG-Responsive

Resources

Books

Interactive Data Visualization for the Web
Free Online Version |
Amazon

Developing a D3.js Edge
Amazon

Helpful Sites

Mike Bostock’s website
D3.js website
MDN SVG Reference

Repos

https://github.com/DustinEwers/D3-js-Demos
https://github.com/DustinEwers/D3-DotNetMVC-Demos

Stoic Wisdom for Software Developers: Introduction

How can a bunch of philosophers who lived over 2000 years ago help you become a better software developer? The ancient philosophy of Stoicism has tools that can help you become happier and more productive.

This post is the first part in a series where we’ll explore different techniques from Stoic philosophy and how to use them to become a better software developer.

History

Zeno of Citium founded Stoicism around 300 B.C. in Greece. The school continued through the Roman emperor Marcus Aurelius in 180 A.D. The Stoic philosophers came from a variety of backgrounds. Some Stoics, like Marcus Aurelius or Seneca, were important and powerful, while others, like Epictetus were slaves.

The modern usage of the word “stoic” would make you believe that Stoicism is about being a melancholy robot. This is not the case. The primary goal of Stoic philosophy is to focus only on the things you control. The Stoics used reason to temper negative emotions. The Stoics worked to achieve a peaceful state of mind, regardless of adversity. They found happiness and peace in a chaotic world.

To achieve this peace of mind, the stoics developed a set of mental tools. These tools can be used to deal with chaotic situations, like those encountered by software developers.

Why Pursue Stoicism?

Stoicism is useful because software development is an emotional and chaotic field.

Examples include:

Most of these situations occur because we get too emotional over things that are not in our control. Studying stocism won’t make these issues go away, but it will make it much easier to deal with them.

More Information:
http://fourhourworkweek.com/2009/04/13/stoicism-101-a-practical-guide-for-entrepreneurs/

http://fourhourworkweek.com/2011/05/18/philosophy-as-a-personal-operating-system-from-seneca-to-musashi/

http://www.ryanholiday.net/stoicism-a-practical-philosophy-you-can-actually-use/