Here is a new project, when can you finish it?

22 June, 2015 (19:13) | Uncategorized | By: seth

You are a seasoned software engineer.  You have plenty of projects under your belt.  With that kind of experience you should be able to tell when this brand new project will be finished….

Three are a couple of things to consider here. One is that engineers almost always under-estimate. But knowing this is not all that useful, as there is no solution to it. You can pad your estimates with an additional 25%, 50%, 100% and they will never be right… not sure why that is… possibly when we pad our estimates we sub-consciously also increase the amount of work to be done? Not sure.

What *is* more helpful is to know that at the start of any large project is when we have the maximum uncertainty about the project and when we will complete it. Over time as we work on the project, the uncertainty decreases. This is known as the Cone of Uncertainty.

[Cone of Uncertainty — Construx]

The cone of uncertainty was popularized by Steve McConnell (he wrote Code Complete) who is pretty much a waterfall guy (the x-axis for the cone above indicates this). I would rather direct you to an agile solution here. This is where Real Scrum comes in handy. Real scrum has the following

  • A Product Backlog with user stories: As a <someone> I want to <something> because <some reason> — each one describing some small increment of potentially shippable product
  • A Product Owner who understands the vision of the product and works with the engineering team to prioritize and estimate those stories in the product backlog
  • Estimates for the user stories that are unit-less — in Story Points
  • Sprints of duration of about 2 weeks
  • Sprint planning to create a sprint backlog for these sprints. Sprint backlogs are actionable tasks for the team to implement.
  • Sprint retrospectives where the sprint velocity is calculated, and improvements made to the process

It may seem like I am just spouting off a Scrum handbook, but each of these items I wrote is CRUCIAL for you to be able to accurately estimate your deliverables. Remember, you really have little idea of when the project will finish when you are first told to estimate it, but using Scrum then only 2-3 sprints into the project you should have a much better completion estimate. Here is how the above bullet items get you to a great estimate.

  • Product backlog user stories are prioritized so the most important items get worked on first
  • Product backlog user stories are estimated in story points — humans are MUCH better at relative estimates (this is 2x that) than absolute ones (this will take 3 weeks)
  • Over 2-3 sprints you will learn how many story points you can complete in a sprint — this is your velocity
  • Given your velocity you now know how many sprints it will take to get to a certain point in completed user stories from your product backlog (you have a date when all the P1 stories will be complete, you also have a later date when ALL the stories will be complete).
  • As you iterate more, and as you complete more, you will have a better idea of completion dates

Also note that Scrum must also include a Demo after each sprint. Here you will get feedback from stakeholders, which will better inform the contents of your product backlog, and help you deliver the most customer-centric product possible.

And that is how Real Scrum can help you to estimate when you can deliver that new project.

Dog bone approach to testing

22 May, 2015 (14:52) | Uncategorized | By: seth

We testers like our testing shapes”.   For example consider the Testing Pyramid


Well then, here is my contribution.   This is the dog bone approach to testing.


It is shaped like a dog bone because it is fat (more effort and resources) on both ends and skinny (less effort and resources) in the middle.   The three boxes represent:

Developer Testing

Unit Testing, Test-Driven Development, Behavior-Driven Development, Component, Inter-component

BUFT: Big Up-Front Test

End to end functional testing, Non-functional testing (see more here)

TiP: Test in Production

  • Using real (organic) and synthetic transactions and data in production
  • This is about Discovery — what we cannot know before contact with production.
  • It includes customer experience as part of the quality assessment
  • (yeah, I went back to using TiP instead of DDQ…)

Note that whenever we find a problem downstream that we could have found upstream, we roll this information back upstream — for example if we find a bug in production that could have been found with a unit test, we add that unit test and then assess why we missed that unit test in the first place.

In summary we need strong up-front testing to prevent where it is cheap and high value (the developer testing), but as test costs increase due to environment and complexity (BUFT) we only want to do as much testing as needed to gain confidence so we can dip our toes into production.   It is in production that we learn things we could not learn pre-production.

Lies, damn lies, and statistics – Your guide to misleading correlations.

10 May, 2015 (11:32) | Uncategorized | By: seth

We should trust data.  We certainly should trust data more than we trust the HiPPO.  But when we dive head-first into the data looking for answers when we ourselves do not know the questions, that can be risky.  After all, Goal, Question, Metric (GQM) was promoted (in that order) over 20 years ago. (here is the original GQM paper). 

I said risky” but I did not say wrong.   As an advanced technique the data wallow” can find insights we did not even know we were looking for.   But the risk is there that you will find a misleading correlation.


There are three types of misleading correlations: 

1. Confounding factors

The first can be understood through the maxim correlation does not necessarily imply causation”.   For example the following data are correlated

  • Use of sunscreen
  • Incidences of death due to drowning


Increased sunscreen usage is correlated to increased death by drowning.  There is a correlation.  But of course sunscreen usage does not CAUSE drowning.  Instead they both have a common cause, warm sunny weather which results in both increased sunscreen usage and increased time spent swimming.  Increased sunny weather is considered the confounding factor for the sunscreen/drowning correlation.

2. Spurious correlation

A world of gratitude goes to Tyler Vigen for educating the world about these.  Here is but one example (unlike the sunscreen chart above, below IS actual data).


These are real statistical correlations, but are meaningless in every practical sense.   They are simply correlated by chance.  The sheer number of measurements in the world being huge, there will always be random correlations that emerge by chance.

3. Type I errors (Alpha risk)

A Type I error is a false positive.  When designing a controlled experiment (or trial) to establish causation, we choose a significance level (called alpha) for the experiment.  Using the common significance level of alpha=5%, if the data we observe would only be observed in 4% (or 4.9% or 4.99%) of the cases where there is NO causation, we conclude there IS indeed causation and report that.  But this still means that there is a 4% (or 4.9% or 4.99% chance we are wrong in concluding there is causation, and this is a false positive or type I error.   Of course lowering alpha for an experiment gives us more confidence in the results — the technical name for this is specificity”, but at the cost that we might miss some true correlations.

Call for feedback

I am sure the stats pros and data folks out there can provide me with any corrections or clarifications I might need to the above.  Please feel free to do so in the comments.

How I spent my vacation…

24 April, 2015 (10:48) | Uncategorized | By: seth


Disney World, FL USA


More on the old versus new Microsoft

19 March, 2015 (09:22) | Uncategorized | By: seth

I found these quotes which tie-in well with my previous juxtaposition of Sinofsky versus Nadella

Windows 8 is a disaster. Period.

Paul Thurrott, SuperSite for Windows (pro-Microsoft author and blogger) [ref]


Windows 10 Undoes the Disaster of Windows 8 (Mostly)

David Pogue, Tech columnist, Yahoo (formerly with the New York Times) [ref]

Win10 Windows 10 preview.  Return of the start menu for keyboard and mouse users. 

Although I think that Thurott still misses the point, saying

The reason this happened is that while Sinofsky had the maniacal power and force of will of a Steve Jobs, he lacked Jobs’ best gift: An innate understanding of good design. Windows 8 is not well-designed. It’s a mess. But Windows 8 is a bigger problem than that. Windows 8 is a disaster in every sense of the word.

I have made the point before (Those pesky customers) that unless your name is Jobs, you would be best served by using real data to understand your customer and their needs.  That innate understanding” will help you to avoid delivering a faster horse, however your design still needs to be informed by genuine understanding of your customers.

Data-Driven in one image

18 February, 2015 (16:50) | Uncategorized | By: seth

Why Windows 10 will succeed where Windows 8 failed –Data Driven Quality

15 February, 2015 (22:21) | Uncategorized | By: seth


You can’t sort of A/B test your way before the product launches, because you don’t have it in users’ hands yet. You need to use your product intuition to make the right choices. You make these choices and people are paying you to make them.

Steven Sinofsky, former Microsoft President Windows

D11 Conference, May 2013



It starts with everyone in an organization having the curiosity, having the questions, trying to test out hypotheses, trying to gain insights, and then taking actions. You need to have a data culture inside of your organization.  …this is perhaps the most paramount thing inside of Microsoft. The thing we need to do better on is to be able learn from our customers and the data they exhaust … and continuously improve our products and services, that is job number one by far

Satya Nadella, Microsoft CEO

Executive Keynotes from SQL Server 2014 Launch, April 2014

(I cannot find the quote recorded anywhere, but I personally wrote it down while viewing his keynote for the SQL Server Launch. Some oblique references here, here and here)

TestOps redux?

13 February, 2015 (08:53) | Uncategorized | By: seth

At one point I attempted to coin the phrase TestOps (pivoting off the increasingly popular DevOps)

Quality in the Cloud: The new Role of TestOps – March 2012

But ultimately there were a few things I did not like about that phrase.  Primarily was that TestOps really is already a part of DevOps.  DevOps is woefully mis-named and should be EngOps or something like that since Test, Dev, PM, and other all share the same goal of producing customer-delighting software and similarly all need to leverage Ops (aka production data) to do so.

What I really prefer is the concept of DDQ (Data-Driven Quality), which includes and expands on the TestOps and Test in Production ideas.

But it looks like my friends at SOASTA are up to something with TestOps….

[And here is where was going to embed the tweet from SOASTA CEO Tom Lounibos, but he deleted it! :-) (it was here).]  Apparently he thought I objected to his use of the term, but far from it. I am more than happy to have friends in the industry like SOASTA leverage good ideas like TestOps (but I really prefer DDQ :-)

Those pesky customers

8 February, 2015 (06:55) | Uncategorized | By: seth

steve_jobs3It is funny that whenever I teach (or lecture, or cajole) software engineering teams about the necessity for customer centricity, I almost inevitably get challenged with the example of Steve Jobs’ Apple.   Jobs did not care about focus groups, Jobs did not task his teams to deeply mine customer data… he just told them what was right and they did it.  Sometimes they got an Apple Lisa out of it, but it was from Jobs’ mind that the iPod, iPhone, and Mac Air sprang forth.

The exception that does not really prove the rule

Well I do not know precisely what to do with Mr. Jobs’ legacy in this context.  I could try to argue how much of his innovation came on the shoulders of others.  The original Mac UI and mouse paradigm having started at Xerox Parc and such… but that is not really relevant.  There is no denying that Jobs’ had the ability to tap into latent need of his customers.

thMAJ42OAXLatent need?  Yup, that is when you give customers what they desperately want, but they do not even know they want it.  For example, Henry Ford is famous for saying

If I had asked my customers what they wanted they would have said a faster horse.

Nobody got rich delivering faster horses… or at least not as rich as Mr. Ford got delivering automobiles to the middle and working classes.

I am the decider

So maybe Jobs and Ford are exceptional.   That does not change the need for understanding your customers (with hard data).   Just recently I was talking to a software entrepreneur who had just launched a new app targeted at fitness trainers and the gyms they work at.   He told me he was dismayed to find all these features that were missing that trainers and gym managers wanted and needed to make the app useful to them.   I started to suggest that this was good news and this limited release gave him precious data about his customers.  His response surprised me.  He said that all these ex-Microsoft guys that work for him suggested a beta release to collect data but he HATED that idea.  He told me this was his vision, and it was that vision he wanted to see realized, resulting in great success for the app.


I am thrilled to see ex *Microsoft* guys associated with customer centricity and data-driven quality.  This is a stark change from just a short while ago, and one that I can humbly claim to have been some part of.   And I guess I should not be shocked that every entrepreneur fancies himself the next Jobs with a Jobsian vision, but I also hope that such entrepreneurs can learn that perhaps they are not, and that data about customers, their problems, and what thrilling those customers with the perfect solution looks like, is perhaps the most powerful tool for success.

Culture of data and experimentation

amazon-instant-video-06-535x535In closing consider the case of my current boss, Jeff Bezos.  Bezos certainly had (and has) a vision for Amazon, but he said early on (and later codified) that Amazon will focus what is right for the customer.  Certainly there are many Jeff projects” that go to production without a focus group and based purely on Jeff’s vision, but there is always metrics that everyone knows up front by which we will assess the idea, use to change it, or scrap it altogether.  I remember in the early days of Amazon Instant Video when we first launched streaming.  Jeff was sure that the free preview should play the first n minutes of the movie, so that is what we implemented.  The data quickly showed that this is not what people wanted, and they wanted the traditional preview/trailer.  Sure we all knew that would be the case… or did we?  Without first trying this idea we truly did not know… perhaps this is what people had been waiting for and it would be Amazon’s advantage?  It wasn’t, and that is fine… this is the culture of experimentation driven by real users and real data that a company needs to embrace to succeed.

Bing’s fault or Bartell Drug’s fault?

1 February, 2015 (13:04) | Uncategorized | By: seth

Not quite Testing in Production…..but amusing nonetheless