Infinite Undo! RSS


 
 

Hello, I'm Noah Sussman

Aug
2nd
Sat
permalink
terrysdiary:

NOTHING WORKS

That’s real.

terrysdiary:

NOTHING WORKS

That’s real.

Jul
18th
Fri
permalink

Please stop trying to sell me on management positions while offering a “QA Engineer” title and salary

Recently I was sent a job posting that embodied what I will call the Pernicious Myth of the QA Automation Engineer. Here’s just one line:

Own automated testing capabilities across Web, mobile and production processes.

Um.

An engineer cannot by definition take responsibility (that’s what “ownership” means — right?) for features that cut across multiple groups. In all but the smallest companies, Web and mobile would be managed by different teams. With different MANAGERS.

Engineers don’t take responsibility for the behavior of MULTIPLE managers across different teams.

This is a Director’s job.

Managing software capabilities across multiple teams is the job of a mid-level Engineering manager such as a director or (in larger organizations) a VP.

Why not? Because decades of computer engineering experience show that it never works.

Management (with all its wondrous hierarchical levels) is responsible for behavior of people within and across teams. Engineers and designers are responsible for the behavior and organization of the product. Not the people. People are a management problem. Especially at scale. Organizations that forget this fail.

Takeaway: do not sign on for a director’s job at an engineer’s salary

I’m not saying no one should take responsibilty for the “tests and testability” of an application or service.

What I am saying is that someone should be explicity responsible for testing across the whole organizaiton and that person should be at the director or executive level. Never at the engineer or team lead level. Ever.

Jun
1st
Sun
permalink

The problem with asking where testing/qa “fit in” to devops

The problem with asking where testing/qa “fit in” to devops is that testing/qa is part of dev.

It’s a historical mistake that test/qa was marginalized to the point it’s now seen as a separate discipline.

May
30th
Fri
permalink

CI systems as a tool for scaling communication between engineers

In the tech world we worry a lot about scaling. Whenever someone comes up with software innovation, one of the first questions they are likely to hear is: nice concept, but will it scale?

But what is it that does not scale?

In growing systems, technology and process may not scale. In the life cycle of a mature / legacy / successful system, it is communication that does not scale. Communication between individuals and between teams, is universally found to be the bottleneck on execution in very large organizations.

Therefore the fact that CI can scale engineer communication almost indefinitely is critically important! It means that CI is a tool for dealing with diseconomies of scale, at least as pertains to an engineering organization.

Any "IT Crisis" then can be re-understood as hitting the steep rightward end of the diseconomies of scale curve (shown below) with costs spiking at the beginning and end of the life of the organization. The spike at the end is due to communication costs, which again: CI mitigates communication costs, at least for an engineering team.

illustration showing the cost curve for a diseconomy of scale: costs steeply dropping on the left, followed by a period of flat per-unit cost, followed at the extreme right edge of the graph by a sharp *increase* in per-unit cost as the organization grows in size

permalink

PHP Static Analysis with SonarQube: everything you need to know to get up and running on CentOS 6

example of a treemap plot of lines-of-code vs. test coverage

SonarQube (formerly known as just “Sonar”) is a tool for finding "interesting" areas of your codebase.

PHP support in SonarQube is relatively new, but I was able to get it set up with a little googling. So I thought it’d be worthwhile to share my notes.

I’ve embedded the Gists containing my configuration steps, below:

Or if you prefer you can view my SonarQube / PHP install instructions on GitHub.

permalink
May
29th
Thu
permalink
More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded — indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest.
May
28th
Wed
permalink
Why are the phrases “devops qa” and “devops testing” so ambiguous?

Because the acronym “QA” has vastly different meanings in different contexts.

Why are the phrases “devops qa” and “devops testing” so ambiguous?

Because the acronym “QA” has vastly different meanings in different contexts.

permalink

The Pesticide Paradox as explained by Boris Beizer

The following is excerpted from Software Testing Techniques, 2d. Ed. by Boris Beizer.

First Law: The Pesticide Paradox

Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.

That’s no too bad, you say, because at least the software gets better and better. Not quite!

Second Law: The Complexity Barrier

Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity.

Corollary to the First Law: Test suites wear out.

Yesterday’s elegant, revealing, effective test suite will wear out because programmers and designers, given feedback on their bugs, do modify their programming habits and style in an attempt to reduce the incidence of bugs they know about. Furthermore, the better the feedback, the better the QA, the more responsive the programmers are, the faster those suites wear out. Yes, the software is getting better, but that only allows you to approach closer to, or to leap over, the previous complexity barrier. True, bug statistics tell you nothing about the coming release, only the bugs of the previous release — but that’s better than basing your test technique strategy on general industry statistics or myths. If you don’t gather bug statistics, organized into some rational taxonomy, you don’t know how effective your testing has been, and worse, you don’t know how worn out your test suite is. The consequences of that ignorance is a brutal shock. How many horror stories do you want to hear about the sophisticated outfit that tested long, hard, and diligently — sent release 3.4 to the field, confident that it was the best tested product they had ever shipped — only to have it bomb more miserably than any prior release?

permalink
There is a myth that as the amount and intensity of testing increases, so the number of bugs in production must correspondingly decrease. THIS IS FALSE.

Complete testing is impossible because there are an infinite number of possible failure modes for any software system. Increasing the (finite) amount of testing can never reduce the (infinite) number of potential failure risks.

In fact as testing increases organizations will encounter the Pesticide Paradox in which the bugs found in production become increasingly subtle and correspondingly more difficult to debug.

There is a myth that as the amount and intensity of testing increases, so the number of bugs in production must correspondingly decrease. THIS IS FALSE.

Complete testing is impossible because there are an infinite number of possible failure modes for any software system. Increasing the (finite) amount of testing can never reduce the (infinite) number of potential failure risks.

In fact as testing increases organizations will encounter the Pesticide Paradox in which the bugs found in production become increasingly subtle and correspondingly more difficult to debug.

permalink

The weird 12 Commandments Of Software that I obeyed throughout my career!

Here are a dozen principles I encountered early on in my career. Each seemed funny when I first encountered it but… in retrospect the joke was always on me. This was because I hadn’t yet mastered the art of systems thinking.

cat as Moses with the two LOLCAT commandments: 1. IF IT FITS 2. YOU SITS

As I found out later, these aphorisms are Ha-Ha Only Serious. That is: they sound funny. But take them seriously and you can design complex systems that grow and evolve and continue to deliver value over the long term.

  1. No one can predict what users will do.
  2. Reading code is how you learn to be a great programmer.
  3. There will always be bugs in production.
  4. whatever tasks can be put off, will be put off.
  5. Buy before build.
  6. Programming is supposed to be fun!
  7. Programming is a creative endeavor.
  8. Undetectable errors are unavoidable.
  9. It’s orders of magnitude cheaper to change a design than to fix an implementation.
  10. It’s always about the next thing.
  11. The specific case has no general solution.
  12. A running system is never complete.

As I later found out at my own expense, these aphorisms are Ha-Ha Only Serious. Jokes like this exist in the hacker culture because complex systems exhibit behavior that is so counterintuitive that… well sometimes you just have to laugh.

A serious ha-ha.

May
26th
Mon
permalink

Gresham’s Law and computer programming

Gresham’s Law states that counterfeit currency will tend to be exchanged by otherwise honest actors. What does that have to do with software engineering? In the programming world, code is the currency of exchange. So Gresham’s Law in the programming world is: bad code will tend to get written by otherwise intelligent engineers.

How does Gresham’s law apply to test coverage?

Consider the case where engineers are asked by management to contribute unit tests such that code coverage remains at/above a numerical target such as 80%. There is by definition no direct business benefit to providing these tests, since tests are never seen by the customers. Therefore if it is possible to fake test contributions by gaming test coverage metrics, then engineers will tend to regard this subversion as the only ethically viable choice. Time not spent on test coverage is time spent increasing business ROI.

May
20th
Tue
permalink

Code Coverage: What is the goal?

In light of recent research serious doubt has been cast on the usefulness of numerical targets such as 80% test coverage.

Modern behavioral research by psychologists such as John Seddon, Dan Ariely and Dan Pink, suggests that numerical targets are actually harmful to the emotional well-being (or if you prefer: reduce the ROI) of knowledge workers.

what makes knowledge workers productive? Venn diagram

I have long been fascinated by the phenomenon of software teams pursuing a hard numerical target for code coverage. Therefore I have always made it a point to find out about this practice whenever I visit a software development shop.

Over the years it has always proved interesting to hear engineers’ responses to the following eleven questions:

RSA Animate — Drive: The surprising truth about what motivates us from Daniel Pink on Vimeo.

Eleven weird old questions that will reveal whether your code coverage efforts are useful or just well-intentioned?

I try to ask these questions of engineers whenever discussing a new or existing test coverage project.

  1. What is the specific, day-to-day benefit of covering every single line of code with a unit test?
  2. What would be the specific, day-to-day benefit of achieving 80% code coverage?
  3. How would a codebase (and a system) with 80% coverage behave differently than it does today?
  4. How much worse would it be to achieve, say 60% coverage instead?
  5. What about 79% coverage?
  6. Why only 80% coverage as a goal — why not 90%?
  7. What are the factors that contribute to system determinism?
  8. Specifically how would increased test coverage contribute to system determinism?
  9. When you talk about “code coverage” do you mean line coverage, branch coverage or statement coverage, or a combination of some-or-all of these?
  10. Does your current code coverage metric include files that have no tests at all?
  11. In other words, does your test coverage metric include all of your untested code, or do you only measure how well you have covered the code for which unit tests exist?

It is important to listen carefully to how these questions are answered. Does the team in fact have a reasoned answer for each of the 11 questions? Do the coverage metrics that are in use actually make sense from a business perspective? Has the team examined low-cost code quality strategies such as code review and static analysis? Is there an explicit mapping of test automation benefit to widespread organizational benefit, at least within the engineering team? Are the problems the team is facing actually soluble via the route of adding test coverage? 

It is unfortunately very easy for humans to place undue faith in numerical targets. This is afaict a consequence of our psychology. That numerical targets are intrinsically deceptive is not a problem to be solved, rather it is a serious limitation that must be considered when designing test infrastructure.

cf “Why most unit testing is waste" as well as "Stop Writing Automation.”


Dilbert on motivation

CODE COVERAGE Y U NO BIGGER?

May
19th
Mon
permalink
Functional testing is very expensive

Developer Testing reduces the cost and increases the potential scope of functional testing. Note this is not about *low* cost, rather it is about making a functional testing effort cost-effective *at all.* 

Low-level technical testing is a prerequisite to useful automation of end-to-end business cases

As small tests accumulate, they provide insight that can lead to the construction of effective medium and large tests. 

This is in contrast to starting with the largest tests possible. This approach is cheaper but I’ve never seen it lead to a great product.

Functional testing is very expensive

Developer Testing reduces the cost and increases the potential scope of functional testing. Note this is not about *low* cost, rather it is about making a functional testing effort cost-effective *at all.*

Low-level technical testing is a prerequisite to useful automation of end-to-end business cases

As small tests accumulate, they provide insight that can lead to the construction of effective medium and large tests.

This is in contrast to starting with the largest tests possible. This approach is cheaper but I’ve never seen it lead to a great product.

permalink
Software Engineering Aphorism #12: a running system is never “complete” or “finished” in any meaningful sense.

Software Engineering Aphorism #12: a running system is never “complete” or “finished” in any meaningful sense.