An audience with James Bach10 min read

Posted on by

In my last post I spoke about the time I arranged for James Bach to visit my work place and talk to our Software Testing Community.

Here is the full video of the event: https://t.co/u7JzjD1ak1

Here are my raw, unedited notes from the event:

Companies that have no dedicated software testers?

  • Developers are the most important part of a software development team
  • You can’t ship software without devs
  • So it’s right that testers should be seen as second class to devs

Agile vs Context Driven Testing

  • Agile Community:
    • There should be no specialist
    • “Development is not a speciality – because we can all write code”
    • Biased opinion
  • Quality is everyone’s responsibility

The essence of testing

  • To gain critical distance from the thing you are analysing
  • Same as writing an important email – get a friend/colleague to review it – spot obvious mistakes
  • They have critical distance
  • Testers give the team critical distance – this is difficult if you’re too close to the product (like developers)
  • Worst thing a tester can do is be a cheerleader for the product
  • Cheerleader vs Security Guard: Different mind-sets / emotional states
  • Being a cheerleader/being too close/too positive interrupts the suspicions mind-set of a tester, meaning they will fail to ask the right questions or spot the right mistakes
  • Let the tester think the worst, so nobody else has to!

Advocacy

  • Tester are there but have to continually justify their existence to people who have never studied testing or human biases (like firefighters are happy when there’s a fire – it justifies their existence, even if they don’t enjoy seeing people suffer)
    • As a result Google/Facebook have big stories in the news and suffer fines, but have enough money to not care.  No problems they can’t absorb
    • Just $25k for Google street view lawsuit
  • As testers we need to learn to justify ourselves
  • We need to learn the latest tech language e.g. DevOps
  • Used to have test managers to do this, but now every tester needs to learn how to justify their place
    • Middle management has been removed
    • Middle management used to push back and protect the testers
  • Canned responses – everyone become their own advocate

Testing in agile teams

  • Testers are debilitated by the pressure of getting stories across the board
  • Testers feel like they’re on an assembly line
  • Push back by citing the agile manifesto:

Testing in production?

  • TiP: Testing in Production
  • eBay: Live Site Testing
  • Very important, should become a sub-speciality
  • Testing something also in use by real users – in a race with users
  • Identify issues as soon as possible
    • Continuously monitor production:
      • Leverage users:
        • Twitter key word monitoring
        • Customer feedback forms (testers job to read these)
        • Make friends with Customer Service team
          • “Have you ever wanted to have a louder voice within the development team? I can help you do that, let’s work together”
      • Test in prod yourself:
        • Not automated checking
        • Testing prod to spot issues
    • Developers see feedback and say “user error”
    • Tester sees feedback and says “interesting, let’s see if I can make sense of this and raise a formal bug”

Testing vs Checking

  • Testing vs FACT checking
  • Learn the distinction or destroy your job security
  • As a tester you see lots more things than just the simple asserts in an automated check
  • Exploratory testing: all testing is exploratory
  • Scripted testing: fact checking /automated
  • An automated script without a tester is like a spiders web without a spider
    • It may still catch something but it can’t become food unless the spider is there to eat it
    • An automated check or tool may catch a bug but it goes unnoticed unless a tester is there to investigate and draw attention to it
  • Exploratory testing
    • Using social competence – tools do not have this.  Neither does machine learning
    • Continual learning
    • How important is this to my customer / stakeholders
    • Analysing the information and making sense/use of it
    • You’re testing with a rich mental model of the product
    • What to investigate and what to ignore
    • This takes times to develop an understanding of what is important and what is not
      • Feedback from testing
      • Feedback from conversations with colleagues and business people
    • This helps to focus testing on certain areas
  • As a tester need to learn about the subject you’re testing
    • e.g. Visual Search on ASOS app: Learn about machine learning

Critical Distance vs Social Distance

  • We want to maximise critical distance
  • We want to minimise social distance
  • However, these are intrinsically linked, reduce one and the other reduces
  • Requires testers to be disciplined
  • Test coaches / leads – their job to help testers to maintain critical distance
    • Pep talks
    • Tools and techniques
    • Explaining and emphasising the importance of critical distance and asking critical questions – this is the testers job
  • Requires trust between developers and developers

ATDD / Specification by example

  • James Bach “hates” ATDD and Specification By Examples!
    • Because they state that “Acceptance Tests are equivalent to a specification”
    • This, in his opinion, is a fundamental misunderstanding of what a specification is and what testing is for
      • Requirements are statements of what the customer wants “I want a product that can do this…”
      • Whereas testing is about critiquing a product “What will the product do in this specific instance…”
      • Tests and specs are not equivalent
    • A spec is like the principle – “I shall not kill”
    • “Did I murder someone today” is a check you can perform against that principle
    • But this does not prove that you are a law abiding citizen
    • … what if you murder someone tomorrow?
    • … or what if you murder someone at night when nobody is looking?
    • The principle / spec has not changed, but the “acceptance tests” are not able to confirm that you have never or will never murder someone
  • Acceptance Tests are useful – they can check fundamental facts about the product
  • The term “Acceptance Tests” is misleading, because they are not sufficient to “accept” the product
  • “Acceptance” is a critical assessment (this includes the automated checks)
  • Logic of Verification
    • “Can only verify something that has a truth value associated to it”
    • Good enough to ship does not have a true/false value
    • This is like automated checking
  • We can Assess whether a product is good enough to ship – this is an opinion
    • No number of verifications can add up to an assessment
    • Human judgement is required to provide an assessment – social competence is required
  • Automation practices can lead to goal displacement
    • Focus moves away from delivering a useful, working product
    • Goal becomes to get just get the tests green
    • This means we lose sight of what it is we are trying to achieve

Arguing and debating about testing

  • Build a collegial network of people who you can be yourself with, without the fear of upsetting or offending anyone
  • Helps to think critically
  • Encounter people with different philosophies
  • Debates on Twitter (not necessarily the best place for an argument)
  • Blog posts and comments/discussions on then
  • Attending conferences
  • Study social science and learn about biases
    • Book: Naturalistic Enquiry (Guba)
    • People frame facts in certain ways to get their own points across
    • “What are the threats to validity”
      • Triggers you to consider the things that haven’t been tested for or the threats that haven’t been considered

V-Model

  • James Back think the v-model is “stupid”!
  • Waterfall style project structure, predicated on the fact that you knew exactly what you were building from the very start
  • Agile models allow for requirements to evolve over time through a process of continual learning
  • If you write a set of testing recommendations or test plan, chances are nobody will read it straight away.  That’s fine, it can take people time to process the information and understand the implications of your plans.  Don’t be disheartened, just be open and available as soon as people are ready to start discussing it
  • Objections tend to emerge over time as the project evolves
  • We need to create a safe space for people to belatedly criticise and discuss your testing recommendations
  • Agile is full of loops, it helps us to do this

QAs vs Testers

  • Quality Assurance or Quality Assistance – These would require involvement in the defining processes to improve quality
  • If you really care about quality you need to focus on craftsmanship. Software craftsmen can create quality, but testers without skilful software developers will struggle to deliver real quality software
  • As a QA, in order to drive craftsmanship you will need to become a developer expert, in order to assist the developers to become better software craftsmen/women.
  • But James is not a QA he is a tester
  • Testers do not create quality, instead they try to see the truth, and help the developers and management see the truth so they can then create the quality software
  • A tester should not have an opinion about quality, instead they should only have an opinion about reality and “cast light so others can see things clearly”, and provide information to others so that they can then make decisions about quality

Test artefacts

  • Test artefacts are very costly
  • They don’t necessarily cost much to create but they can cost a lot to maintain
  • Treat artefact creation like brining a child into the world – you will have to support it for a long time, so take it seriously, as it can become a burden
  • Make artefacts lean and as maintainable as possible
  • Be ruthless and delete artefacts that are not offering value
  • Regularly review your documentation: Don’t neglect artefacts/documents/code as they can become a burden to the future
  • Magic Oak Tree Syndrome – Don’t think of historic artefacts as magic, because they’re not.  Don’t trust them – question them and try to demystify them
  • Try and find the original creator and ask them about them
  • 10 minute handover documents are great and allow the next person to better understand the value of the artefacts

Test strategy

  • Tests are rarely challenged about their testing decisions and their approach
  • Instead testers are asked how many test cases have they written and whether they have been automated yet!
  • Not asked questions about how they arrived at their test cases, what oracles they used or what principles they applied to determine they had sufficient functional, data, platform and structural coverage of the product

Test coverage

  • Need to define test coverage. Coverage with respect to what model of the product?
    • State coverage
    • Requirements coverage
    • Risk coverage
    • Functional coverage
    • Oracle coverage
    • Code coverage – this can only teach you 2 things about the product
      • It tells you that a percentage of the product hasn’t even been looked at (unless coverage is 100%)
      • It tells you whether or not the product has been set up for unit testing or not

Test oracles

  • “The means by which you recognise a bug”
  • An oracle gives you the ability to recognise a bug that is occurring
  • No oracles means that you will never recognise a bug
  • If you have seen a bug, then you must have some test oracles – it tells you that something bad has happened
  • Coverage and oracles go together – you need to observe/cover the product to find the bugs, and you need oracles to recognise them as bugs

Modelling

  • Mental models are at the core of testing
  • You cannot test a product without having a mental model of that product
  • Models allow you to systematically cover certain aspects of a product, rather than incidentally covering them as a consequence of your other testing activities
Share this post: