I ran across this page recently - a blog by a Software QA tester named James Bach:
http://www.satisfice.com/blog/archives/638
it's interesting. I also ran into another QA blog that had similar sentiments. It's a backlash against the momentum of taking Software QA into a BDD/Automation driven realm.
I was once just like that. But my opinion changed drastically when I saw the benefits to writing code, using BDD and building an automation framework.
eHarmony presented me with the opportunity to learn some code and be part of a Continuous Integration / Continuous Deployoment program. In that program, I picked up the language Groovy - and we ultimately used a BDD framework in GEB/Spock. GEB would be like Cucumber (a automation framework) and Spock is like Gherkin (the BDD language.)
An Example
James Bach's post though has some problems. In his comparison he has a BDD example of ONE test of a epic arc (Validating an ATM machine), to a manual tester covering tons of "stuff" (i.e. an entire test plan.) That's not very fair. So let's make this fair. Lets talk apples to apples.
No BDD framework would have only ONE test. It would have as many as our need to cover the entire scope of the code being developed. For a website, for example, you may have tests that cover:
- UI Navigation
- Subscription to the website
- Communication between members of the website
- Cancelations
- Each Section of functionality
- Advertising
- Special events (free communication weekends)
- Registration
Your site/company may have different requirements, but here's a few off the top of my head. We would never sum up a single BDD test to cover the entire functioning site. Who would?
Let's take an example, like a form based Registration. Simple example. Registration is just a form with fields. We would break this down to individual tests (these tests are still the same whether you are using BDD/Manual Testing/Automation Testing):
- There would be a test on how to send the data perhaps (json? service call? form post?) However we are capturing data, we have a test for it and verify the data is captured
- Another test might be front end, happy path: fill out the form in the UI and submit. Then verify the data was captured
- Another test might be violate the fields with white space
- Another test might be passing in invalid data (i.e. invalid postal code)
- Another test might be double clicking submit and verifying only one data entry is captured
- Another test might be using words on a blocked list
On and on this can go, till we have flushed out our test plan. In other words, we'd have multiple BDD's to cover each test above. Just like you would have multiple test cases in a test plan to cover the above areas of tests. That's the key. James is glossing over the fact that this is all part of a structured test plan. Whether the manual tester writes an official test plan or not, they are in fact going through a series of steps, covering specific areas and validating input/outputs. The same thing as an automated test would do.
Lets say we find some bugs while doing some manual testing (yes manual testing is still used). Say we discover that if we enter special characters for the name field, that they are accepted, when they shouldn't be. So we open a bug on it.
We'll, we also create an automation test to cover each bug found. That is then added to the BDD Feature.
What about cases where the page is dynamic, so inputting one value would
get multiple dynamic changes on the page? The tester, as Bach points
out is testing all that. So is the BDD test. You just create BDD's for
each change expected. It's basic input/output.
Unlike Bach's example where he had a solo BDD test (not considering the entire Feature), and then compared the manual testing of a fully flushed out test plan (which wasn't a fair comparison), I'm going to be much more fair here. The reality is that the BDD tests should match the test plan and test scenarios manual testers would be doing anyway. There should be little difference.
In other words: You still have to write a test plan (whether you manually test, use BDD automation, or use a non BDD automation strategy.) No matter what you need to structure your tests and report your results. Every tester should agree with that.
Once you agree with that, it comes down to apples and apples. We're now comparing the same thing. The same manual test plan vs. the same automation test plan.
James mistakenly thinks that BDD just focus' on some simple subset of testing. You should have a BDD test to cover each test case... each test case in your test plan, validating the same results you would validate in manual testing.
Efficiencies
James Bach in his responses to comments on his blog post, says "
James' Reply: The idea that it helps you release faster is a
fantasy based on the supposed value of regression check automation. If
you think regression checks will help you, you can of course put them in
place and it's not necessarily expensive to do that (although if you
automation through the GUI then you will discover that it IS expensive).
BDD, as such, is just not needed. But again, if it's cheap
to do, you might want to play with it. The problem I have is that I
don't think it IS cheap to do in many cases. I'm deeply familiar with
the problems of writing fixtures to attach high level "executable specs"
to the actual product. It can be a whole lot of plumbing to write. And
the temptation will be to write a lot less plumbing and to end up with
highly simplified checks. The simpler they are, the less value they have
over a human tester who can "just do it."
James is calling out efficiencies here.
But he betrays himself. He states "if you think regression checks will help you, you can of course put them in place." I've never heard of regression as a option. It's never an option. You MUST have regression. James knows that's the shining star of automation and is trying to downplay it to bolster his position. How could any QA team not do automation? Many issues, in my experience (and I dare say in general software development) are breaks to previously existing code, due to commits to new features. The new features get a commit to trunk and the commit unkowningly stifles an event that is needed for another feature - or changes a base class, or property file and something seemingly unrelated snaps. Regression is NECESSARY. It's never "well if you want it..."
As we read on we see he's claiming that it's more efficient to "just do it." That there's just too much overhead in writing all that plumbing. If you are testing a new feature, you don't have the luxury of knowing all the ins and outs of it. You have to start from scratch. In that moment you can also write code. That's right, WRITE THE CODE BEFORE YOU GET THE FEATURE. When you go to planning for a new feature, the developers will need time. As they spend time to write code, you will write out your code and test cases. BDD makes this easy - YOUR TEST CASES BECOME YOUR CODE. How cool is that?
Meaning you write out your test plan in BDD. At eHarmony we use GEB and the BDD is added in planning into each story. But this could be Cucumber. You plan it out, copy the BDD to a Cucumber feature file, for example, and then write the code to work against.
BUT, say some, There isn't any development yet, how could I possibly code for it? Good question. That's why you work with your developers hand in hand. You know the URL you're hitting. You know the basics, but you won't know the div's in the page. The Class's and ID's you're trying to select. You may not know the property file name, etc. But you work with the developer and agree on the naming conventions and write your test BEFORE you even get the code.
RED GREEN CLEAN
That's what red, green, clean means. You write the test before you get code - so it fails. You get the code (if it works) it goes green (if your test needs changes you make the changes and see it go green - or report the bugs to the developer) and finally you refactor (clean) the tests.
In this sense, you are literally building the "plumbing" as James Bach calls it, while you are writing the test plan. Is it really that hard? Cucumber makes it easy. As easy as writing text.
Of course if you can't write test plans, and you just "wing it" - then this will be very horrible for you. But then you aren't really testing with quality.
HOW BDD HELPS
So how does BDD help? As I went through James Bach's comments and posts - he suggests it doesn't help at all. I'll respond with my take on how it does help.
Test Features become Test Plan Repositories
As you build upon your test feature set in Cucumber, you'll have a growing repository of your entire test plan history. The feature files are easily human readable. No code. The step definitions that they reference/call is where the code goes. So Cucumber feature files contain no code. That means anyone in business can read the test and understand it.
Rapidly Reusable Tests
Tests can be quickly run and rerun. You don't need to call someone at 10pm to run through a test, or read some testers documentation on how to do something you've never done before.
Anyone can kick off the tests
Plug cucumber into Jenkins and you have a user interface to simply kick off a test repository against a test environment. No special QA person or Deploy person needed.
Everyone can see the failures and understand them
Builds Tests that Anyone Can Read
If you're writing a automation framework without BDD, you'll have tests that no one will want to look at, except coders. You'll have code, with little guidance of the test - save for the occasional comment. This happens when people take Java and Selenium and try and make a automation framework just from the two.
Tests are as readable as a test plan
Again, the tests are easy to read. Each Scenario ads to the overall feature. You can organize your features by sections, pages, functionality, etc. But the feature files are simple to read and understand what's going on.
Data Validation
In Cucumber and GEB, you can build data tables into the test, to rapidly run a test multiple iterations and pass through a variety of values and verify the result. This is faster then humanly doing this. I have tons of examples of this on github. You can have a table of 50 movie titles that you are passing into a service end point, and validating the JSON data returned for each title. Validating the MPAA RATING, jpg img path, etc. This test can be kicked off and finish in less then a min.
Failures are obvious
When tests are run from a tool like Jenkins, and a test fails, the failures are obvious. The failure state is captured as the BDD section the test stops at. Example, if a test fails at "Given a user is at http://www.google.com" then you know the failure is at that point. Further details will be in the error message (such as), "time out waiting for page to load." So you may have a network or proxy error. Even a non technical person can get an idea of what's going on .
HOW AUTOMATION HELPS
Rapid regression
Unlike James Bach's feeling that regression isn't a necessity, I feel it always is a necessity. Unless your business is making one off pages, I can't see how you would never reuse code! If you reuse any code, you MUST REGRESS.
At one job, we had two week core code deployments. This was a major deployment of a large web application/site. Both front end and back end code, as well as services might be deployed. Manually regression the trunk for each deployment would take 6-8 testers, something like 4-5 days to cover all regression in 5 browsers (FF, IE8, IE9, Chrome, Safari) + iPad Safari. Not to mention mobile application regression. That's a lot of testing.
We would build out a task list, assign people to the tasks, and then have a grid. Each column in the grid was a browser. Now figure each test you do, you have to redo 5 times. Why? Because were a customer focused company. People gave us money. If they use Mac Safari and have a issue, it's a problem for us. If they use IE8 and can't subscribe, we loose money. We must cover regression in all supported browsers.
If, instead, we cover regression with automation, the automation might take 6-8 hours to cover the entire site, but that pales in comparison to 4-5 days!
Understanding
When you manually test, you are most likely always in the "black box." That's not necessarily bad, but you don't have any idea how something is made, or how it is working. The more you know, the better edge cases you can find.
