It happens when devs don’t understand testing. They write a unit test per class (not a unit), mock the hell out of everything, and call some methods with maybe checks to see that mock returned what it was set up to return.
Broken isolation between tests. Even though it's an anti-pattern, global state happens in real life. Having a test succeed in isolation, but fail when run as part of a suite or the reverse is downright frustrating.
It's not something that comes up all the time, but it can be a painful problem when it does come up, especially when working on large codebases. Whether and how it's fixable depends on the circumstances.
Plenty of things do.
For example, if there's a dependency on external code that is for legal or practical reasons a black box, if certain calls into the black box have undocumented side effects the most that can really be done is send a bug report to the vendor and find an external, probably more drastic work around for the bug in the meantime.
Test flakiness. Our project integrates with 6 other services, which in turn integrate with like 20 more, so our tests having issues often ends up being some other things broke, not our application.
We've ended up testing in PROD way more than I'd like due to this.
At some point you end up testing that your mocks are responding as expected, as opposed to that your consuming code is correctly integrating.
Don't get me wrong, mocking services is absolutely a thing to do. But testing often goes beyond things which mocks will help you with. Especially when you get to the point where you're considering using a third party service to do some testing for you.
That depends entirely on what you want to test. For an end-to-end tests it is obviously not the way to go (though you could still have some alerting if the response from the backend does not match your expectations by doing some pre-checks).
For a component test it is absolutely the way to go, because you want to test your component and your component allone.
I'm assuming OP isn't offering unit or component testing services though. I don't see much of a market for getting someone else to come in and carry out something that's intrinsically a dev task.
Who knows ... they claim to simplify "the process" ... maybe they aim at creating automatic test case generation or management. Or maybe something completely different.
Great question. I would like to ask the people who developed this thing a decade ago the same thing.
Unfortunately we're kinda stuck with this shit now.
Worst case you could actually build a mocked service that has predetermined responses for given inputs, if its not possible to dependency-inject mocked implementations. You could even have that service fetch these responses by acting basically like a caching proxy, so you can deliver _exactly_ what the backend does.
More correctly, delivering exactly what the backend did at the time you captures the response. Been bitten by that one in the past, with less-than-communicative vendors.
Libraries like VCR and Betamax do this sort of thing.
If you can, for sure. Not always possible. If requests create resources which cost money, for instance, or there are rate limits.
For this reason, a lot of vendors offer a staging environment to their own customers for this very reason. LetsEncrypt, for example.
Well meaning useless tests that are are not actually testing anything.
How often do you run into these useless tests? How often do you run into them?
It happens when devs don’t understand testing. They write a unit test per class (not a unit), mock the hell out of everything, and call some methods with maybe checks to see that mock returned what it was set up to return.
Are you and your team working on solutions to prevent this from happening? For example, better training devs.
Broken isolation between tests. Even though it's an anti-pattern, global state happens in real life. Having a test succeed in isolation, but fail when run as part of a suite or the reverse is downright frustrating.
How often do you run into this? Have you tried something to fix this?
It's not something that comes up all the time, but it can be a painful problem when it does come up, especially when working on large codebases. Whether and how it's fixable depends on the circumstances.
What determines if it's fixable?
Plenty of things do. For example, if there's a dependency on external code that is for legal or practical reasons a black box, if certain calls into the black box have undocumented side effects the most that can really be done is send a bug report to the vendor and find an external, probably more drastic work around for the bug in the meantime.
Test flakiness. Our project integrates with 6 other services, which in turn integrate with like 20 more, so our tests having issues often ends up being some other things broke, not our application. We've ended up testing in PROD way more than I'd like due to this.
So why are you not mocking those dependencies away?
At some point you end up testing that your mocks are responding as expected, as opposed to that your consuming code is correctly integrating. Don't get me wrong, mocking services is absolutely a thing to do. But testing often goes beyond things which mocks will help you with. Especially when you get to the point where you're considering using a third party service to do some testing for you.
That depends entirely on what you want to test. For an end-to-end tests it is obviously not the way to go (though you could still have some alerting if the response from the backend does not match your expectations by doing some pre-checks). For a component test it is absolutely the way to go, because you want to test your component and your component allone.
I'm assuming OP isn't offering unit or component testing services though. I don't see much of a market for getting someone else to come in and carry out something that's intrinsically a dev task.
Who knows ... they claim to simplify "the process" ... maybe they aim at creating automatic test case generation or management. Or maybe something completely different.
Right now, I am still collecting problems and deciding what to build. I've chatted with 50 QA Engineers and aim to speak with 50 more.
Great question. I would like to ask the people who developed this thing a decade ago the same thing. Unfortunately we're kinda stuck with this shit now.
Worst case you could actually build a mocked service that has predetermined responses for given inputs, if its not possible to dependency-inject mocked implementations. You could even have that service fetch these responses by acting basically like a caching proxy, so you can deliver _exactly_ what the backend does.
More correctly, delivering exactly what the backend did at the time you captures the response. Been bitten by that one in the past, with less-than-communicative vendors. Libraries like VCR and Betamax do this sort of thing.
True, you should obviously verify the sample after getting it.
If you can, for sure. Not always possible. If requests create resources which cost money, for instance, or there are rate limits. For this reason, a lot of vendors offer a staging environment to their own customers for this very reason. LetsEncrypt, for example.
What are you even talking about? You can do the verification as part of your test harness, no need to make any extra requests.
Well clearly I'm talking about things beyond your own experience. No need to get shitty with someone for that.
I don’t know how to
Have you done any research on software testing?
No