Do you know this situation, when you implemented some automated tests based on a requirement, but a test has been failing for ages because a bug in the implementation was never fixed?
Yeah, that tends to happen. When you create an automated test, you read the requirements and the acceptance criteria. You discuss technical solutions. You, together with the product owner or business analyst and the team establish what is the expected behavior of the feature you will build. So, when you write your automated tests, you know exactly what should happen when using the feature and that any deviation from the requirements represents a bug. The product owners will surely agree, based on the acceptance criteria that they wrote, that if such a deviation occurs, it is clearly a bug that must be recorded as such.
But sometimes that bug is so non-important, that although your tests are failing due to it occurring in the system, it never gets fixed. When would this happen? Well, if the customers will never find it, in the best case scenario. When that abnormal behavior is only exhibited when trying the edge-case of an edge-case of an edge-case, that the users would never try in reality when using the product you are building. In this case, the fact that the test failed is good, since it clearly signals that an acceptance criteria was not met. But since the customer will never get into this situation, you could agree with your product owner that the requirement needs to be amended to reflect this behavior.
Another situation when the bug is not seen as important is when there is only little impact on the customer’s experience. Usually, if there is only a slight annoyance on the customer’s part when the bug occurs, or if it comes up rarely when the customer uses the feature, product owners tend to dismiss the bug as not worth fixing right away. But then, it never ever gets fixed. Simply because each sprint there is something more important that needs to be worked on. And sometimes the bug report falls into oblivion, as nobody will remember when it was raised and where to find it. The test that caught the bug will still fail every time you will run it, but somehow in the back of your mind you will start to ignore it. And you will get used to having a test suite that is always red, even if green is such a prettier color. Also, customers are annoyed once, they are annoyed twice, but by the time they get annoyed too many times, they will simply switch to another provider of the same service you are offering, or use your product less than they would normally do.
A bug is still a bug. Even if it is low priority. If the customer encounters it often, then it might lead to loss of reputation, so make sure you bring this up with your product owners. Ideally it should be fixed it in less than a month, if it causes a lot of annoyance for customers, or else you will get quite a lot of calls to your helpdesk. What helps is having the product owners use the product as the customer would, for a long period of time, just for them to understand and feel what the customers feel. In this case, do not delete the corresponding test. It will become green once the fix is in place.
If the bug is there, but the customers only see it rarely, there is no need to rush the fix. Just bring the topic up from time to time, so that people don’t forget about it. Also, don’t let years pass by without it getting fixed.
If the bug is never encountered by the customers, the bug can become a feature. It will probably never get fixed anyway, unless, if encountered, it would cause major failures inside the system. It the bug becomes a feature, you can keep the old test but just change the checks so that they accept the behavior as expected. Just make sure that the behavior is clearly specified as being accepted in the official requirements and documentation.