Regarding the limitations of common reliability and safety testing standards, I came upon an interesting video from automaker Tesla recently:
In it, you can see that the Tesla teams have data about the real ‘failure modes’ (car crashes) and how they happen. Based on that, they have identified some configurations that are common but not covered by the legal safety standards.
So, they are using those data in the following ways:
- Modifying their product design (some code that has to trigger an action within 10 ms of the start of the shock, the shape and position of the airbags)
- Doing their own tests by replicating those common configurations
- Collecting a lot of data about injuries on the dummies etc.
- Confirming if the product design is adequate for the safety of passengers
The problem with reliability and safety testing standards…
It’s the same general idea as Kevin Howard, an experienced packaging engineer, wrote:
Many companies use standards such as ISTA (International Safe Transit Assoc), ASTM (American Society for Testing and Materials), JIS (Japanese Industrial Standards), and others. If one compares these standards, you’d find that no two are identical. This means it may be possible to pass one set of tests but not another, even though the products end up next to each other on the store shelf.
The problem with standards is that they know nothing about a specific product and its supply chain. Seeing that no two products and supply chains are identical, then why should different companies use the same test standard?
ISTA 1A is a test that verifies some minimum amount of protection, thus helping reduce business risk of catastrophic failures, but it does not purport itself to simulate actual hazards found within a specific supply chain. There have been no studies to indicate that a product would be dropped 10 times from a maximum height, so the drop test may well be excessive. At the same time, because the standard neglects certain orientations, it may cause a company to excessively package a product and yet still see damages because of a particularly weak orientation having not been tested.
One of the issues with test standards is that none of them call for margin testing, where one tests until failure. Instead, all of these standards are go/no-go tests, but what would you do if you found out you could pass not only the 30” drop, but 48” or 55”? Wouldn’t you want to know that and then save some money by reducing the amount of materials and the size of the box?
Both ISTA and ASTM have standards that allow the user to employ knowledge about their specific supply chain into a distribution simulation test. Very few companies have taken the time to survey their distribution system and see how things are handled and whether or not there are consistent failures occurring. This is akin to a product designer who decides not to plug in his new design and only guess that it works fine.
It is impossible to have good laboratory tests without witnessing the supply chain in a variety of places. Not only is it important to see if anything is failing, but also to know how well things survive.
A few years ago I surveyed the distribution system in India for Hewlett-Packard. Some HP products had terrible problems while others looked absolutely pristine all the way to the stores. The distribution system in India was the worst of any country I’ve ever seen, so it was great that certain products were able to handle that system so well.
Unfortunately, those products used the exact same packaging for shipments through Europe, the US and Japan, all of which had far more benign handling. Why would one use the same package for the worst environment and the best environment? Because they had only one test standard to pass in the lab, which was fairly abusive.
Seeing that sales in India represented about 1% of total sales, it didn’t seem to make much sense to employ the same abusive test standard for the 80% + of units traveling in developed nations.
I have had many clients who’ve complained that they were able to pass certain ISTA tests and yet sustained high damages in the field, while I also have personal experience with products that could not pass ISTA tests and yet had no damages in the field.
So, here’s the secret: testing is vital to limit business risk, minimize damages, and provide a good experience to the consumer, but to really minimize costs for materials, damages, shipping and handling, testing must incorporate far more than the basic tests prescribed in ISTA 1A.
He wrote the same thing in a different way here:
Of course, most of these companies, along with packaging suppliers, simply can’t give better advice since they don’t fully understand the distribution pipeline the products are travelling through, so they take a cookbook approach, simply using standards from ASTM, ISTA, JIS, FedEx and others, but those standards all miss some very basic issues, especially in regards to free-fall drop tests and whether or not the test results replicate actual issues found in the field on a consistent basis (and never causes issues not found in real distribution).
Agilian’s head of New Product Development, Andrew Amirnovin, echoes this
He said:
14 years ago when I created the AT&T reliability standard for testing mobile phones, I felt exactly like Kevin in this article. I was thinking “why do I have to use a standard that is meant for a tank or a space shuttle for a mobile phone?”
It was really hard to push back to the old fashioned reliability directors in Blackberry and Palm and alike that “no, I won’t need to use any of the military standard or any other existing standard. I have a customized reliability test for mobile phones.”I agree with everything Kevin says in this article and that’s the way I have been driving reliability, yet still, I get strange questions like: “what standard did you use to test..?”
One thing I’ve learnt during my career is that it takes time to change the minds of people who are set in their ways!
“Can Agilian help me to devise a customized reliability test plan?”
Yes, we can. We have an in-house testing lab and reliability testing and product development experts on staff who examine your product and suggest and devise the right plan for your needs, not simply a standard test that may not cover the bases you need it to.
It often seems natural to test to some of the most common reliability and safety testing standards, but actually the most important is to follow a testing plan that replicates the real stresses the product is expected to experience in the field when used by consumers.
Andrew makes this point really well in this episode of Sofeast’s podcast: Why Product Reliability Testing Is A MUST During Product Design [Podcast]
