Security penetration testing, or pen testing, can be a useful but often misunderstood security test method, particularly by companies who are new to security, so what is a pen test, is it suitable for IoT/embedded products, and when should it be used?
tl;dr
Key points:
- Penetration testing is probably not the security test you are looking for if you are developing a product. Unless you are already confident that you are secure and want 3rd party validation.
- If you are using a pen test as the only security check after product development, be prepared to spend a lot of money and delay your product release.
- You need to think about security as soon as you start developing your product.
What is a pen test?
A pen test is a quick, low-cost way of confirming what you should already know about your technical security risk.
The term “pen test” is recognised by many people as a type of security testing, but it is often mis-sold to companies who are developing IoT, embedded systems, and embedded software as a way to secure their products after they have been developed. Pen testing may be good for testing IT and web application cybersecurity (and are often necessary to complete security audits of these), but it is a bad fit for trying to secure products.
A penetration test is a way to gain confidence, or “assurance” in security jargon, that you have developed your product securely and that it is safe from a malicious attacker using commonly-known attack methods.
Pen testing is commonly available for IoT and embedded system products, but it should only ever be used as a quick, independent, 3rd party check on security. It can also be useful for checking the security of a cloud-based interface used in an IoT system (if the cloud provider allows it). Even then, an independent vulnerability assessment of the whole system will probably be of more value.
A reputable pen test company will:
- Gather information about your product. You can choose how much information to supply, from all the internal technical details (known as a “white box” test), a limited set of information such as only that which will become publicly accessible (“grey box” testing), or no information (“black box” testing). In the case of a grey or black box test they will try to extract as much information from the product as possible by reverse engineering. Grey box testing is probably the most realistic scenario for pen testing.
- Analyse the product for the most likely weaknesses. They will then try to determine what they think will be the most successful way to attack it based on the information gathered in the previous stage. This is called “threat modelling”.
- Perform practical tests. They will use attack tools against the potential security problem areas identified in the previous stage. These are commonly automated tools, but if a higher level of security assurance is required then they will often use tools that they have developed themselves or use more manual, highly specific methods.
- Provide a clear report. This report should identify what they tested, why they tested it and the results of their testing. The best companies will also provide some suggestions on how to fix the problems that they find.
What a product penetration test will NOT do
A pen test of a product will NOT allow you to accurately determine your product security risk - it provides poor security risk coverage.
If you get three different pen test companies to test your product, you may well end up with three different sets of results as a pen test will target the most likely vulnerability areas determined by a tester based on their experience. A pen test is normally performed on a fixed-time basis to keep the costs constrained, so if these avenues of attack are not successful then you will gain knowledge on which parts of your product are less vulnerable, but there may still be other parts that are.
A penetration test, like any one-off test, is also only valid at the time at which it is performed. Attack methods and tools evolve and improve over time, so the results of a penetration test can quickly become obsolete.
When to pen test
Perform a pen test if you have a completed product that you already think is secure, and you want a 3rd party check. It is a way of checking that you have done a good job of securing your product.
Do NOT perform a pen test:
- To determine your product’s security level or risk. You will end up going through expensive iterations of pen tests and still not have a clear idea of your risk.
- If your product is not complete. A pen test should only be performed on the version of your product that you will be releasing. As soon as your product changes, you may introduce new security problems or have regressions in previously secure areas. This invalidates any previous testing.
- If you do not already understand the security of your product. A penetration test is only for assurance and if you have developed your product securely then it should only rarely find a problem. If a penetration test results in lots of identified security issues, then you need to take a step back and secure your product.
- If your product is a not a stand-alone component. If your product is a hardware or software component that will be integrated into a larger system, then it must be self-contained and have well defined interfaces before it can be tested. Even then, it is best to pen test complete products as security issues often arise from the integration of components.
As part of your product development, you should:
- Determine the level of security you need. This can be driven by regulations or market requirements.
- Design your product for security right from the earliest stages. You need to build security into even the initial architecture. Trying to add security at the end of a product’s development will be a very costly mistake.
- Develop security requirements. In the same way that you have functional requirements, you should also have security requirements. This provides a list of things that can be tested against.
- Analyse your product for potential security problems. You need to determine if your product has potential security problems, called “vulnerabilities”, at each design iteration.
- Evaluate your security risks. Each vulnerability brings a level of security risk. You need to determine those with the highest risk.
- Implement risk reduction measures. You need to find ways of reducing the highest security risks during development, while still keeping within time and resource budgets.
- Perform internal testing. Tests should be made against the security requirements.
This process is part of a “secure development lifecycle” and external security companies can help to guide you through this process if you have not done it before.
The part that deals with analysing a design for security problems and risk and coming up with ways to solve those problems, the security “controls”, is called a security vulnerability assessment. This provides a much more complete view of the security risk of your product and allows for proper management decisions to be made on how much risk your company is prepared to accept.
Once you are confident in your product security, THEN maybe consider penetration testing.
Selecting a security partner
Some tips:
- Avoid companies that only perform automated testing. You should be already doing this yourself and it will not add any value.
- Choose a company with the appropriate technical skills and market knowledge. For IoT and general embedded system testing, make sure that the security company you select has the necessary experience of the type of product you are selling. There are plenty of security companies who only understand IT security and do not have the necessary technical skills to probe things like debug ports, board-level busses or understand advanced physical attacks such as fault injection or side-channel attacks.
- Avoid security companies that recommend pen testing as a primary test method for IoT/embedded products. For these products, a secure development lifecycle approach is required. Not just a quick test at the end.
- Work with a security company that can pass on their knowledge. The best security companies work as partners. They will help you to build security practices and processes into your product development and teach your organisation to be self sufficient.
- Beware of publicity-seeking or scare-mongering security companies. There are many companies that promote their latest find of an insecure product, and some unscrupulous companies even find vulnerabilities and then threaten to expose them if they are not paid for the testing. Unfortunately, it is not that difficult to find insecure products. That is not to say that exposure of insecure products is not a good thing, and is common in academia, just be wary of a commercial company’s motivations for doing so and try to avoid “security circus”.
How can we help?
At Cerberus, we will NEVER recommend a pen test for a new product unless your team has previous security experience and you are confident in your security. We generally recommend a more in-depth security vulnerability assessment and will also partner with you to introduce the necessary security knowledge, culture, skills and processes into your organisation to create a sustainable secure development lifecycle.
If you want to find out how we can support you, then please contact us through the web contact form or email us.