Penetration Testing
3/10/2025
Responses by Felicia King – CTO/CISO QPC Security https://qpcsecurity.com
I recently recorded a podcast for some lawyers who requested more information on penetration testing so that they could properly advise their clientele.
https://qpcsecurity.podbean.com/e/the-real-skinny-on-penetration-testing-debunking-the-myths/
Featured speaker on Managed Services Journal regarding this topic, see their post here!
What are the key differences between penetration testing and vulnerability scanning?
Vulnerability scanning can be agent-based or credentialed which provides a great deal more visibility into the status of systems. Vulnerability scans provide directly actionable data which can be prioritized using industry standards such as CVE severity, age of vulnerability, or EPSS.
Penetration tests can be used to identify weaknesses in an organization that vulnerability scanning would not reveal.
While both are useful in assessing an environment, vulnerability scanning provides a great deal more visibility into the status of systems. On the other hand, penetration tests can be used to identify weaknesses in an organization that vulnerability scanning would not reveal. The terms are not interchangeable, and one does not replace the other.
How do one or both tools ensure comprehensive coverage of potential vulnerabilities?
Vulnerability management overall is a progressive scale of operational maturity. Organizations should always seek to fully utilize the tools they already have and master them before acquiring more tools. An organization that is doing a poor job of patch management should not acquire separate vulnerability scanning or penetration testing tools or services as the fundamentals are not being properly handled.
How do I prepare my environment to use these tools effectively?
The very first thing an organization must do in order to have a hope of being successful at vulnerability management is to have a comprehensive and accurate inventory. Organizations must employ a high-quality asset inventory system with multi-point visibility. One cannot manage assets which are not tracked in an accurate inventory.
Once patch management is systematically proven to be competently handled every month, an organization can mature to a vulnerability scanning tool. That tool can scan for things which the existing patch management platforms may not reveal. Be very cautious and avoid use of an inaccurate tool. There are only two tools on the market that I would deem to provide accurate enough data to be usable. Neither are inexpensive. Both are designed for the enterprise market.
Overall, competent vulnerability management is very expensive and time consuming. An organization that is unwilling to establish a written vulnerability management policy and allocate adequate resources to that endeavor will be unsuccessful.
How can I use these tools as an executive / business decision maker?
A vulnerability assessment platform can centralize a layer of visibility while providing attestation reporting and prioritization of vulnerabilities in reports that executives, compliance personnel, and insurance company personnel can easily interpret. It is essential that company leadership has governance and oversight to vulnerability management instead of simply believing that the IT team is handling it. Remember that it is the executive’s signature on contracts and insurance policies. Executives should not simply trust IT personnel, but should verify by reviewing attestation reports themselves.
What does penetration testing look like, and when is it necessary?
Penetration testing generally comes in two flavors.
- Regular scheduled automated testing
- Human driven testing which can include physical access and social engineering
It is rare for an organization outside of highly regulated entities such as banks to have achieved a level of operational maturity in their vulnerability management such that they would receive value from a human-driven penetration test. Most organizations should focus on becoming proficient at patch management and secure configuration management. There are economical tools and methods for both which provide a high ROSI (return on security investment) compared to penetration testing with is often incorrectly scoped and lacking actionable intelligence as an outcome.
Penetration tests are a good choice when they are required for a business certification such as mandated by insurance or customers. On the customer side, I would advise B2B customers that penetration tests are very easy to manipulate to manufacture outcomes that are more favorable than reality. As a result, I put very little value in them.
In order for a penetration test to provide comprehensive view of the vulnerability of an organization, it would likely cost $100,000 and need to be conducted by a company such as Black Hills Information Security which would not allow for the pen test to be incorrectly scoped.
What enables these tools to scale to accommodate large and complex IT environments?
Highly competent management will make or break vulnerability management; tools will not. All the systems scale without issue, limited only by the competency of the humans which manage the systems.
Any agent-based patch management, vulnerability assessment, or penetration testing system only works well when the agents are deployed and configured correctly, or the system is provided visibility into things which cannot have an agent installed. IT personnel often fail to deploy agents comprehensively or to configure visibility into all assets, leading to inaccurate results and unpatched vulnerabilities. Very few organizations have comprehensive and diligently maintained asset inventory. Mis-scoping is the norm rather than the exception.
Scale, relevance, and value should all be understood more deeply.
A comprehensive penetration test for an 8-person company can cost $30,000. An 8-person company is not likely to see ROSI for that expense. Additional budget would need to be allocated to remediation. The company would be better off to apply pen testing budget to operational maturity improvements and secure configuration management executed by highly competent IT personnel. The exception would be if the penetration test was required to obtain revenue whose predicted profit margin exceeded $100,000.
The annual budget for secure configuration and vulnerability management for a 100-person company with around 200 assets costs about $75,000 annually.
What types of vulnerabilities do these tools specifically target?
It is important to know that “vulnerabilities” are generally defined as missing patches or known, documented vulnerabilities for which a CVE number has been assigned. There are many unknown and unassigned vulnerabilities which exist. Only secure configuration management can prevent unknown vulnerabilities from being exploited. While there are some secure configuration management tools that exist, they are no substitute for highly competent IT management personnel.
The easiest vulnerabilities to identify are the current state of systems in contrast to the state recommended by the system manufacturer. This could be bios, drivers, firmware, software, operating system, applications, and more. A good company vulnerability management policy would include installation of all OEM recommended patches within 30 days of release.
Penetration testing may reveal insights into security configuration gaps which are known venues of exploit. Penetration testing can reveal items that a vulnerability assessment tool will not, if the penetration test is scoped properly, which is fairly rare since it is very cost prohibitive to do so.
How do these tools integrate with existing security infrastructure and systems?
Modern zero trust endpoint protection software includes patch management. Compliance agents exist which can provide another point of view and visibility. Dedicated vulnerability assessment agents will be separate.
It is important to have multiple systems which can provide a point of view for IT security management personnel where they can validate the status of managed assets from different perspectives. No one tool is known to be comprehensive in all areas.
It can be desirable to integrate data from a vulnerability assessment platform into something like a GRC platform as part of automated attestation, but that is not required. The minimum which is required to provide attestation is the ability of the source system to generate reports which prove the state of systems at a point in time and deliver those reports into the attestation platform which may or may not be a GRC.
What kind of reporting and alerting capabilities should these tools offer?
Summary views as well as full detail exports are mandatory. The summary view is intended for oversight personnel such as executives and compliance managers to quickly assess how compliant assets are with policy. The summary reports are insufficient for legal attestation proof, which must be full detail on a per-asset basis. Both types of reports should automatically publish to the attestation platform or GRC without the requirement for human upload which would deem the data as possibly being tampered with prior to upload. Few systems achieve this level of quality.
How do these tools handle continuous monitoring and real-time threat detection?
Continuous is a misnomer because a network or public-facing resource (website) which is continually scanned may be overwhelmed. The same could be said for an endpoint that has a continuous scan going on. “Continuous” would likely be seen as a denial of service.
Instead, vulnerabilities on high exposure systems such as end user computing devices should be scanned at least once every 24 hours. Networks should be scanned at the time of day when it is most likely to find issues, which inherently means avoiding scanning at midnight when many systems may be off or simply not on the network anymore.
Secure configuration management should be done daily and continuously through human discipline coupled with policy-based enforcement technology.
Penetration testing should be done on the frequency which aligns with the organization's policy. If the organization has no policy on penetration testing, then there is no governance structure to ensure that testing will be comprehensive enough, comply with the company’s contractual obligations, and results used to close gaps.
What support and training should VARs/MSPs receive to ensure effective use of the tools?
I am not aware of training classes on how to competently manage vulnerabilities or engage in secure configuration management. Training is available on how to use a product, but that assumes that the person receiving the training is being provided a set of policies and governance structure to guide them. In my 30 years of experience, only a highly technical CTO with at least 15 years of experience is likely to provide an organization with the kind of leadership required to develop the governance structure. I have never seen a person with the title of IT Director achieve those goals.
QPC published two podcasts on vulnerability management which are highly educational.
https://qpcsecurity.podbean.com/e/vulnerability-management-part-1/
https://qpcsecurity.podbean.com/e/vulnerability-management-with-felicia-and-dan-part-2/
Can you provide an example to illustrate how one of these tools helped a client avoid a potential security breach?
The majority of breaches occur through email/website browsing vector. Many involve browser software configuration or unpatched vulnerabilities. Some involve vulnerabilities in secure configuration management of cloud assets such as a Microsoft 365 tenant.
Use of a very operationally mature Microsoft 365 tenant configuration management tool can provide visibility into configuration weaknesses. Unfortunately, these tools are rare and expensive, generally being too cost prohibitive to be employed in the MSP industry. Some highly skilled MSPs have developed their own automated systems.
Endpoint security configuration of browsers, ensuring that browsers are all fully patched, and that only authorized devices are accessing company resources are three essential functions to prevent breaches. Browser patching should be handled by a patch management system. Browser security configuration should be handled by a secure configuration management tool. Ensuring that only authorized devices access company resources can only be achieved through custom programming of hardened technical access controls, the success of which is highly dependent on the skill of the IT management personnel.
What tips can you share regarding how VARs/MSPs can monetize these tools with their clients (please share any “Do’s” and “Don’ts”)
Do require clients to have written policies for vulnerability management, secure configuration management, and penetration testing. If they fail to have their own policies, then you can tell them what your policy is on the matter, but they must acknowledge that you cannot be held responsible for their compliance and risk management when they fail to define requirements in writing in a codified, approved policy.
Do require clients to obtain your written authorization for any penetration test performed by a party other than your company. Some clients will use this approach to “test” a MSP, but in reality, they will cause costs to be increased. An unauthorized penetration test can cause a flurry of billable activity to the client by the MSP or security personnel who are responsible for monitoring the security events in the environment.
Do require the client to provide the MSP the results of any penetration test or external assessment. I have seen the results frequently and intentionally misrepresented by the third party and used as a deception to lead the client to believe that the MSP is incompetent. The client may have declined services that the MSP offered while some personnel at the client think the MSP is providing those services. Unless the MSP is provided the direct results of testing with an opportunity to challenge the results, the process is likely a deception or manipulation.
Do use a secure configuration management tool like Senteon and produce the compliance reports. Include the pricing for secure configuration management in your standard endpoint protection service.
Do ensure that you have a system for compliance attestation regardless of whether the client has a GRC or not. The MSP should be able to prove the results of configuration status when in scope of services they provide.
https://www.qpcsecurity.com/2024/08/14/how-msps-can-legally-protect-themselves-when-their-clients-do-not-want-responsibility/
Do ensure that your service catalog and statements of work on these services are detailed and thorough.
Do encourage all clients to use an asset management platform and not rely on the MSP to provide that for them.
Do sell secure configuration management and help clients understand why it is mandatory. https://www.qpcsecurity.com/2023/02/16/addressing-information-security-fundamentals-with-cis-and-community-defense-model/
Do require the client to have all devices which access company resources covered under management. Without a full, comprehensive inventory and management plan, risk cannot be effectively managed.
Do not lead clients to believe that simply monitoring Windows Update status is sufficient. I have seen many MSPs do this and provide the fictional perception to the client that patch management is occurring. MSPs charging $50/mo/server to “patch” a server are likely engaging in fraud. When I have inspected their SOW language, I see that they are only monitoring Windows Update, which is grossly insufficient.
Do not configure exceptions for penetration testing. Penetration testing companies often want their systems to be whitelisted so that the security protections in place will not block the scans or hack attempts. This deception results in false findings.