SPONSOR 0:02
RSA offers business-driven security solutions that provide organizations with a unified approach to managing digital risk and designed to effectively detect and respond to advanced attacks, manage user access control, and reduce business risk, fraud and cybercrime. For a limited time, get mobile multi-factor authentication from RSA secure ID access for free to help secure your remote or dynamic workforce long term leverage push notification biometric and one-time password authenticators to secure access to your cloud applications on-premises systems, legacy systems privileged accounts, and more at no cost for six months. For more information visit securityweekly.com/RSAsecurity
Did you know that over 63% of cyber breaches are caused by a third-party vendor? CyberGRX is on a mission to modernize third-party cyber risk management. Built on the market’s first third-party cyber risk exchange, CyberGRX armed you with fast and accurate data of proven and innovative approach so you can make rapid informed decisions and confidently engage with partners. Learn how partnering with CyberGRX will help you become cyber cert. Visit securityweekly.com/CyberGRX
Cyber risk and compliance automation is finally here. Legacy GRC systems cannot simplify the complex use cases and deliver powerful automation that cyber teams need. CyberSaint’s integrated risk management solution ingest data from your existing tech stack, dynamically lighting up controls using patented AI. Leverage your expertise and showcase business value. Let your risk and compliance solution work for you. See why the most forward-thinking CEOs of the Fortune 500 support their teams with CyberSaint. Maximize your cybersecurity program today visit https://securityweekly.com/cybersaintscw
Jeff Man 1:49
Welcome back to security and compliance weekly. Hey, Do you always end up missing our live streams? Do you need somewhere to flag security weekly podcasts that you want to listen to? You can do this by subscribing on your favorite podcast catcher. We’re on our YouTube channel. Also, you can sign up for our mailing list. And finally, you can join our Discord server to stay in the loop and all things security weekly. Where do you start by doing all this? Go to securityweekly.com/subscribe. Also, if you have a specific guest or topic that you want us to cover on one of the shows, or if you want to just worm your way onto the show like Dmitry did today, you can suggest guests for visiting by going to securityweekly.com/guests and fill out the form. Find us on the discord server to suggest topics. We review all the suggestions and we’ll get back to you once we’ve reviewed your request. Alright, let’s get back to this discussion. You guys were setting up at the end of the first segment where I kind of wanted to shift gears and go and that is generically at a high level. What is the place? What is the purpose? What is the role of a penetration test in this whole grand scheme of things that we today call cybersecurity? More specifically, you know, if you’re a company that wants to have a pen test performed, and in to touch on what you said a little bit, Liam, I think this is where it gets very muddied and cloudy because the vendors have so much leeway to say what they want to say about pen testing or things that they do that participate or are part of a pen testing process.
So let me take a real quick minute to just lay out some some some of my thoughts based on PCI actually. And first, I’ll get the PCI counter up. But also I just want to set the stage for our discussion in the second segment, which is going to be hopefully focused more on what’s the point of a pen test? What are you trying to get out of a pen test? what’s the purpose or the goal of a pen test? And then and how, you know, if you’ve determined the goal, how do you most effectively produce a result so that you can achieve your objective. So food for thought. And this is maybe where some of my this is where some of my feelings come from about pentesting because I’ve been exposed to PCI for so long. PCI at a high level was 12 major security requirements revolving around six goals and six goals of security is secure your stuff. It’s a family show, sort of secure your stuff, secure your data, keep everything current, control access, do the security stuff, have it all written down. Those are the six goals of PCI at a very high level.
Things that are often associated with pen tests, like vulnerability scanning Getting, which is where I, you know, when I was starting pentesting, there were no courses, there were no sans-50 million – 500 level courses that you could take to learn how to be a pentester, we were kind of winging it back in the day. And we came up with a methodology that it’s the way you do a pen test. And it’s roughly the same as what’s being taught today with one exception. We never put vulnerability scans as part of the pen testing methodology. We simply would do, and I apologize for not knowing the modern terminology, we would do network probes, I know TCP IP port scanning, just to see what was out there. And what were they you know, what were they talking what ports were they talk talking, we…
Scott Lyons 5:47
Asset discoveries.
Jeff Man 5:48
…weren’t vulnerabilities. Thank you.
Josh Marpet 5:50
probing…all the probing…
Jeff Man 5:53
PCI, the – they have a section that’s called vulnerability management. Vulnerability management covers two requirements in PCI and that’s requirement five and requirements six. Requirement five is antivirus anti-malware keeping, you know, having some sort of technologies in place that are kept current, and they’re doing periodic scanning, and so on, so forth. That’s requirement five, that’s under the umbrella of vulnerability management. Six is where it gets a little bit more interesting. It’s have a program where you determine risk ranking based on you know, whatever your environment is, but have a way of classifying vulnerabilities in terms of risk, have a regular patch management program, have to change control procedures, do secure application development, do it based on things like OWaSP, make sure your developers are trained in writing secure code, that’s the bulk of requirement five and requirement six, which is what PCI calls vulnerability management. Vulnerability scanning comes much later in requirement 11, where you’ve secured everything you’ve secured the data, you’re keeping it current, you’re patching, you’re monitoring with, with any virus, anti-malware, you’re patching on a regular basis, you’re making changes with change control procedures, and it’s authorized, then you do access control, both physically and logically. And then you get around to the security stuff with like vulnerability scanning,
If you follow the logic of it, and I’m not saying it’s by design, but I think it is, you’re doing a whole lot of activities, you have a whole lot of procedures in place before you ever get around to doing vulnerability scanning. Vulnerability scanning in that context, then in my opinion, is meant to be more of a safety net, to find the things that fall through the cracks. Because nobody is perfect at anything, you know, there’s always going to be stuff missed or overlooked for whatever reason. And then the ultimate live-fire test after doing a vulnerability scan on a regular basis is to do a penetration test. So in that way of thinking, a penetration test is somewhat consistent, I hope with the way I’ve defined it initially, that it’s a test of your defenses.
It’s how well have you done setting up your security operations and all the different things that are you’re doing? How well are you withstanding attack? And we’re and or how well are you equipped to detect that somebody is attacking you? To me, that’s the essence of the penetration test, perhaps with a little bit of a PCI flavor. But this brings to light what’s my biggest frustration in our industry, which Liam kind of touched on, you’ve got sort of the vendor side that in a large way dictates “This is when you should do this, this is when you should do that”. As a perfect example. You know, we’ve had this big SolarWinds thing happened in the last couple of days, I got my email notices from Tenable last night, hey, we have a new plug in to help you detect, you know, the presence of whatever vulnerable and I don’t know the details of it yet. I apologize. But whatever’s vulnerable in terms of Solar Winds will help you find out if you have it. And to me that’s ass backwards, because vulnerability scanning technologies are meant to be a safety net after you’ve done a whole lot of other things not well, let’s try to decide what’s wrong with our network. Let’s run a vulnerability scan or by extension. And this is honestly how I started in pentesting. Back in the early days was, well let’s figure out what’s wrong with our network. Let’s do a pen test. Let’s figure out what all the vulnerabilities and the problems and the weaknesses are. And it’s that part where I acknowledge that’s how we started when I was helping to build methodologies and doing this stuff.
Almost 30 years ago, my god liberal 27-28 years ago, I expected that to evolve to something more like what PCI implies, which is the vulnerability scanning and the vuln and the penetration testing, however, we define it are much more safety net, how are we doing type of exercises? Not – Well, let’s start everything by doing this kind of stuff because it is in, in essence, a form of automation. And it’s in, it’s a way that we don’t have to think and we don’t have to be in control or have our arms around what’s going on in our network. And every time I hear a vendor saying, well, asset discovery, run our tool, like, why not asset discovery, you have an inventory somewhere, and you know what everything on your network because you have some sort of process in place that controls what gets put on your network?
I’ll shut up. That’s the setting the stage, what’s the goal? What’s the purpose of the pen test? And how does that influence how we define it? And more importantly, when? When is it appropriate to do this thing that we call a pen test? And what do we get out of it, gentlemen?
Scott Lyons 11:04
So wondering, I’d be able to define what a pen test is, is what we tried to do in the first segment, and we went all over the place, right? But I think the one thing that we can all agree on, is that pentest has grown to be an amalgamation of multiple activities that result in trying to illuminate issues, problems, or areas of potential compromise on any given system. Right?
Dmitry Zagadsky 11:30
Yeah. Like if you’re going through, if you’re going through, and you’re starting with buying a pen test to find out where all the vulnerabilities and weaknesses are in my network, I think you failed to start with, that’d be my one inflammatory comment today, I guess. But you really should already know where those things might be, by having a good understanding of all your assets, having a good understanding of where any technical vulnerabilities are. Well, I think, a good use of a pen test would be to help you prioritize remediation, if you’ve got a lot – to say, okay, well, these are the ones that really are, you know, exploitable by in the wild or by real person. So we should focus on those first, given limited resources. I think that if you don’t even know, where you might have problems, you really need again, to get a handle on that first.
Josh Marpet 12:19
But I think your..
Scott Lyons 12:20
As a consultant, listen, listen.. Josh hold on..
Josh Marpet 12:24
The point where you said, you need an asset discovery, you need an asset management, you need an asset inventory, you want to be inflammatory, tell people, they actually have to know what’s on their network, because yeah, none of them bloody well do.
Scott Lyons 12:35
Yes, that’s what I was about to say, as consultants, we step into companies all the time. And they’re like, well, we only have 50 machines. And we say, Oh, really? Right. What about your servers? What about your routers? What about your access points? What about everything that requires an IP address, and it comes out being, you know, 700 to 1000 by the time we’re done with it, and it’s just like, come on, man. You don’t even know what your engineering is, let alone where your machines are, let alone what IP space you’re using. We literally had somebody tell us, hey, I need my network scanned. Right. I need you to find all the machines. Oh, by the way, here’s an A-Class, B-Class, and C-Class, go have fun. And we’re like, Wait, what? Huh? You know, so getting people to understand their own networks. Yeah. So networks is like the first problem that we have to overcome.
Liam Downward 13:25
While the problem to overcome is not about whether they need to do it, it’s about okay. My main problem is a resource issue time, money and people, right? That’s the three issues of why people don’t do crap on their network. Because they’ll say well, every time I try and…
Jeff Man 13:39
There’s a fourth one to Liam and i’m not – not to disparage anyone – let’s say it’s naivete, they don’t know that they need to or nobody’s holding their feet to the fire.
Scott Lyons 13:52
Respect my naivete!
Dmitry Zagadsky 13:53
Or even a trust issue.
Jeff Man 13:54
And enter compliance.
Dmitry Zagadsky 13:56
Yeah, they don’t believe that it’s a real issue even though the tool says it is. That’s often…
Josh Marpet 14:02
It’s the same thing, It’s ignorance or naivety or you know, lack of…
Jeff Man 14:07
I was gonna say they’re stupid! but I’m trying to be fair and objective because they’re not…
Liam Downward 14:07
I get that – IT want to be security, right? And which takes priority. If your IT is your main job. That’s going to become your priority security. Take second fiddle. Right. So now we’re now you play in that juggling act, which comes first and ignorance is always bliss. I’ll focus on what I want to do what I need to get do because some VP or someone executives as they get my Get, get this done and get this fixed, or here’s an application, I need to install it as next six weeks, then security is not going to get focused on.
Scott Lyons 14:40
Yeah, but security is always going to be the second in line, right because it produces the product, therefore, it is turned a into a profit center. Not a cost center like security is most of the times. Companies struggle with being able to turn the security team into profit center and pull it away from a cost center inside of the business. Right meaning cost center, meaning it’s a black hole, right? Much like owning a boat bringing out another 1000, It’s a black hole to dump money into. And hopefully, you know, they can protect the product that you’re putting out (Solar Winds). Um, so how do you how would you go about doing that and getting somebody to understand that no security really should be right in line with the other profit centers inside of the business?
Liam Downward 15:30
That’s a good question. Good question.
Josh Marpet 15:32
(laughter) It was just Jeff’s comment about what’s the difference between ignorance and apathy. And we both I actually got it and we both came up with I don’t know, and I don’t care, you know?
Dmitry Zagadsky 15:43
Okay, now I’m looking Yeah, that’s good.
Jeff Man 15:45
Well, but in the business world, it’s a combination of, it’s not that they don’t care. And it’s not that they don’t know. Very often, it’s being overwhelmed. Not clearly in your job description. People are assuming you do it, because you’ve got it in your job title. Yeah, there’s, there’s a litany of reasons why. Well, it’s somebody else’s job, and certainly, somebody else has got it covered. And so it’s all being done. But the bottom line is, don’t make me think don’t know, don’t put it on me. I don’t have time I don’t have the resources. Can we automate it? Is there something to do? Can I set it and forget it? Is there a box I can buy with a Blinky light that’s, you know, just tell me what colors they need to be, you know, which is where I love PCI and I hate PCI, because PCI at the end of the day is in terms of the 500, whatever detailed requirements, they are 60% of it is all about? Do you have documentation? Do you have stuff written down in a policy? Do you have stuff written down in a standard? Do you have stuff? Do you have all your procedures documented? They call it a formal procedure.
Josh Marpet 16:55
Let’s talk about this for a second, Jeff, because I want to build on that if you don’t mind. Cuz somebody asked about talking about in Discord. I forget who I’m terribly sorry. Somebody asked about scoping. And yeah…
Jeff Man 17:05
I wanted to bring that up -Yep.
Josh Marpet 17:08
Go for it if you want to, I mean, PCI…
Jeff Man 17:09
No, no – please set it up.
Josh Marpet 17:11
It’s because in PCI, it’s all about the cardholder data environment, the CDE. So obviously, the scoping is actually a very important part of PCI. I hope you’d agree with that. And what’s interesting is that in other not so carefully designed standards, scoping is very much more difficult to get properly correct. And even in PCI, we’ve run across situations where we handled the CDE as the scope of everything we did, and we said, you know, things that are sorry,
Scott Lyons 17:42
CDE – explain it?
Josh Marpet 17:44
I did – cardholder data environment. I said it a few minutes ago. So yeah. So but we said let’s look at these, the CDE adjacent servers, and we ran scans and found 10s of millions of card numbers, because was another department that shouldn’t have even had access to card numbers that was grabbing them. And that was a problem. So even in PCI scoping is important and sometimes miss done not not by the PCI department or the PCI relevant compliance people or security people, but by the people around it. In this case, it was a department that was grabbing numbers back from the processing company, and they didn’t know not to send it to them. It’s a relevant contact. Right? So in other standards, you know, how do you scope properly when somebody goes, Oh, no, no, we’re not going to do that. That’s a production system. Well, that’s kind of important. Oh, no, no, we’re not going to do that. That’s a system that we just use for testing, and code promotion, and staging and hot fixes. And well, isn’t that important? So scoping is what the devil? and the discord is doing some interesting, interesting pictures right now.
Jeff Man 18:58
I…. and that, and that’s from a woman too. So yes,
Josh Marpet 19:03
yes. Yes. So..
Jeff Man 19:04
I’m glad that you brought up scoping because I did see I think it was M that asked the question and I had already forgotten. So I’m glad that you reminded us to talk about scope. In a PCI context, it’s sort of like the way we were talking earlier about how penetration testing has been kind of misused and abused and exploited in terms of, you know, unscrupulous vendors trying to sell things. In the early days when the PCI data security standard was first released. The presumption was this is an assessment of your security program. How you’re doing security in your organization, with particular attention to a specific type of sensitive or critical data, which is payment data cardholder, data, CHD. The presumption was your a flat network and everything is in scope. Until you, unless, and until you can prove differently. And so the concept of network segmentation was introduced early, and it was introduced as a way of limiting the scope of the assessment. What is an assessor a QSR gonna come in and look at and sample, and so on and so forth.
So the good and the bad of it was that companies sooner or later figured out that, well, if we can isolate things, we don’t have to pay God knows how much money to have people come in and look at everything in our network, they can just look at a relatively small percentage of our network, that’s cost savings, let’s go for it. Let’s start doing scope reduction by creating this thing called a cardholder data environment and isolating it and segmenting it. My problem with that has always been that the motivation was, let’s dodge, let’s get around not having you know, let’s try to find a way to limit what people are going to look at. I’ve always assumed as a QSA, meaning that they’re not doing all the security stuff that they’re that they should be doing according to PCI everywhere else in the network. And you can argue, and I would, that just about everything in the PCI data security standard is stuff that you should be doing in your entire network anyway – I understand that’s hard to do. And, you know, depending on certain other requirements, but as a general principle, the PCI standard is a framework of how to do security for your network.
The good news is that in company’s efforts to try to limit their scope by trying to isolate systems to do that, and to get a blessing to get a pass on it, they had to find out where all their card data was, through either data flows, process flows, you know, the business flows. You know, Josh, you mentioned, it was somebody that, you know, what I heard you saying was, it was somebody that had a function that had nothing to do with like a payment transaction was sort of an over and above thing, which was very often overlooked in the early days of PCI. There were people that interpreted PCI as only having to do with transaction traffic and the systems associated with payment in the early days where it was, if you read the thing and read the small print, so to speak, anywhere card data – anywhere in your network, you know, you know, wherever it is, and in the early days, card data was everywhere, it was in logs and routers and switches, it was in the accounting department, it was in the marketing department, it was in the the groups that were building customer loyalty programs, because the easiest way to find an individual, you know, the key search discriminant, what was going to be unique to all of them was the credit card number. So in terms of scope, I don’t even know just send me remember what the original question was about scope, because it’s about 200 messages ago.
Josh Marpet 23:06
Yeah, it was basically how scope can change and make useless entirely the results of a pentest. However you define it, well like it into that at the moment, but basically, it’s about how just…
Jeff Man 23:16
Interestingly enough, you know, that PCI has evolved to the penetration testing now includes if you have segmented your network, and so you’ve created a card data environment, you’re also required now to prove it with a pen test. So part of the pen test objective is to demonstrate network segmentation. Sounds good. The problem is, PCI never bothered to define what constituted adequate network segmentation other than they take sort of a while there should be complete separation, there should be no it should be sneakernet, there should be an air gap between the card data environment and the rest of your environment, which most people would acknowledge is not terribly practical. So what constitutes a segmented network, you know, given that you’ve got Active Directory that’s going in and out of supposedly an isolated environment, you’ve got administrators that need to go in and out of an isolated environment. You’ve got all sorts of technology that’s automating network monitoring, security controls, all sorts of other controls, the payment transactions, the audio, there’s all sorts of stuff where there’s stuff going in and out of this supposedly isolated segmented network.
So my company and I’m not saying it’s good or bad or indifferent, but you know, we offer penetration testing services and what we do to do the network segmentation validation is basically you know, do an end map, you know, just you know, give us put us somewhere outside of your CDE give us some address space in you’re CDE. And we’re going to do an end map scan or something equivalent to, you know, what’s communicating what can we see ports and services in and out of the network. And there’s not even a value judgment made on, you know, we’ve seen 10, 20-50 communication paths, therefore you fail. It’s simply, this is what’s visible. And somebody else has to decide, is that meeting whatever the objective of segmentation was?
Liam Downward 24:34
No – Sorry. Go ahead.
Jeff Man 24:41
No, go. I’m talking too much. You guys go.
Liam Downward 25:36
Yeah, I just want to I want to ask a quick add to that and say, if you’re doing segmentation and you’re scoping out your pen test, right, guilty by association, so if you’re not focusing on this, basically not included or not in the scope of your purview from a security standpoint, PCI, then how do you look at the risk factor of, you know, you know, crap rolls downhill, right? So if you’re doing kind of like a supply chain, you’ve got people sitting on this on unrestricted environment, connecting back into your, you know, CDE environment to perform actions, right, whether it’s admin, reconciliation, information, or transactions, going back into SAP different things for banking stuff. If you’re not paying attention to that information, then what’s the risk associated of just focusing on the security aspect of that, and not including the other environments, if PCI says, Let’s focus on this, we use use PCI can be any other regulatory compliance or standard out there. But what’s stopping people to say, hey, let’s include this in that realm. So that becomes secure. So we don’t guilt you don’t get guilty by association, because that becomes a crappy Swiss cheese environment. But we’ve made our our CDE environment Fort Knox, but it only takes one path to be able to break down break down that bridge that that wall that you’ve created. So how do you call out that as individuals, right, so
Jeff Man 26:58
Well, you know, ironically, and to bring it back to our “topic du jour”, that’s where I think a pen test could be an opportunity to actually demonstrate, you’re used to have a supposedly segmented, isolated environment, with maybe a couple channels open for you admin of administrative network health type of stuff, whatever it is. Let somebody that’s like a, bonafide you know, pentester hacker can use their brain and kind of problem solve and figure it out, can they break into it or not? I’ve never seen that. All I’ve seen is, well, here’s a port scan, this is the services, this is what we see, this is what it looks like. But that’s to me is where in terms of what penetration testing could be, at least for PCI customers, it could be an opportunity to really kick the tires on and prove or disprove what you’re asserting in terms of network segmentation, in terms of security controls that you have in your environment, and so on and so forth. But I never see that – all I see is what’s commonly become accepted as a pen test, which to paraphrase is a is a glorified vulnerability scan, with maybe, you know, a little bit of Metasploit thrown in on top and a cover letter.
Scott Lyons 28:24
You know, you bring up a really good point. And I want to ask everybody, what pieces should you expect to see in a pen test that you bring in an outside contractor to complete? What are the key elements of that report that you look for?
Josh Marpet 28:43
Oh.
Jeff Man 28:47
Look for it, but I’ve never seen?
Scott Lyons 28:50
Well, to me, it doesn’t really matter. Like if you sit back and you say, Okay, I want a pen test of my HIPAA environment, right? To make sure that it meets high trust standards, right? Or PCI standards, right? What in the report do you want to see? What do you not want to see? Like, what’s the good, the bad, and the ugly of the reports that you have seen in the past? And how do we steer people who are listening to the show to say, this is really what should be in the report, and that hits most of the compliance sets. This is the way the report should be laid out. Like, what are the different flavors that you that the four of you think should be in the report?
Jeff Man 29:30
But it’s funny that you put it that way? I’ve been giving a lot of thought to this lately, and pretty much the question you just asked, and, you know, given that I’m like, you know, I’ve never seen a real pentest. I’ve never seen a real pentest report the way I’m narrowly defining it, which is, you know, test can you break-in, can you be caught that type of thing. Given what’s becomes sort of accepted in the industry as a pentest and what is certainly accepted as a pentest for PCI purposes, for compliance purposes, I wonder if there was a company out there that actually did a pen test the way I define it, which is? And to answer your question, Scott, what I would see in a report is – okay. You know, let’s say the objective is can you break-in, so you give the pentesting company, a red flag, so it’s a capture the flag exercise? Can you get here to this specific point?
Or it’s a little bit more? It’s a little bit more generic, you know, we’ve got sensitive data in our environment, can you get to any of it? Can you see it? And by extension, can you exfiltrated because that’s the ultimate goal. If you were performing a pen test, sort of the industry norm, you’re just getting a list of vulnerabilities. What I typically see in terms of a pen test report is here’s a list of vulnerabilities and the categorize, you know, critical high, medium low. Which begs the question, because the PCI requirement is do a pentest and resolve all exploitable vulnerabilities. And then you get a report that says, Here’s pen, here’s your vulnerabilities, critical, high, medium, low, and you’re not saying whether they are exploitable or not, which, you know, conventional wisdom says, How do you prove it’s exploitable? You exploit it. Or you take the word of a vulnerability scanning technology tool, and a CVSS score that says it’s exploitable based on certain conditions. I don’t know. That’s where it falls apart. But what I never see is, in what and what I would love to see is we came up with this, you know, 3-5-7 different attacks, or in scenarios based on the goal of capturing the flag. We attempted it. This one failed, this one failed, this one failed. This one we got so far. And then it failed. This one worked. This one we were we were caught in the first 10 minutes. This one we were detected and blocked after 80 minutes. And at the end of the day, we got in or we didn’t get in and we got in this way. If we did get in and you didn’t see us and you didn’t catch us, you fail from that perspective. Totally different from a list of vulnerabilities and there are so…
Scott Lyons 32:36
So you’re saying elevate, elevate the pen test game, right and turn it more into a purple team report than an actual pentest report. Is that correct?
Jeff Man 32:47
You’re assuming that I am agreeing on your definition, if not acknowledging your definition of pentest and purple teaming.
Dmitry Zagadsky 32:56
So I have seen a report the likes of which you described, Jeff..
Jeff Man 33:01
That’s awesome!
Dmitry Zagadsky 33:02
Right. So there’s hope for the industry. This is great. But so the and, you know, again, like my view is that purple teaming and red teaming, etc, are extensions on, you know, the standard network pen test. But I think that the purple teaming idea of of having a real collaborative effort of being able to say okay, the pentesters going in, okay, and then the operation security operation center says, okay, yep, you triggered this alert. Here’s a screenshot of it, make sure to include it in your report. That way, say okay, yeah, we know, even though you may have gotten in, we didn’t notice you. So we’re not just sitting on our feet, you know, sitting on our thumbs doing nothing. There’s definitely, that’s probably among the best kind of reporting that you can get where you get. And to your point, the, you know, here’s the, here’s the exploits that did work, here’s the exploits that didn’t work. And perhaps having some explanation as to why they might not have worked, or why they did work. It’s, but it’s always…
Jeff Man 34:03
Here’s my question, Dimitri. And I’ve been pondering this for a while now, you the type of test and reporting you’re just describing, which I love. I wonder if that was submitted for a company’s PCI assessment. So you got to QSA sitting there. Let me see your pentest report. And that type of report was handed to them. When they’re accustomed to seeing a list of vulnerabilities critical, high, medium, low, and we’ve remediated and patched or whatever. I question whether a QSA would accept it as meeting the requirement.
Dmitry Zagadsky 34:42
I actually I’m with you there because that’s part of my, my problem. My internal love-hate relationship with compliance is that sometimes you can’t be fully open and honest, because you’re going to get caught in some weird technicality. I’d rather show, I’d rather say, you know, here, here are the things that we know we need to fix, or here or here is an example of, yep, they got in on this thing, but we did 90% of everything else well, so we there needs to be some credit. You know, like, we’re, it’s, you know, the organization is trying. And it’s not just, well, you that you failed the pentest. So that’s it, you’re done. There’s got to be some give and take. But, and maybe that’s the wrong perspective. I’m not sure.
Liam Downward 35:31
I think you hit something there. And I think that the thing itself, as I’ve seen and experiences is that a lot of compliance, people don’t have a lot of technical background. So they’ll just go what’s actually written in front of them? And they will either interpret it to their own in perspective, and then look at something and say, No, that’s not it, because they have no understanding of what goes behind it right, to actually generate that report or generate the details. What’s the scope of it is, and I think that’s where I think if there was a kind of a happy medium, that you have the tactical aspect of cybersecurity, right? And then you have the strategic, which is compliance, but I think there has to be an overflow where some of the strategic need to have a technical background in order to understand what the pentest is, rather than trying to think, okay, here’s what PCI says, here’s what HIPAA says, here’s what NIST says, here’s what you know, the top 20 from CIS says or anything these things out there, they’re interpreting, that the for themselves of what they would want to see and then goes back to your Jeff probably wouldn’t, because it doesn’t state that in a document they’re reading, because they don’t know the technical and is that going to meet the standards? Right. So I think that’s a problem as well that we have in cybersecurity from a strategic standpoint, not have enough technical experience, and you have the tactical having so much technical experience enough of understanding compliance, and they can marry the two together. But then when they pass it across where they need to, then that’s what it fumbles because they don’t accept it.
Jeff Man 37:04
Josh you still thinking I’m wrong. Review
Josh Marpet 37:09
Sorry, what?
Jeff Man 37:11
Well, you said I was wrong in my definition of pentester. Early on,
Josh Marpet 37:15
Yeah – you were
Jeff Man 37:15
…. in the way our Congress?
Josh Marpet 37:18
Yes, you are.
Jeff Man 37:19
I am wrong, in the sense that of how the industry accepts the definition and uses it as an umbrella term, I am pushing the envelope, trying to suggest that a pentest is a specific type of a something else I do something else would be well, I’m trying to change the world, you can’t disagree with…
Josh Marpet 37:40
That’s valid I can, I can say that you can try to push your view, and get everybody to accept it until you can get everybody to accept your definition, which by the way, I like your definition, it’s actually a very valid one. I like it. Until you can get everybody to accept your definition, the best we can do is say that pentest is a generic term with no inherent meaning, other than as a categorical label for a whole slew of different kinds of security exercises = is exercises, the best word? – any of which could be labeled as a pentest. But they’re each individual types of pentests.
Jeff Man 38:17
So let’s take the last few minutes. And everybody you know, we got to drop in a few minutes because Johnny’s got to move on, but we’ll take this up in the after-party. You know, everybody give their thoughts, at least one or two or half one, whatever you have in your brain. You know, what could a pentest be? What should a pentest be? What is a right way that perhaps doesn’t exist today of performing a pentest and getting the most with what’s the question that we tried to answer and didn’t on Paul security weekly last week? How do you get the most out of a pentest? Thoughts?
Scott Lyons 38:58
It’s it all comes down to scope, scope, scope, scope, scope, scope, scope? Right. Um, even though there are compliance regimes, that currently say we need a semi-annual or biannual or once quarterly test, right? That test is defined as per the scope of the compliance regime, right? So if you can give the accurate scope to the people that you’re hiring, to do the work, and make sure that they give you the product and then work with them on the report, right? Don’t just let it set. work with them on the report to make sure that it answers the compliance regimes, you’re going to get that ROI much quicker than just saying, Well, I have a list of IPs. Here’s a black box have fun. Oh, by the way, I want to know absolutely everything you did.
Dmitry Zagadsky 39:48
So the knowing everything you did isn’t that isn’t necessarily a bad thing. But, and I’m with you on you know, I would say that, you know, if you go back to like one of the very important DevOps principles of making smaller batch sizes, you can go in and have a pen test and say, here’s my entire environment, just go away and hack me. But that’s you’re gonna end up with too many squirrels where they’re going to try to go over here and over there and over there,
Scott Lyons 40:12
and the cost is going to be through the roof. Right?
Dmitry Zagadsky 40:14
It’s highly, it’s not, it’s not affordable, unless, you know, even then it’s expensive. So, if you have smaller scoped tests more often, that might actually end up with, you might end up as an organization getting more out of a pentest, if we’re, you know, strategically if we’re saying, okay, we want to test this one thing, or, you know, this path or we believe that we have a, an over-reliance on this one tool. So let’s see what happens when that tool is failed. So let you know it will scope a test to say, okay, the attacker system that’s in our network, or whatever is being is bypassing that tool, what, you know, testing defense in-depth, you want to, you know, obviously social engineering is involved, there’s a whole bunch of other avenues you can do.
But if we’re talking network pentesting, I would think that because a lot of the regimes as well say, well, you need to have it every year, or whenever there’s a significant change. And that’s like the big, the big thing, I think, even PCI says, There’s my one PCI for the day. But you know, what is a significant change, and that’s up for debate. But I think that, in the best-case scenario, having a test, before you make a significant change like if you bought if you’re onboarding a new system, or changing the, you know, authentication method of something, it’s a good idea to know, okay, well, you know, what you think everything is going to work, like, but then have someone go and bang at it and, and make it do things that wasn’t designed to do and see what breaks, see what falls out. That’s a really good use of the time, but again, is, you know, if for an internal team having to hire outside consultants, that can get really costly. So, you know, it’s, we’re always going to be struggling with it.
Liam Downward 42:06
All the other aspect of it is, is how you write the report from a being technical to also having nontechnical, right. So in a lot of my past of developing pentesting reports is to do a storyboard, right? So we can easily walk through it kind of like the building out the scope, the steps that you’ve taken, but also put in the mindset that an executive-level person has to read this, to actually allow you to be able to obtain your goals, whether you need to enhance your security budget by some additional tool to get some more training and so forth in place. But these are the things that I think get lost because a lot of pentesting results end up being more focused on technical, then you got the technical person trying to convert that into executive jargon. And I think he gets lost in translation. Because the cop out is of the security services and not willing to write something that the executive leadership needs to read in that document.
Jeff Man 43:01
Hey, unfortunately, our time is about up for today. I feel like we could we I feel like we’re making progress, actually, which is more than I think we were doing last week on the panel. But that’s my own impression. Dmitry, thank you for joining us today. Thank you for giving that talk besides Boston that got us got me spun up about this topic. Again. Liam, thanks for joining us today. To all of my co-hosts and to all of our listening audience. Want to wish you a happy holiday restful, safe, be socially distant, wear the damn mask! Let us know when you’re first in line to get a vaccine. We will come back again I’m flipping the calendar. I think we’ll be back on January 5, if I’m not mistaken. And we’ll keep going. Until then. Thank you for listening and engaging. We will of course jump over to the after party chat on Discord. Now we can keep talking for a little bit. There’s been a lot of questions and a lot of good comments raised in the discord and we weren’t able to pull it all into the conversation. So that’s gonna wrap us for security and compliance weekly for 2020. Take care, be safe, be safe until next year.
