As the old Bon Jovi song goes, “good guys don’t always wear white.” Today, we’re talking to Greg, a penetration tester who makes his living by legally hacking companies to test their security posture.
The media frequently portrays hackers as guys wearing hoodies clacking away at a BASH interface at 2am. The truth is that a large majority of hackers fall into two categories: organized crime and the professionals (AKA “penetration testers” or security researchers) who operate legally on behalf of companies. Neither wear hoodies, typically.
I recently did an interview with Greg, an ethical hacker who makes his living by hacking companies that have hired him to find flaws in their security posture. Today, he reveals how he got into this field – and how you can too!
I add in some clarification when Greg used acronyms or names that may not be familiar to everyone. When I did so, I put my clarifications [within square brackets] to differentiate them from Greg’s answers.
Also, a word of caution: if you don’t have permission to hack someone or a company, doing so is illegal. I’ve included a lot of links here, but you should only use the resources on a network you own or if you’ve had permission to do so by the owner. Seriously – this stuff can land you in prison, so don’t mess around with it.
Now, without further ado, here’s Greg!
What exactly does a pen tester do?
What does a pen tester do? That’s a great question, I’m still trying to figure that one out! What we do is look for vulnerabilities in client’s networks and web applications. If we are hacking in an Active Directory environment [I touched on Active Directory in this post], then the goal is to get Domain Admin, which is the highest privilege account. A network penetration test from start to finish would look like the following: first we will perform reconnaissance of the client’s network. Recon is using online resources to start to create a picture of the client’s external attack surface-using various web sites to figure out their email format, building an employee list from LinkedIn, and using Arin to find which subnets belong to them. The second step is enumeration, which is where we’ll actually start touching their network. This phase is where we will port scan their external attack surface, looking for vulnerable services. If we find any anomalous service or port open on Shodan, we’ll poke it a bit. [Note: I included the link to Shodan for educational purposes, but you should not use this to illegally hack anyone. I’ve heard prison is no fun!] Shodan is essentially a search engine that scans the entire public IP [Internet Protocol] space. It’s very useful for finding that old router in the server room that has not been patched in 10 years. The third phase is probably my favorite: exploitation. During this phase the penetration tester is using all of the information that they learned from recon and enumeration to start attacking the clients infrastructure. Remember that employee list that you built from the clients page on LinkedIn? It’s time to ‘password spray’ their O365 [Office 365, which is essentially Microsoft Office tools that companies use in the cloud as opposed to on their premises] authentication portal with those names. A password spraying attack is where you use several commonly used passwords against a user list. An example of a ‘commonly used password’ would be “Winter2019!” Once we gain access we begin the fourth phase of the penetration test, privilege escalation and maintaining access. I can’t stress this enough, you want as many sets of credentials as possible in case a user gets wise and changes their password. You can get more credentials by getting into email inboxes and searching for the word ‘password’ that will recursively search through the employees inbox and pull out any emails that have clear text [clear text means anything that is unencrypted and can be viewed by anyone] password credentials to other accounts, which is great for gaining more access. Oftentimes users will have password stored in a plain text document in OneNote in their O365 account. The last step is usually the one that gets forgotten about while the pwnage [slang term – pwning means taking over someone’s computer or server] is going on, it’s reporting. Its also usually the least liked phase of pen testing from an engineer’s standpoint. Reporting is when we create a very colorful story of how we gained access to the client’s environment, all the way through gaining Domain Admin, which we always do.
There is also web application penetration testing, which is hunting for vulnerabilities in websites. The type of vulnerabilities is very different than network-based penetration testing. In web-app pen testing you’re looking for SQLi, XSS and the other vulns in the OWASP Top 10. [These vulnerabilities might be a bit too technical to get into here, but if you’re interested, they’re explained in the link.] OWASP stands for the Open Web Application Security Project.
The last type of pen testing is ‘physical.’ Physical penetration testing is testing the physical security of a building. Oftentimes it goes one step further, and we test the human element as well. To test the ‘human element’ we will attempt to talk our way into a building and gain access to sensitive areas like the server-room.
So if you’re going into penetration testing, are you expected to know all these aspects or do people typically specialize? For example, someone would do mostly network penetration testing but may not do web app penetration testing?
Right, so there are companies/people who do all of them. At VDA Labs where I work we work, we perform all facets of offensive cyber security. However, there are companies that specialize in one specific area. For example, maybe they only pen test mobile applications, or only web apps. So it’s not a clear path; you can do one thing, or any mixture of them.
What does a “typical” day at work look like? And I realize that it’s somewhat of a loaded question.
(Laughs.) Wow. A typical day where I work…doesn’t really exist. And that’s what makes it exciting. I can work from home, or I can go into the office because VDA Labs has an office right here in Grand Rapids. I tend to work from home more, so I’ll wake up around 6am, get the coffee going, do some Python [Python is a programming language that’s very popular in the InfoSec field. The link is an affiliate link to the book that I used to learn Python. It teaches you by having you write games.] training or study for grad school. I’m currently attending the SANS Technology Institute for a masters in cyber security engineering. Around 8am everyone starts working. We tend to have several jobs going on at the same time because we’ve been growing very steadily. I think we’ve doubled in size in the last twelve months. So, now VDA LAbs has become so big that there are jobs that I don’t even touch anymore. So, for instance right now, we’re doing a network penetration test that’s just a normal penetration test and the client also paid for a web application penetration test. So I’m doing both those jobs right now. But as far as a normal day, we end work around 5pm [17:00 for international readers].
Ok, so you guys don’t try to sneak in in the middle of the night or hack when all their IT teams are asleep, or that kind of thing?
We have not done a physical penetration test in the middle of the night. The hairiest physical penetration test that I have ever done was when we had to get into a data center. A client was paying us to try to get into it and syphon off some client data. So, I reconned this data center; I think it was three times I went there just to try to look for any vulnerabilities in physical security. They were very buttoned up. There was a steel fence all around the data center, there were spikes sticking up out of the fence that faced outwards so throwing a rug over to go over the top was out of the question.
To get into the parking lot, there were only two ways in and you had to have an RFID badge so we could have used a box splitter or a Proxmark [I’m including a link here for educational purposes. However, you should not use these items unless you’re a pen tester with permission from a client to use these. Doing so without permission is not only unethical, it will land you in prison! Also, the link is an affiliate link.] Basically, you would make a replica of the RFID badge. Then, once you’re inside the parking lot, to get into the building is another RFID scan, then there’s an iris scan on top of that, and then security meets you at the door to escort you to get your ID.
So what we ended up doing is, we were doing an internal penetration test for that client at the same time and we got into their ticketing system. [Most companies will have software for users to file support tickets] and we basically found where you could submit a help ticket and pose as the AT&T or Verizon technician that works there. So we ended up doing that. I “became” a technician that works at Verizon for the day, and we posted this ticket on their online system. They were actually expecting me to be there! So, I pulled up to the gate and they let me in. And I had no RFID badge, which is expected, because why would a technician have an RFID badge for the data center?
Sure, and you’re using your actual name, so you just show up with official ID if they ask and you’re good to go.
Exactly. So I had my Verizon badge and my real ID. This all looked very official. At that point, they let me into their demarc [this is short for demarcation point, which is where the public network, such as a phone network, meets a company’s internal network]. That’s obviously very bad. We did not end up planting a device on their network to siphon off traffic because a data center is a very sensitive operation and we didn’t want to disturb anything that was going on. We didn’t want anything that could pull back towards the company.
Did you have to get prior agreement, let them know you would be trying to get into their data center? Are there any ethics that go into that, or does anything go since they hired you?
Great, yeah. So you probably heard about what recently happened with the two penetration testers at Coalfire recently?
Yeah, they got arrested.
Yeah, exactly. So, you need a couple of people at the company to know this is happening. Usually we have sign-on from the CEO or someone in the C suite [C suite refers to people who hold high level positions that usually start with the letter “C” such as CEO, CFO, and CTO] that’s high up in the company. If anything, before going on-site we carry around this piece of paper that says “your company approved this action” just in case things do go sideways. I have personally never been rolled-up on a physical penetration test, but another engineer at VDA Labs has, and that piece of paper is essentially your get-out-of-jail-free card. It’s very important to have that.
Changing tack a little bit, how did you prepare for this position? (College degree, certifications, any kind of experience?)
Getting into penetration testing, the barriers to entry are pretty high because it’s the sexy thing to do in cyber security, right? So I would say I got very lucky – it’s where a bunch of hard work met with opportunity and I got the job. My undergraduate degree is from Grand Valley University and it’s in Political Science. That’s a Liberal Arts degree that has nothing to do with anything IT or cyber security whatsoever. Then I did seven years in the Army Special Operations.
When I got out, I was a stay-at-home dad between 2017 and 2018. But while I was a stay-at-home dad, I built a home lab because I wanted to get into cyber, I got into the SANS VetSuccess program, and I picked up three SANS certifications. One of them was specifically in penetration testing. Also, during that time I reached out to Dr. Jared DeMott who’s the owner of VDA Labs and we started a conversation: what are the things that I have to do to get into cyber or even come work for you? And he had a lot of good suggestions, so I basically did those things. And then in August 2018, I started working at VDA Labs as a junior penetration tester.
So, I have a question for you based on what you said. The Army actually has cyber jobs, but you did not have any kind of cyber job when you were in the Army?
I was in 18-Echo. That’s a Special Forces Communications Sergeant. It’s an enlisted position in the United States Army. I did do some networking: I was working on Nipper/Sipper networks [these are interconnected networks. Nipper is non-classified and Sipper is secret] and also VoIP phones [Voice over IP, a protocol for voice communication] at the same time. I was working with satellite data, but nothing as technical as what I do now.
It was like scratching the surface of technology and after I got out I wanted to dig deeper, which is why I got into offensive cyber security.
Is there a book or other resources (you already mentioned SANS) that you would recommend for someone wanting to learn about this job?
Yeah, there’s so much. There’s Pluralsight. It’s an online training – I think it’s $30 per month. There’s Udemy. Udemy is more like you pay for one course and you have it forever. So recently I purchased “Testing SOAP and REST APIs” [APIs are “Application Programming Interfaces”. Think of an API as a piece of software that allows other software to call upon it. For example, your phone interacts with a weather API that pulls weather data onto it.] I also just purchased “Penetration testing mobile applications.”
There’re also tons of good Youtube videos like Ippsec – he does all the hackthebox walkthroughs for the hackthebox labs that have been retired.
There are also a lot of good podcasts out there – Darknet Diaries is a really great one that’s focused on offensive security. Risky Business is another great podcast out of Australia. It’s not really as deeply tech as a Black Hills Information Security podcast but it’s still a really good one for just keeping up with what all the businesses are doing for big mergers and acquisitions. SANS does have a podcast also, The Storm Center. VDA Labs also has some really good blogs out there if you’re interested, like IoT or fuzzing. My boss, Dr. Jarred DeMott, basically wrote the book on fuzzing [this is an affiliate link].
You spoke about the route that you took to get here. If you could redo the route you took to get here, would you take the same path or would you do things differently?
That’s a hard question because hindsight is 20-20, so I could go back and redo something, but would I still be in the same position if I changed something? If I could redo one thing, I would probably not have gotten the CISSP certification [I also have my CISSP – here’s the book I used to study for the CISSP. (Affiliate link, thanks for supporting this blog!)], I would have gotten the OSCP while I was a stay-at-home dad. I was doing a lot of studying at the time, and that cert took about four months of my time studying full time. It doesn’t have a lot of application to what I’m doing now. What I’m doing now is very deep, kinda in the weeds penetration testing, which is what the OSCP and OSCE certifications are more based towards.
Right, the CISSP is more a mile wide and an inch deep.
Right, it’s more the government/risk compliance side of the house.
And you did that in four months? It took me nine months, so I’m a little jealous.
(Laughs.) I believe you were working full time at the time though.
Still, it’s impressive and I’m a little jealous. Ok, and just a quick follow-up since you and I both have Liberal Arts degrees. I want to call out that your degree, the fact that you did not have an IT degree did not hinder you in any way?
I would say it hindered me slightly because someone coming out with an undergraduate degree in Computer Sciences would have a much stronger base of fundamentals than I did coming out with a Political Sciences degree, but you can build those fundamentals as you go along.
This question is actually from a Redditor, u/HeftyNull3: Business interests sometimes compete with security. As you perform penetration tests for various companies, do you ever encounter situations where that happens? Or are you shielded from the politics?
You know, that’s a great question because every company out there wants to send you some appliance that has bells and whistles and has a bunch of false positives and is constantly telling you that you’re being breached, right? I think there’s a balance. There are so many free tools that you can use with your network! For example, you can implement LAPS, which is a Microsoft product that randomizes local administrator passwords. You can use the ELK Stack for log data analysis. You can segment your network so that if one part of your network is breached, you don’t have a flat network and the penetration tester can’t pivot all over the place and get everything.
There needs to be an emphasis on training your people and getting your processes firm and then maybe there’s a little piece of technology at the end. Maybe we can get an EDR [Endpoint Detection and Response technology] after we get our people trained and processes in place. But having the really expensive EDR should not be the first thing that you go to.
What is the most frustrating thing about your job?
Oh man, the most frustrating thing about my job? Well, the thing is, you have to love the journey. You have to love the pain. If you don’t love the pain and you don’t love cognitive dissonance, you’re not going to like cyber because every day I’m learning. There’s constantly Googling going on. How do I do this? Why can’t I install this Burp [Burp suite is a web vulnerability scanner] certification as a user on a new version of Android? That’s a problem that I had last night, so I Googled around until I found it. And that’s just one example. I run into road blocks every single day. You just have to break through the wall or go around. So if you’re not that kind of person that’s not intensely curious, about solving problems, you’re probably not going to like tech. But if you are, man, the sense of accomplishment after doing a really good job, like getting DA [Domain Admin, the most privileged of user accounts] in an Active Directory environment [see my previous post on the topic], you just rejoice!
Does this job require lots of continuing education? If so, is it mostly on the job or outside working hours (such as taking night classes, working on certifications, etc?)
Yes, cyber moves so incredibly fast because it’s constantly a game of cat and mouse. What worked last year probably isn’t going to work six months from now or even three months from now. For instance, when I was getting into offensive cyber security, everybody was using Empire [an exploitation tool based on PowerShell], we were using PowerShell [Affiliate link – this is the book I used to learn PowerShell, and I can’t recommend it enough!] for offense and now, PowerShell is essentially dead and all the platforms are going more towards C# [this is a programming language] based.
You have to stay up on your blogs, you have to stay up on what’s going on in the market, what are the APTs doing, the Advanced Persistent Threats. It just moves fast.
Ok, and to clarify, a lot of that is going on outside of working hours because that’s when you’re reading the books, blogs, and whatnot?
Absolutely. And if you have a really good employer, they’ll allow you to do on the job training or research. Maybe they’ll allow you to do one hour while you’re working of research and study during that time.
Is this job mostly solo work, or do you interact with others a lot?
At VDA Labs, we have a tiger team approach. If we’re doing a pen test, we’ll have two guys on it. If it’s a bigger job, we’ll have three guys on it. But then, the really big consulting firms, they’ll have one engineer on a penetration test. I think VDA has the right approach; our clients definitely get a lot of value for their money and we consistently hear that we are the most in-depth, the most detailed and thorough engineering firm. Some of our clients have these penetration tests annually and we always hear during the report delivery: “wow, you guys really crushed it. This is the most thorough test we’ve ever seen.” And that’s obviously what we want to hear.
What is something you would want to tell people about your job that’s not immediately obvious for someone who might be researching it right now? Is there anything people should be aware of?
Yes, you definitely want to be aware of trends and where everything’s heading to. I mentioned something a couple of minutes ago, about offensive cyber security moving towards C# and away from PowerShell. That doesn’t mean you don’t have to know PowerShell. You still have to know PowerShell because it’s very efficacious if you’re doing some incident response.
If you’re really learning, where you start is with the blogs, learning Python [Affiliate link], learning BASH scripting, really just learning about how networks work also. Because your security really comes at the network layer, so building a network in your house, that’s probably the best way to start. And basically becoming your own sys admin [System Administrator] for your own Active Directory environment.
What is something that surprised you about this position?
Probably how exciting it is. I really like web application penetration testing and physical penetration testing and the depth that you can go with it. There’s always something new to learn. It’s never over. It’s not like some fields where you graduate and get your degree, work for thirty years and then retire. The learning is ongoing, it’s very intense, it’s definitely a struggle you have to love.
What is the coolest or most memorable thing you’ve done on the job? You already told us about how you got into a data center, and I’ve been in data centers, and they don’t look like you can actually get in them if you’re not supposed to be there, so that’s impressive. But is there something that tops even that?
One time I had to hide under a desk for a half hour during a penetration test. So this specific penetration test I did, it was a double tail-gate. There was an RFID badge scan, then I followed a woman into the building, and then there was another internal RFID badge scanner. And she was onto me after the first one. And so she actually stopped in what’s called a man-trap [a type of door that only allows one person through at a time] to see whether or not I could scan into the next door. So what I did, I got on my phone a little bit, I went on Facebook, and tried to look like I was doing something because the truth of the matter is, I needed her to go through the next door before I had access to the internal building. At that point, she looked at my badge very thoroughly, (I had a fake badge for the client), so she eventually let me in. Once inside, she never took her eyes off me. I used the bathroom a couple times because I had no idea of the internal workings of this building. So I’m walking around, I find an unused cubicle, I plug in my device so we can syphon off some data from their network to reach back out to VDA Labs servers. She walks by because she’s looking for me, and at this point I’m hiding under a desk. I see her walk by, and I’m wondering, do I come out now? So I basically waited there for a while. Then I came out and she saw me come out from behind the desk. At that point, I left. She didn’t pursue me though. I got out of the building and then I left. For that specific client, they had called the cops and they showed up five minutes after I left.
Did they find what you planted on the network?
They did, after a couple of hours. But by that time we had already gotten what we came for, data-wise. That’s probably the closest I’ve ever gotten to getting arrested.
Well, I’m glad she didn’t chase you. If you had nefarious intent, somebody could get hurt that way. So they probably shouldn’t be chasing you. And anyhow, you’re a big guy, so… Yeah, I wouldn’t chase you either. Well, last question. When you’re preparing an interview, you can think of questions to ask, but you’re never sure that you asked the right ones. Is there anything you wish I’d asked you about? Did I miss anything important?
I’d like to re-iterate how to get into cyber. A lot of people think that they need a job to get experience. Cyber is one of those unique fields where you can get a lot of experience in your home with a lot of free resources. You can build your home lab with virtual machines. Virtualbox, it’s completely free. Kali Linux [seriously, don’t use this for anything illegal! I’m including the link for educational purposes.] is a free penetration distro of Linux. You can buy a switch or a router, set up a home network. That will cost a little bit of money. Reading blogs, listening to podcasts, all that stuff is free. There’s really no excuse to not be getting experience if you want to get into this field.
A lot of times, when people ask me about getting into cyber, at least, the part of cyber that I’m in which is just a little bit more administrative than what you’re doing, I always tell them they should come up through Systems Administration or Network Administration and get some experience before you jump right into security so you know what you’re doing. But for pen testing, it sounds like you’re saying that you can gain that experience at home and then you can jump into it?
If you can just go out and get a job in cyber security, kudos to you. I had the CISSP and various SANS certifications and a top secret security clearance and I couldn’t get a job in cyber for a while. That also could be where I live – I live in Western Michigan and it’s not a super hot tech area. But there are tech areas like San Antonio, Texas, and Augusta, Georgia, where the scene is pretty hot. If you’re not in one of those top tech areas, you can still get a lot of experience for no money or very little money in your house. But yeah, if you can get a job with no experience, kudos to you.
Makes sense. Well sir, thanks for talking with me today.
Thanks for having me on!
With each post, I cover a new topic to help you get your start (or keep progressing) in your IT career. If it’s your first time visiting this blog, start here. Or, see all my posts about interview questions you should definitely be prepared for.