City and County of San Francisco’s Nathan Sinclair Share His Experience of BitDam’s PenTest
Nathan Sinclair heads the Cybersecurity Defense team of the City and County of San Francisco providing IT security services to about 30,000 employees. He has recently engaged with BitDam, used its PenTest in several ways and got to some conclusions. In this interview, he shares his experience with BitDam’s PenTest including some specific insights about the process, how it helped him assess different email security solutions and even push for doing more in less time.
Nathan, can you please give us some background about yourself and your job?
Nathan: I manage the cybersecurity defense team for the city and county of San Francisco. We are a central service for cybersecurity monitoring and alerting which serves the entire organization.
One of the newest additions is that now we are also focused on email protection. Our biggest challenge was phishing because we knew it’s a growing problem but didn’t have much visibility on what was going on, so that was the main trigger for our email security solution search.
How did you hear about BitDam?
Nathan: Our CISO, Mike Makstman, brought it to my notice. I heard about BitDam before but didn’t have any direct touch with them. Then Mike told me about them and that they use an interesting approach. So I did some research and found out that it is indeed a different approach to how all others do email security and it sparked my curiosity. That was when we started to kick off, saw a demo and understand what it does. Understanding the technology underlying behind it, I realized how valuable it could be. That’s one of the reasons we went forward with procuring it.
Ok, so what was the next step?
Nathan: To start testing we used the BitDam online PenTest and forwarded some malicious emails to the BitDam portal to see how it works. Just like we did to other email protection solutions. I know that this wasn’t the perfect test, but that was the best we could as an initial step.
Alright, can you tell me a bit more about the PenTest itself? What was done there?
Nathan: Well, the Pentest – that was interesting!
I started with the free online PenTest – very simple. You just put your email address there. The first time we did that was actually very helpful because we tested multiple solutions using the same PenTest – sent the same emails to mailboxes equipped with different solutions so we got a true comparison.
Then we rolled in into the advanced part of the BitDam PenTest working with the company’s team. That was really good because the number of emails that were sent to all solutions was high and it gave us a representation of what emails the products could see, which ones saw what, whether they were able to detect malicious files and so on. This helped us narrow down the solutions very very fast. This is the fastest POC that I’ve ever done for so many solutions at the same time in my whole career.
How many solutions did you test?
Nathan: We’ve examined about 5 solutions in total. We had licensing set up from different solutions to some internal mailboxes so each mailbox used a different solution. It was interesting to see in real-time how different solutions handle different malicious emails, which alerts they send etc.
What kind of products did you check in this PenTest?
Nathan: All products we’ve compared to were email security solutions. Some of them had additional functions like sandboxing and advanced analysis of the messages, so it was kind of a mix.
How would you evaluate these solutions without the BitDam PenTest?
Nathan: It would have been a similar process but a lot slower…We would have to wait for certain malicious or phishing messages to come to us for real in order to send it to each of the solutions.
How long did the process of comparing these 5 solutions take?
Nathan: Honestly, once BitDam started to send all those messages the test was very quick. This PenTest was way more efficient than how we’ve been testing other solutions before. The PenTest analysis took about a month in total, and that was only so I can pull data and make sure I’ve tested all the features and covered all bases.
How easy was it to operate? Analyze?
Nathan: The initial one on the website was super easy. Literally, put your email address in, click a button, and click submit. The advanced PenTest was also easy. We just had to let the team know which email addresses to send the messages to. I had alerts set up so I knew when it was coming in, what time. It wasn’t anything that was complicated.
Anything worth sharing with others who may do this PenTest?
Nathan: We had to figure out a way to count the messages that did pass and came in, and there were hundreds of such messages. To deal with that, one of our guys set up a rule so he could tell me every morning how many messages actually made it to his mailbox. He just created a folder in order to track it and it was very interesting to see how many did make it through.
Also, to us, the PenTest helped us assess how we will operate on those systems when we will get a false negative. Good representation of what’s going on is a pretty big deal to us since we serve different departments.
Were you surprised by the results?
Nathan: You know what, no, I wasn’t. We asked our peers what other solutions and services they have, and the actual experience they had with these solutions. So when we tested one of the first ones I wasn’t surprised, it was typical.
I was surprised by the speed of this PenTest which gave us the amount of time to be able to do everything that we wanted and even more.
And what was your impression of BitDam?
Nathan: I can definitely tell that it’s a company that doesn’t just sell a product but really builds a partnership which really fits how we operate with vendors. I think it’s really cool how the product looks at email very differently. The BitDam approach – creating the baseline of how something is supposed to work – was a key driver to make the decision to have it as a security blanket, especially for mailboxes that are more targeted than others.
Are there any cyber trends that you notice at the City and County of San Francisco?
Nathan: Our biggest target is our end users. That trend is going to continue. Malicious emails are looking more and more real every day. There have been a lot of messages that were targeted to us, that looked very genuine from where they come from and they are not. They send you to websites or places that look just like the website that could potentially send it. Once the user has clicked on it the damage has been done. I think we have to combine education of end-users and technology such as more intelligence and dynamic analyzing of those messages.