Cybersecurity at Nanosecond Speed | Securing High Frequency Trading

Now we are talking about single digit nanoseconds these days. Everything adds up to this.

Welcome to the Security Cocktail Hour. I'm Adam Roth.

And I'm Joe Patti.

Adam, looking good today.

Thank you.

So this is going to be great. I'm getting back to my roots. We're getting back to the stuff I did back in the day. So this is going to be fun. If you know what that is, put it in the comments. Today, we have a great guest. We have Jay Manipalli. Jay, welcome to the show.

Yeah, honored to be here. Nice to be here. Of course.

No, it's great. It's great seeing you. So Jay, why don't you tell us a little bit about what you do?

I am basically, I have been in InfoSec for close to 10 years now, but a part of it, now I'm working for a company called IMC Trading. We are like a high-frequency trading firm based out in Chicago. So I'm basically the InfoSec officer, basically doing work for IMC Trading. Anything related to trading, anything related to financial markets, anything related to exchanges, that would be us. So on a day-to-day basis, you will probably see me configuring firewalls to configuring, I don't know, identity protection SaaS vendor, or you'll be seeing me working on an AWS service, AWS account and trying to figure out, butt my head towards it to figure out how to make things work in AWS or any other kind of cloud service that we use.

Yeah.

Um, and, but yeah, before this, I was basically into like management consulting, revolving around security, uh, like working with healthcare firms to make sure that they are bringing up their security up to, up to speed. But life just happened. And, you know, I came into high-frequency trading firm and life has been amazing so far.

I knew one guy once he, um, When it was a big thing, I don't want to take it away from Cisco, but when it was a big thing to be a CCIE and the guy was working on a trading floor and he was working with somebody else and the other guy, unfortunately, what he did was he made a misconfiguration of the firewall that cost or lost 13, 14 seconds of trading. And they literally just walked him out the door after that and said, you're done. If I understood correctly, it translated into millions of dollars lost in like 14 seconds. High frequency trading.

Like a second in a trading environment is almost like, let's say an year of downtime. That's a lot of money. I mean, I was talking to my friend recently about how we try to shorten our time and bring like low latency as much as possible. And this is where it kind of gets tricky, right? Because when you're introducing like security controls or let's say even firewall, right? it kind of breaks things or it makes things really, really slow. And even like a microsecond of delay can cause like so much of losses. And then you have to kind of figure out what works where and where can you make that trade off.

This is a perfect segue, right? I think, you know, even myself, including, right, we all understand high frequency trading from a, you know, from a layman's term, but maybe we should kind of describe that, right? What is high frequency trading? What does that really mean to somebody who's listening to this? Who wants to understand that?

I mean, high-frequency, think of us like this. I mean, we are high-frequency trading firms as well as market makers, right? So, who are we basically at the end of the day? So, basically, when you go on to your trading platform that you use, let's say Robinhood, or any other app, you go and see, oh, Apple today is trading at $500 or a stock or something, but who decides that it is $500? It's basically firms like us that basically get all this information from exchanges, see what is the trading volume based on the demand that various people are buying and selling. And based on that, each of these trading firms compete with each other to like see what is the most optimal price can I offer to the person who is trying to buy or sell them. And this has to be done at like really, really lightning speeds. Whichever trading firms offers the most optimal price, the lowest price wins. And I think all these firms like us, Jane Street, Citadel, we often compete with each other to make sure that we are providing the best price while making a fraction of it when somebody is trading on these stocks, if that answers your question.

Thank you. I mean, obviously, of course, I could always use a clarification and understanding because you think you know, but when you deal with somebody like you, then you kind of really know. So thank you.

Yeah, you're a little, you're a little understated. I mean, it's like high frequency training. It's like, yeah, you know, it's a little fast and we got to be careful. I mean, this is the big money. This is the serious big money. I mean, like you said, these guys are making fractions of a penny, but you know, millions of times a minute or whatever, or even faster. And you know, this is big money. This is big stress. Like you said, it's the you know, 14 seconds of downtime costs millions of dollars. That's what we're talking. And this is the show.

Yeah, probably tens of millions, about hundreds of millions. And that's a, that's a good segue to Joe, right? So we, so Jay, we, we, we talk often about people that do incident response and the stress beyond incident response, having to deal with people who've been ransomware, having to deal with the CEOs and everybody else is worried about their files. What is a stressful day look like to you and how come you still have hair on your head?

Usually on a stressful day, what does it look like? I mean, I've seen days when people are literally pulling their hair because their trading platforms are not working. And since their trading platforms are not working or there is a downtime between various offices that we have, and we are various offices that we have exchange information in real time. And that has to be like super duper instant. So, and when that happens, we are literally everybody at the end of the day blame security. They're like, Oh, there is something in the middle. So I'm sure security did something. And this is not just for like trading, but I've seen that happen with various other places as well. So when something breaks and people are like, Oh, that must be a security control, or that must be somebody from security who must have done that. And then you're literally pulling your hair trying to understand what these systems were to begin with. Because a lot of times these are super proprietary information or super proprietary systems, right? that you probably haven't seen much before in your previous places, but then you are trying to figure out how is the information flow working, how are radius controls making it work, and who did this change, who probably might not be working anymore, and how do you fix this over a period of time? I remember when we had something called a ransomware, not here, but from previous places, It took us almost like three months to bring everything back online. And it was a nightmare. Making sure that all the control, all the systems are cleaned before they bring them online, making sure that we put the right signatures in place. Do we have the right host interaction and prevention system on all these systems that are coming back online? Do they have like proper access control? When you're expecting people in the office is the office Wi-Fi clean? Are you giving them, are there the right controls in place to make sure that there are no worms on the network?

So I got my next question, Jay. I'm sorry. I'm a kid in a candy store. So I work for a company and we provided kind of a sort of trading platform, right? And one of the things that I was tasked for as the director of operations was to make sure my connections were diverse. And one of the things I've learned about, even in New York City, but this is 15 years ago at least, Even in New York City, you can have 15, 20 different internet providers. But at the end of the day, 18 of them might be all going over the same last mile. So my goal was to always ask the provider, what is your last mile? And I would find out. And one of them, very popular location in New York City, on 10th Avenue, we found out that the last mile majority was their copper or their fiber or their circuits. So you might have, I don't want to say names of companies, but like let's say you had ISPA and ISPB and ISPC, all three of them, even though there were different providers, they went over the last mile, which is provider D. So if that provider got cut, then you lost any of those other providers. So how does somebody like you make sure you have diverse connections to trade without having problems?

Wait, before you answer that, Jay, I got to give a little more history. Because what Adam's talking about is totally right, but it's from like 20 years ago. You know, when a lot of people, and I don't know if you're aware of this, but when a lot of people in trading in the financial markets discovered that, especially, well, obviously in New York, when they discovered that they didn't have that redundancy, that redundancy that they had were really going through the same fiber or through the same wire or anything. That happened for them on 9-11, which was bad. And a lot of people who thought, oh, we're diverse, we're covered, it ended up They weren't. So there was a big change after that. And one of the things that was built was there was actually a financial network that was built specifically for a lot of trading. And it was fiber. Do you know the term dark fiber? You ever hear that one?

Yeah, I've heard that.

Yeah. Okay. It's kind of, I know it's kind of old, but you know, dark fiber just meant you had your own fiber. You weren't leasing it from anywhere. And it was the only way to be sure that your stuff really wasn't redundant. And that gets to what Adam is saying, because the way you've solved it now is really interesting because fiber was the fastest, cleanest thing we had at the time. But how do things look now?

Yes, actually, that is a good segue, right? Because in the past, indeed, fiber was the fastest way. It certainly still is in certain contexts. But when you're talking about Earth in general, right, everybody is fighting for, like you said, internet service providers, redundancies and whatnot, and something breaks in the middle, and that kind of causes things. But now things have changed in the last few years, or at least since the time I've been here. Now we are talking about something called radio towers. Yeah, was that crazy? They told radio. So the thing is, when you are transmitting information or transmitting data from point A to point B, in an ideal world, you put a straight line and that solves the issue. But things are kind of variable here, right? When you're traveling from Chicago to let's say Tennessee, it's not a straight line. You will have like tunnels, water bodies, name it. And then you kind of add more turnarounds for this and that reduces the speed. Even though optic fiber gives you what? Two thirds of the speed of light. And now that you are kind of going around mountains, water bodies and whatnot, that even reduces the speed. So people were like, yeah, what can we do to reduce this speed? Now people are doing radio towers to make sure that information is just going equally fast. It's not the speed of light but at least it is more reliable these days. It is definitely able to like solve a lot of problems that people had in the past.

That is just amazing to me for a couple reasons, because, you know, like when I was in learning telecommunications back in the late 80s, you know, they would talk about, a lot of the engineers and all would talk about the old days of when data transmission was done by towers and stuff. And in fact, I live close to Bell Labs here in New Jersey. where they did some of the first tests with long-range radio. But that had been dropped years ago. And they're like, no, you've got to use fiber. Fiber's fast. It's got less latency. Just boom. And to hear that to get faster, you've gone back to radio is just wild to me. And there's also the security of it. On the fiber, it's very secure. I mean, what are you guys doing about security over the air?

Um, I mean, that is a problem, uh, with, uh, radio towers and radio waves, right? A lot of times, uh, you have to figure out how to encrypt that information because it's not encrypted. And that is a challenge. A lot of people who work within, uh, trick trading are trying to make that work. How can we add encryption to these? How can we make sure that, I mean, a lot of times I think I've seen with one of my colleagues works for a different, uh, company. They mentioned that. There were people who were trying to add like jammers in the middle when there was transmission of data. And because of that, they had like a downtime of at least a couple of hours and that was a huge loss for them.

Yeah, Jay, I'm going to jump in because I get everything you're saying and I've been involved in this, right? I mean, the problem with analog is you can't encrypt analog, but you can use digital over radio waves. When you use digital over radio waves, there's a level of overhead. Even back after 9-11, I was doing FSO, or Free Space Optics. And what I was doing was I was shooting lasers from one building to another building when all the circuits went down, because we had our own level, we had a BLECT, or Building Local Exchange Carrier. And basically, for people who don't know what a BLECT is, it's basically you're a telephone company in your building. You have all the infrastructure, you have these five ESS, or these big switches, and it can do everything else. Even back then, we had DSLAMS or DSL multiplexes. DSL was an old type of service for those who don't know. But these days, people with Elon Musk and his own satellite services, it actually makes a really good backup. It might not be as high speed, but there are new alternatives to everything. And that's one of the good things. But as far as going about jamming and signals, We have the same issue with drones. We have the same issue with terrestrial connections into space. Space is one of the biggest domains now that's being pursued. I just went to a show or a conference called Graycon at Capital Tech University. And that's the big thing. Everybody's using satellites, thousands and thousands of satellites. And what I heard is even Russia moved their satellite into place and went right in between two satellites to stop the connectivity to satellites orbiting the Earth. So now we're in a new domain where people are intercepting or blocking signals in space now.

That's what was interesting because in the financial markets, we said, we're not going to deal with that. We're just going to have private fiber. You know, so everything's tight. You almost, you can even make a case that you don't need to encrypt it. It's so private. And you know, it's hard to disrupt. You got to literally go into a you know, go into a manhole to go and disrupt it or mess with it. But now you're back out in the air. And you're right, Jay. I mean, it can be disrupted. You got to worry about denial of service at a scale you never had to worry about before. You got to obviously worry about, I mean, what about weather? I mean, if you're shooting hundreds of miles, aren't there weather issues with that?

That could be an issue. I mean, the thing is, when you talk about, when you compare, I think that's a very good analogy that you mentioned here, Adam, why not use satellites, right? And that's a very good, that's a good thing to think about. So when you're talking about satellites, you're basically talking to like a satellite that's like miles and miles away. But when you're talking about like these radio towers, right, these are definitely not like thousands of miles away, but at least the way their setup is, we at least try to make sure that there is like some level of redundancy happening because at the end of the day, it's about latency. the more you try to reduce the latency, the better. I can definitely not say how much speed it is adding. From what I remember, because I had a coworker of mine who also was asking me, hey, why don't we use satellites or why not use that? Well, technically, yes, that is doable, but that will definitely add at least a few microseconds. And given that now we are talking about single digit nanoseconds these days, everything adds up to this. And that's why we have like, a lot of these firms have like continuous radio towers, right? Each tower is like at least a few hundred miles away probably. It is not known where they are, but yeah.

Well, that's the funny part Jay, right? Any one of us, I don't care who it is, anybody listening to this, and I hope a lot of people listen to this, I hope the millions of people that listen to this really enjoy this thing, the millions. But anyway, if you're driving down a street, a main street, anywhere you are in New York City, in Chicago, people tend to look up and go, oh, okay, look at all those towers. Oh, those are the microwave towers for my cell phones. People don't realize that these landlords are getting ridiculous amounts of money to rent their roof rights to these companies to place their equipment on there So it might not just be a microwave tower for your cell phone. It might be a tower for, you know, a proprietary connection for a corporation that needs to do a long haul connectivity from the end of one borough to another borough because they're doing it. It might be government. People don't realize how much government stuff is out there. And guess what? It's also on top of bridges. The top of bridges is some of the most highly technical locations and people don't realize how much they're watched. Like government, law enforcement agencies, they watch that stuff. So yeah, there's a lot of infrastructure going up and people don't realize it and thank God they don't know.

But what you said is right, though, that, hey, having like a Starlink satellite or have a satellite that would make us as a security people life much, much easier. You don't have to worry about like data in transit or encryption or things like that. And that is everything that is covered. But hey, that's the trade off a lot of these trading firms have to make sometimes as to security or do we make more money?

All right, so let's get into that because here's why this is interesting. This is kind of the punchline to a lot of this stuff. You've got to do super low latency. You know, low latency stuff's got to get there fast and everything. It's nuts. Huge volumes going very fast. Back when I was doing this, we were shooting for like latencies of like 50 microseconds, you know, and I told that to Jay and he kind of laughed at me. What do you say you're doing that single digit nanoseconds like? And, but the idea was that, you know, because it was so fast, at least by, by those days, there were a lot of things with security we couldn't do. You just couldn't do it. There was no, firewalls couldn't keep up with it. There were no IDS, you know, getting an IDS to keep it up with it was very difficult.

That's always the issue, Joe, right? Cause as things started moving from hardware to software for firewalls, you know, everything was always hardware. It was this chip and that chip and this and that. And then Firewall started inspecting the connections for things like malware and ransomware. It wasn't like that in the original. It was like, oh, is this TCP? Is this UDP? Is this port 554? Is this port 80? You know, things like that. But now it's like, we're looking for like needles in a haystack. But here's the fun, and I know this is not what Jay does, but what Jay does from an infrastructure standpoint is just as important to people performing surgeries when one person is in, I don't know, Germany and they're doing the surgery in New York. It's got to be low latency. It's got to be highly redundant. It's got to be nanoseconds because if that person's operating that robot and they lost a nanosecond or two, they might've snipped somebody's artery instead of cutting out that cancerous thing. I mean, I'm not an expert at that, but that's kind of like a very loose way to say it.

No, no, I think you're right. I think latency indeed is like, it's very critical when it comes to like healthcare, financial markets, and it's just becoming more and more, uh, thanks to like all this development and like AI and people building like GPU clusters, that is becoming more and more of a requirement these days that you try to, um, send all your data that you can into your GPU clusters that you have, or if you have any want to begin with. And actually, funny story that you mentioned, because you said in the past it was all about hardware. It still is about hardware. People are still sticking to hardware because a lot of times people have to create their own FPGAs. People have to set up their own hardware. Because without that hardware, you cannot trade with high frequency.

So you're doing custom-built hardware now.

Yeah, yeah. I didn't have that before.

That's wild.

I don't know Jay Jay has about fifty two hundred million dollars in stock and then video and I'm joking I'm joking. If only.

But indeed, I mean, that's the thing, right? When you have like custom hardware, I think this is something probably you guys have seen it too. So when it comes to custom hardware, and now we're talking about like, what kind of, what kind of software does these systems have? Is this custom built? Is this in-house? Or who is developing this? What kind of platform are they using? And that kind of creates a whole different kind of forms. If it is in-house, Usually we normally consider that as safe, but again, these days with third-party and malicious libraries that is out there. So we never know people who are developers who are using third-party libraries, using their Python or C++ encoding, they could still be using like malicious software and like bringing in something on their FPGAs. That could just cause another havoc. I have seen that happen a lot of times with other places where you are bringing in all these third-party malware and that is causing a huge set of different set of problems.

Well, I've experienced that. I mean, there's been a lot of stuff with Python and a lot of other libraries and stuff lately. But, you know, I never thought about it to the point where you're doing so much custom stuff, you know, down to the hardware and all. And it's like, even with The tremendous amount of money these firms have, they can't rewrite all the software from the bottom up. I mean, they got to be using stuff that's out there, right?

The thing is exactly what you said. These are like, this code was created several years ago, and it is still using like libraries, custom libraries that were previously used several years ago. And this was an example that we saw. that people often push new versions to these libraries and without any, some of these versions will have malware on them. And it's intentional. Though this code has been around for like several years, it is still fine and it's still working, but since it is still using an old library, that is causing a different set of problems. Yeah, we kind of have to look at the code, make sure that, you know, we have the right sort of people who understand this, like static code scanners. And once the code is out there, Even when we deploy things onto things like Kubernetes, how does the container images look like? It's wild out there.

So supply chain compromises, obviously nothing new, right? But I hear stories, you don't have to be government contractors and the CIA and the NSA, but a lot of these companies, what they do, they obfuscate who they are. And what they do is they say to somebody, we need you to write this kind of code, but they don't say we're like a high frequency, you know, trading platform. They're like, yeah, we need this for this and that. So they obfuscate who they are. They go out for to an organization and they get the code written. But meanwhile, it's like a third party company that you never heard of. It's no different from like feeding the president. You know, the president has a people that go out and shop for his food and these they're the nondescript people and they go into supermarkets. They never go to the same supermarkets in the same amount of time. They go there, they buy the food. And then they bring the food back. It's the same thing with some of these companies. They go to different companies. They get code written. They tell them what they want. They don't tell them who it's for. Or they lie who it's for. So this way they can bring the code back, not worrying about somebody in Ukraine, somebody in India, somebody in China. Because a lot of the coding platforms are there. And then they bring it back in. And then they scrub it and clean it. And then they turn around and say, oh, yeah, it's fine. but they make sure never to really go to the same place twice in the same period of time. Does that sound right or am I making that up? No, no, no, you're right.

Don't tell him he's right all the time. That becomes impossible.

I just want to know whether or not I'm making it up.

I thought you were, but no, okay. Fair enough. Well, I'll tell you, this is really interesting. Because, you know, I had thought, and I guess it's my, you know, bias from where I've come from, that, you know, some of the biggest challenges you had would be with speed and latency and even monitoring the traffic or being able to see it or control it. But, man, the software issues, since you have this custom hardware. Now, I'm not going to ask you what you guys run, you know, that would be unprofessional. I know you won't tell me. I'm gonna guess that, you know, besides what people have to deal with now, like if you've got a web application and you're using a lot of Python and they say, oh, the Python's been, there's a compromise in it, or someone got some bad code into Node.js or whatever the hell it is, you know, you've got a lot of stuff to do on your application. But I'm gonna guess that you guys are maybe not, probably not running custom operating systems, but using really tweaked stuff like versions of Linux or something where if you find out something you have has a problem, it ain't exactly like just getting a patch from the vendor and putting it on. It's way more complicated than that.

No, no, no. I mean, so let's think of it like this. So when you are having all these FPGAs and that you're building, they need some kind of an operating system to begin with, right? A lot of times, since you're building it yourself, it's not like the vendor is going to give you, or you can just go and purchase it online. Let's have like a Linux operating system on it or something. A lot of times you have to build the firmware yourself. A lot of times these firmware are made out of C++ because that's the fastest programming. One of the fastest programming there is out there along with C. And you are using all these libraries along with it, right? And that is why a lot of developers, what they do is they build this in-house to support these FPGAs to make it just like more faster and more and more faster. There's always like new, they try to work on launching like new versions of it every now and then. And that is also a security risk within itself, right? Okay, you're tweaking a code. Okay, how does it look like now? What kind of issues does it translate into? In an ideal world for a company, say, A, it would be a web server, they'll launch something new and that introduces a new malware. But in a high-frequency trading firm, what it could mean is you get a malware or something, it fries your chips. Basically, it has happened. Fries the chips? It can, it can. Wow. And basically, yeah.

It overheats it, right? Yeah, it overheats. That's wild. They do that on purpose. They purposely increase the consumption so that the power gets more intense as it gets more intense or fries it. So let me ask you a question, Jay. Do you guys ever feel like, I don't know if we're allowed to even ask this, that you are targets of nation states and other criminal organizations that because they know what's at stake, you know, they can make money off you guys if they were able to, you know, get insider information, poison the well, you What are we whole stuff? Do you think do you guys look out for that?

I mean, we definitely have to keep up with what is happening out there and thanks to a lot of Meetups that that happen. So a lot of times what happens is a lot of these trading firms come together They bring their own security leaders. They sit down together And I like this that I probably have not seen this anywhere else and I'm glad that something like this exists for high-frequency trading firms so a lot of like people like me or like other for other firms they come together and they just kind of share a What are the issues that they're seeing? What are the nation threat actors are they seeing in their environment? I think I've seen that happen with other firms. I think, I'm not going to name, but there was one specific firm here in Chicago that had a really bad day. They got, basically what happened was there was a new version for a specific library and this version was barely there for like a couple of hours. By the time it was known, or the person who owned the library deleted it. But for those two hours, they had malware on their system. And when that happened... On trading systems? Yeah, it was down for like almost six hours or so. Do you guys use ISAC or no? Yes, yes, we do. Yeah.

Yeah, I think it's a fair question to ask. I don't think that's proprietary. You know, when I went to this Capital Tech Raycon, I saw the most amazing thing I've ever seen. There was a space ISAC, but the space ISAC had a real command and control center. They had people staffed what looked like NASA.

Those were always for show. Those are fake.

No, it was not for show. Lots of vendors had that. And usually everyone's like in a closet. By the way, that person might be a guest student, so be nice.

I mean, you're right. I mean, usually, I think for us, it is FSISAC, Financial Service ISAC. So I think we have, from what I think I've seen this with my previous workplace, which was also a financial industry. I think they do a good job of like sharing information. But when I say, and I think that's the limit of it. They share information, you talk to them and they give their own insights. So, hey, we are seeing this for these kinds of financial institutions. So you should do X, Y, and Z. and they'll probably bring some of their customers just to get an idea. But when I say like high-frequency rating firm, right, these are like a little bit more knitted community. It shouldn't be like this, but I guess since the business...

It shouldn't be though. Jay, we know why. Because you guys are trusted. You're like family. And even though you might be competing with each other, it also behooves you to communicate. When Joe and I worked together, Jay, right, we did LSI SAC though. Joe didn't give me really access to it. He wasn't a nice guy when I worked for him. But that being said, the LSISAC would give us information. These threats, these IPs, ingest this, do that. And it was really good for us because we learned a lot from people, even though there might be competitors. It's funny, right? It's like frenemies. We might be competing with each other for work, but those are the attorneys. But the people like us, the security people, we have to work together because if we don't work together, divided, We're nothing. Together, we conquer. So, good stuff.

No, no, no. You're good. You're right. The way, I mean, the good thing is these days with a lot of controls that are out there, like the intrusion detection prevention systems, they kind of get these feeds. If you mention it to them that, hey, we are high-frequency trading firm or we are like a healthcare kind of try to source this information directly from ISAC. I've seen that happen quite a lot of times. So you get this kind of information as to what should be blocked and what shouldn't be right away. But yeah, I mean, when you talk to these kind of coordinated communities, you kind of understand what kind of security strategy do they have. Because they also face the similar challenge, right? Okay, speed versus security. What do we choose in a situation like this? I'm sure you saw this, person A, what have you done when you had something of a similar situation? Or let's say you had a breach, how did you come up or what kind of authorities did you have to notify? Because that is required by a lot of states, a lot of countries these days, depending on where you're operating. So that kind of interaction with them often makes life so, so much easier. It definitely does.

Yeah, well, we've got to give a little context too. Adam was talking about the LS-ISAC, the Legal Services ISAC, Information Sharing and Analysis Center, and that's what it's about, sharing and analysis. The LS-ISAC was a really small one, and it was fairly new and was just getting spun up. But the FS-ISAC has been around for a really long time, and it's probably the biggest and definitely one of the most active. I'll tell you, their summits, I think they're twice a year. Oh, you ever been to one of them? They're a lot of fun. They're a lot of fun.

It's like make it rain.

But there is a tremendous amount of sharing, and they offer a lot of resources. And it sounds like from what you're saying, there's even more sharing going on than there was in the past, because it used to be, when I was in financial services, people in security were pretty tight-lipped, just weren't allowed to talk a lot to our peers or competitors.

I think given that these days every system is interconnected, I mean, with these third party breaches that are happening, supply chain, I think sharing this information is becoming more of a need than, you know, being luxury that was in the past. A very good example was this CrowdStrike that happened. It wasn't a breach, but it was definitely a downtime issue, right? Just because somebody pushed code that they were not supposed to. And that caused a whole different set of havoc for a lot of trading firms as well. I mean, it kind of wrecked a lot of people's lives as well. People were stuck in airports, people were stuck anywhere in the world. But here, when it came to high-frequency trading firm, a lot of our competitors were down, including us to a certain extent. And we were trying to nail this to understand how are other people doing it? How are you kind of dealing with this and talking to them constantly? figuring this out with them based on like concerned response as we are bringing systems back online, because a lot of trading stuff can happen on Windows machines as well. Those are the ones that were mostly impacted during this downtime.

Yeah, I want to say this though, like, you know, I want to be very fair. So much has happened over the last 10 years in our lives, whether there's a last pass compromise or CrowdStrike, you know, bricking of machines. People People are so quick to judge. Everybody's a Monday morning quarterback. I'm probably gonna get threatened after this, but you know, we gotta be fair. You know, it's not a matter of if, it's a matter of when. Every company is probably gonna suffer some kind of outage. AWS, Microsoft, Amazon, Google, they're all gonna suffer outages. This is just the world we live in. The expectation of five nines, which is literally five minutes of downtime the whole entire year. It's a hard thing to maintain. It's like, you know, it's like running through a highway while cars are going a hundred miles an hour. I give a lot of credit to a lot of these companies. I want to give credit. Maybe others won't, but you know, this is, excuse my language, the shit that keeps you up at night. You know, somebody like you that has to make sure that a trader doesn't come to you. Oh, I shouldn't even say this. with something in a threatening manner because you put three seconds of outage and they lost $62 million. This is a hard world where we're very, very critical of people, very critical when they can't do their Sudoku or their Tetris or their online gaming or, you know, everyone's worried about privacy and, you know, and about protecting kids. Yes, we should all be and we can always do a better job. But unless you're on the inside seeing what's going on, everyone wants you to do more things with less money. They want you to be 100% perfect, never have an issue, never go to the bathroom, never get lunch, and always answer every single phone call in the middle of the night. All right, Randolph.

Yeah, I mean, I've seen that happen where you kind of do something. It was one of the previous financial firms I was with. You make a small change. that causes a downtime of five minutes at best, and the person was just let go that very day. And all they were trying to do was just updating the firewall or something, right? It isn't supposed to be super duper critical sometimes when you have the right HA pair in place, but on this day for this person, it was just a really bad day that that didn't work.

Yeah, but I've seen cases where people have gotten fired. In fact, I know of some notable and famous one where the person wasn't so much fired for making a mistake or getting something wrong. He was fired for doing something outside of process and during the hours he wasn't supposed to do it. And, you know, that's one that's hard to defend. It's like, you know, it's like when you get pulled over for speeding, you know, you were going fast, you know, you were breaking the law, but everyone does it.

Wow.

Just tell the judge, you know, something happened to me once where, um, I was doing something for a company and it wasn't with you. Don't worry. And we were working on some circuits and, um, I knew that the circuit blipped and I knew it came back up. What I didn't know was they were monitoring the IP addresses of the both sides, the physical IP addresses. What I didn't know is I took over the organization not more than two months after that, that they weren't monitoring and they were using Microsoft operations manager. Do you remember that mom?

Oh, that's right.

So then it became another name. But what I didn't know was that there was an encapsulation, a tunnel that had RFC 1918 or private IP addresses. And that tunnel, even though the physical connection came right back up, the tunnel didn't. and they weren't monitoring the data passing. That's like going into a candy store and saying there's 6,532 pieces of candy. Who's counting every single piece of candy? Who has the time to literally go in and review the thousands and thousands and thousands of things that are being monitored? You have to hope that the people before you were doing the right job. And yes, it behooves you. and you have a duty to act to make sure everything's being monitored? But how do you know the absence of, the delta, of something that was never there? You know something that's there that's down, you don't know that something was never monitored.

Well, that's why you gotta test. You gotta try these scenarios and see what happens.

No, if you're there two months and you're trying to fix all the wrongs, you can't fix every wrong that fast.

Well, if you're there too much and the thing is a disaster.

I'm an advocate for myself, but go ahead. I mean, let me not get you in trouble, but let me ask you a question. Do you think you're a hundred percent sure of everything that's being monitored and managing perfectly?

I mean, that's like asking, um, you know, is everything perfect in the world now?

Yeah. And what, what are you a white house reporter or something? I mean, that's an impossible question.

I'm making a point. We do the best. We, we, we, we had the integrity to do everything we think is right, but we can't be, Perfect. We're not we're not robots. There's no way to know every single thing Yes, we go in and we test and we do down and we bring things back up. Sometimes we don't even have the capability of Testing every single aspect of something all at the same time. So you might say let's drop this tunnel Let's bring this tunnel up. Let's drop this hardware. Bring the hardware up. Are you gonna drop everything to do fail? Except don't get me wrong. Netflix is pretty good with the chaos monkey I mean, they're really good, but go ahead.

Let me, let me give you a story of like one of the breaches that I saw when it came to one of my clients that I was working with. So what happened was I used to work with a person. So what happened was they were like a developer, but also they, it was a very small organization. So everybody was doing basically everything. So the IT guy was basically just one person or maybe like two at best. So while it was a big business, it was definitely doing some millions of transactions and whatnot. So what happened was this person, he is a developer, he was using, he had every kind of cool detection system on his computer that was supposed to tell him that everything was working fine and everything is working fine. It did, at the end of the day. But what happened was he didn't have any controls when it came to like coding and whatnot, so he actually installed something from a third party library and that kind of introduced the malware on his system. He didn't know about it. Neither did the control on his machine because this was coming from a library, right? This is not something that he downloaded. It's just basically a code running on a container. So basically the malware jumped from his container to his machine locally. And afterwards, since he's a domain admin, he has to log in into a domain controller. So he connected to a domain controller and the malware got onto that domain controller. And given that domain controllers are like really, they replicate each other, right? Every hour or so, depending on the kind of business you operate. This malware basically spread itself on like 10 different domain controllers within a matter of few seconds. And next thing we know, we were getting all these alerts at that time. We're like, oh, there is an alert coming in. There is a malicious PowerShell script that is trying to do something. Okay. What is that? And when everybody connects to a domain controller at the end of the day, when they have to log in into a company system, Everybody who was logging in got that malware and they had like a small note, a text pad on their laptop saying that, hey, your files are getting intercepted. And they were like, what is happening? We then figured out at the end of the day that this was coming from like something that this person just basically used as a library at the end of the day. So my point being is, to your question or to your point, that yes, it's hard to keep track of what is happening in this case. And especially if you don't have the right controls in place, you will always miss something. Always and always and always. Yeah.

Well, I can tell you though, I mean, hearing that story and you know, I don't mean to beat anyone up or anything, but you know, I don't want to say it was predictable, but I'm not shocked that given, you know, the lack of controls there, that something happened. I mean, those are, you know, sometimes we see things like, you know, it's so subtle, this is tough to find and all these things. And then sometimes we see things where like, you know, we're like, this is an accident waiting to happen. You know?

Yeah, everybody should have PAM. I get it. Privilege escalation. You should never log in with the domain. You should never log into the domain admin. You should always privilege escalate to that domain admin. You should be going through a privileged access management. You should be using detection and prevention. There's a million things, but we don't always get that power to purchase those items. Sometimes people, it's about, you know, it's risk versus reward. Look, I'm going to piss Joe off. Watch this, Jay. I figured one of the best things you can have in place was to have a packet inspection, North, South, East, West.

It's too bad we're not drinking because that's a shot right there. If Adam mentions PISM or packet capture, yeah.

So what I wanted to do was we were working for, I don't want to say the organization, and they had very proprietary and intellectual property. And the problem is, yeah, there's also encapsulation and encryption, but you want to know North-South. North-South is always the best way to go coming in. But you also want to know East-West. So for those who don't know, North-South is coming from your higher level internet services down to your organization. East-West is going from one network to another network where people move.

Within your own...

Within your own domain, your own domain, your own network. So if you're an organization, you're a corporation, East-West is going from your legal computers maybe to your human resource computers, going to your financial computers. But let me just tell you this, Jay. I'm a novice. I'm not smart. We were doing a red team somewhere and a purple team. And what I did was I used a service account and I logged on to the red team's computer and I took their files. And these guys are 50 times smarter than I am. These people are 50 times smarter than me. But at the end of the day, you can't have eyes on everything. They saw a service account. They didn't know type one versus type two. Who's sitting there watching that on your laptop? I took their files. I took their hashes. I might've changed some things. Somebody might've gotten mad at me. I've said this before. I had to go put their files back in place. They didn't know it was me. Someone told them it was me, whatever. But the point I'm making is you don't have to be smart, but you can't blame these people that might be 50 times better than you because you can't have every single control in place. It's impossible. It's like watching surveillance cameras. You have a hundred surveillance cameras in front of you, you're not going to capture everything.

Okay, for the record, Adam was kind of cheating because one of the reasons they weren't looking for this was because he wasn't supposed to be doing that.

Maybe they should have been looking, Joe. They should have been looking anyway. Every network is a hostile network. If you're doing reconnaissance, you got to worry about somebody coming after you.

I know, I think that's a good example. I mean, you can't have controls, indeed, but at least what you can do is have visibility, right, of some sort.

Well, you said civility?

Visibility.

Oh, I thought you said civility, because I don't have any civility either. I'm not civil with anybody. Go ahead. Sorry, Jay.

No, no. So this situation that I told you, right, where I saw ransomware happening right in front of my eyes just because of the library, the client never had a SIM to begin with. And that made things really, really worse. Basically, at that time, they were only relying on what the intrusion detection prevention system would give them. And that was basically just watching it as a watcher. not doing anything else. And since everything was just falling apart, we had to just go into these systems, look at these systems manually because they were storing them to a certain extent. A lot of times what was worse was a lot of times what the backup team was doing was they were backing up these systems on the system itself. So when the ransomware hit, how do we recover the system? Well, it's the backup is on the system. Okay. How do you get that? I don't know. So basically that's why when this kind of things happen, it takes months. For us, it took three months because we barely could see backups anywhere.

Look, this is a tough job. Security is tough. And you can find a lot of these subtle things, and it is difficult to sort it out. But you don't have to be a genius to know you shouldn't be packing, putting the backups on the same server you're packing on. Seriously.

So these are all tabletop exercises. It's hot, cold, cold, warm, warm, hot, hot, cold, warm, hot, hot, warm. So when you're backing up, are you backing up Electronically are you backing up or you're backing up with traditional tapes and stuff? If you're backing up electronically, the systems are connected. Notice what you can do air gap the system, you know, like, you know, shoot the signal one way, you know, asymmetrically, you're always going to have some kind of vulnerability. And this is what I've seen over and over again. People like, oh, shit. or our warm site is corrupted because it moved, malware moved to where we were backing everything up once a week to over there. I mean, at least back in the old days with that, you know, those tapes, there was always a good tape somewhere because not every tape got infected because it was never there. But as you start backing up online, as you start backing up via, you know, some kind of connectivity, there always runs the risk that that malware will get there. Even in air gap systems, malware gets into systems. So how can you, yeah, even in air gap systems, you're getting screwed. Somebody's moving from a Bluetooth phone to somebody else's system. Oh shit, let's hop on here. Let's hop on there, you know?

I know, but you gotta try it. Like I say, some of these things are subtly complicated, but some of these things are gimmies too.

Joe, I'm not saying give up. I'm just saying that If you throw tens of millions to hundreds of millions of dollars, even our own government gets it wrong sometimes, not because they're horrible. It's because there's so many moving parts. People move from traditional internet into secured networks because they buried themselves all the way through and they found the tunnel. They look for the light. Oh, there it is. You know, it's hard. It's such a hard world. We're all interconnected. My brain is probably connected to my computer now.

Well, Adam, you've been such a bundle of joy today that I'm going to ask Jay to give us the last word as we get to the end here. Don't say but, but, but, and say something that's going to send me to the car.

He's never hired me, so it's good.

I mean, The way to look at this is, again, at the end of the day, it's a breach that is going to happen. And when that happens, and you were talking about this, right, that companies are trying to do things for free or at the minimal cost as much as they can. But that significantly changes when there is a breach. The client I was working for, the security budget became 10x the year after. Get all the controls that you can, get Splunk, get everything that you can, that is the most expensive.

Microsegmentation, IOT, DLP, you know, IDS.

Yeah, we've talked about that, and we've talked about that before, saying like, you know, sometimes after a company gets breached, they become world-class, you know, because they just go out and get everything, you never know.

Jay, check this out, Jay, you're on LinkedIn a lot, right?

Okay.

What's one of the things that you see on LinkedIn It's really disturbing. I'm trying to lead you to this. I'm going to say it. I see so many people begging for jobs. There's no cybersecurity jobs out there. How can there be no cybersecurity jobs out there when, when people, when these places are getting breached and everything else. And here's one thing that's really upsetting. And I'm a sucker. I've seen more than one person online. You see me for the last eight months. Look at my feed. I've been looking for a job. I'm becoming homeless. Can people send me $20, $30? Because I'm going to get thrown out of my home. How is it possible that all these cybersecurity people, some people that really came from really good companies, can't get jobs? So the point I'm making is, we talk about all these companies putting in these security practices in place. But nobody's hiring.

No, that is, that is true. And thanks to all these vendors out there that are like, I mean, I think that's kind of an innovation in a way, if you think about it, that are introducing the quote unquote AI or machine learning onto their toolings. A lot of, there is a bit of a misconception that, Hey, if I get this tool that has this new AI capabilities, that is going to just cut down the workforce that I have by like 1% or 2%.

I think we have seen that happen with a lot of firms out there.

And this is the big brunt of it is something that a lot of graduates are coming out these days. Because since you basically don't have much of an experience. And since a lot of these firms are automatically doing like some level of entry level job, because of these new AI capabilities, people are not able to get a job because of that. And I think I've seen that happen more and more. with security these days. I mean, yes, indeed. I mean, I've seen a lot of even certification-based organizations like IC2 and whatnot who say, yes, there is a lot of demand for security. But if I were to indeed, like you said, go out and actually see LinkedIn, yeah, it breaks my heart. It kind of makes me scared as well. Okay, so say if I have to look for a job tomorrow, how is that going to look like for me? Even though I have, I can basically do a lot of these stuff. A lot of people are talking about these days, GPU clusters, AI and whatnot. I've been part of that, but still it's scary out there.

Wow. Well, both of you seem determined to take us out on a downer here.

So I'm going to finish this joke.

Oh God. Yeah. What's what's in that. So I'm going to have to take matters into my own hands and say, at least Jay, you mentioned AI a couple times, and that's great for the SEO. So thank you so much. That's good for us. You know, that's awesome. And we can come back and do a whole other show on that. Yeah, the hiring situation is crazy.

are hiring a lot. So I'm sure they might be hiring a lot of security folks as well because these days markets are super volatile. So that's what happens when the markets are super volatile. High frequency trading firms make a lot of money because everybody is buying everybody selling and based on that they're generating revenue on top of that. So even though I say that the future is bleak, at least some part of it isn't fully bleak. So there is revenue. If there is revenue, that means there will be hiring and there will be like fully equipped.

And we'll be posting Jay's email address on our website so you can contact him.

I don't think it's that bleak. You know, I mean, I know I said a lot of things, guys. I said a lot of things and I know there's a lot going on, but yes, you know, You know, as people might have known, we've done a lot of shows with recruiters and we speak to recruiters a lot. And the recruiters are saying that, yes, we're on our third year of hiring issues. But I do think there's a light at the end of the tunnel. I do think there's a lot of move to technology that we weren't doing before. Ironically, my son, you know, he had this mining rig and I said, let me go sell the mining rig. It actually got delivered today. We no longer do mining because of proof of stake versus proof of work. But what we didn't know is that these mining rigs are being converted into AI farms. So people are taking the same NVIDIA, even the older cards are 3090s and stuff. And they're using these NVIDIA cards to create their own AI. So yes, while some things traditionally I have not been hiring for, there is still a lot of AI to learn. And even though they say AI are taking jobs, there's still a lot of people, I mean, The big people that were hiring for AI, I think it was Mark Zuckerberg, he was paying them like sports recruiters, sports salaries. You know, 30 million in five years, you know. It's not a lot.

Everyone, you're talking about like five or six guys.

But everybody wants to become that major league baseball player, that NFL football player. So I'm saying you could still have that. You couldn't do that 10 years ago, Joe. You couldn't be paid like a sports player. But nowadays, you can. So my point I'm making is there is some positivity there. Become a Division 1 sportsperson in AI and then move into the big leagues.

There you go. All right. Well, we're going to have to leave it at that before you say anything depressing again. But Jay, thanks so much for joining. This did not go where I thought it was going to go with everything that's going on. So it shows you what I know. So thanks for coming on and giving us some new info.

Yeah, the pleasure is all mine. Thanks for having me. I'm really glad I was able to talk to you guys, really able to share what I know. I'm sure you guys definitely know still a lot more. Yeah, at least I'm glad I'm able to share the ins and outs of like how high frequency trading firms work and at least based on my experiences with breaches and whatnot, I'm able to share and at least share my two cents that a lot of people may not have seen before. So I'm glad and I'm thankful. Yeah, appreciate that.

Thank you for coming on. And thanks everyone for watching and listening. We love our audience.

Like, subscribe, comment,