Thoughts on security

PCAPs and Wiretaps

Sometimes when you investigate a problem, you come up with some interesting results. For instance, take the time when one of my clients called and asked me to investigate a mysterious router issue at a remote site in southern WV. It started with an outage, and I called and talked the user through resetting the router and gave him the admin password so he could set some things back for us. I know it isn't a good practice, but with the client's permission it saved us three hours of drive time for a five minute phone call.

About a week later, the client called again and said they were having problems again and asked if I could go resolve the issue. When I got onsite, I started a packet capture and then started looking at what I was getting. I talked to the user and asked if he had written down the password I gave him the other day so I didn't have to look it up. I thought it was strange that he denied ever hearing it, so I dropped it. I looked it up and collected the logs, then located each piece of hardware connected to the network. I started looking at the traffic coming out to see if anything stood out, then went up and talked to the guys in the other trailer.

They were running the same setup with the same equipment on the same satellite about a half mile away. As I was copying the configuration file to compare, the foreman asked if I found the problem. He said he thought that the user I talked to was doing it on purpose. I asked why he thought so and he told me that his father owned the company paying the bills so he gave him this job, but he wanted to stay in his room instead of coming to the field every day. I thought about what he said about not having the password, so I went outside and played the recording for that call back, and sure enough, there was me giving it to him and him saying hold up, he had to write it down. It is legal in WV to record a call as long as one party is aware it is happening, and in this case, it was well worth it.

I went up to the other trailer, and as I was sitting there, the internet went out. I just happened to be capturing the traffic, so I sat down and started looking at it. I found that there had been something that jumped onto the network and started sending out LLMNR (Link Local Multicast Name Resolution) packets flooding the network and causing it to crash somehow. Naturally, since the internet wasn't working, the user had to pack up and go to the room to finish the day out. The thing was, I was there this time and sitting right beside the router when he left. To this day I will never figure out his motivation, but I assume it was because he figured since his dad was writing the checks it didn't matter if I caught him. Or maybe he actually thought he was smarter than me, or that I didn't record my calls, or whatever. Either way when he left the internet started working again, and stayed that way until I left.

I went back and talked to the foreman and told him what I suspected and could prove and asked what he wanted me to do about it. He told me to drop it since the user was leaving the following Monday. Surprisingly, the internet never had a problem in that location after that.

Lessons Learned

There is such a thing as a malicious insider

Don't give out Admin credentials over the phone unless you are prepared to accept that risk and make sure you document it.