🌮 ✍🏽 👾 SOC on a remote site. What is worth thinking about? 🕵️ 🏇🏼 🖖🏾

One SOC analyst, when they decided to transfer him to work from home, asked a sacramental question: “I live in odnushka with my girlfriend and her cat, which I (as a cat, not a girl) are allergic to. At the same time, we have only one table, at which we eat, and in extra-ordinary times, my girl lays out drawings on it, she is a fashion designer-designer of clothes. And where do you order me to place 3 monitors that I was allowed to temporarily pick up from work? ” Well, since this is only one of the many questions that arise when transferring SOC (Security Operations Center) employees to a remote site, I decided to share our experience; especially consideringthat right now for one of our customers we’re just designing a cybersecurity monitoring center (SOC) from scratch and he decided to change his shoes on the go and asked to take into account the possibility of work of his analysts and incident investigation specialists from home in the project.

In a pandemic, many companies relocated or were required to transfer part of their employees to remote work. The bowl of this and SOC experts did not pass. However, it should be noted right away that domestic regulators, by their requirements, prohibited specialists from outsourcing SOCs that provide services to their customers from working from home, since one of the conditions for obtaining a license for IS monitoring (and commercial SOC is a licensed type of activity) was not only to indicate the address of the provision services, but also the certification of its information system, which also makes it impossible for an emergency transition to a remote site (and not an emergency one either). True, in our state, legal IS nihilism and a temporary refusal to conduct inspections,you can ignore this nuance - the requirements of business and the preservation of people's lives and health are much more important in this case. Especially considering the fact that cybercriminals did not reduce their activity. Moreover, they raised the theme of coronavirus to the flag and began to attack users at home, who were left to their own devices and became an even weaker link than usual. Therefore, the topic of SOC (or information security monitoring) on a remote site becomes quite relevant for many companies. Well, since Cisco helped build and audit a considerable number of SOCs, and our own SOC has been working on the “virtual SOC” model for more than 10 years, we have accumulated a number of tips that we would like to share.they raised the theme of coronavirus to the flag and began to attack users at home, who were left to their own devices and became an even weaker link than usual. Therefore, the topic of SOC (or information security monitoring) on a remote site becomes quite relevant for many companies. Well, since Cisco helped build and audit a considerable number of SOCs, and our own SOC has been working on the “virtual SOC” model for more than 10 years, we have accumulated a number of tips that we would like to share.they raised the theme of coronavirus to the flag and began to attack users at home, who were left to their own devices and became an even weaker link than usual. Therefore, the topic of SOC (or information security monitoring) on a remote site becomes quite relevant for many companies. Well, since Cisco helped build and audit a considerable number of SOCs, and our own SOC has been working on the “virtual SOC” model for more than 10 years, we have accumulated a number of tips that we would like to share.and our own SOC has been working on the “virtual SOC” model for more than 10 years, we have accumulated a number of tips that we would like to share.and our own SOC has been working on the “virtual SOC” model for more than 10 years, we have accumulated a number of tips that we would like to share.

But I would like to start not with the technological features of remote monitoring and incident investigation, we will talk about them below, but with what we often forget. The main problem in transferring part or all of SOC analysts to telecommuting from home is related to the human factor. Even if you have built all the processes, and the toolkit allows you to remotely connect to the SIEM or SOAR console and also remotely collect artifacts from computers of remote users, then it is people who can become the weakest link in the SOC that has left its established places.

Mindfulness and stress

According to studies, the transfer to homework leads to a decrease in productivity by 15% (a person can spend more time on performing official duties, but is it effective?). Stress and discomfort (if children are running around and a neighbor suddenly decided to make repairs), as well as inability to concentrate when a wife cooks borsch or a friend practices yoga in the same room, lead to a decrease in attentiveness. And the dream that beckons so? While working in the office, you can still deal with him somehow, but being at home, seeing a warm and alluring bed with a loved one sleeping there, it is much more difficult. And this despite the fact that the performance of any person at night falls simply disastrously.

Therefore, it becomes very important to measure the effectiveness of SOC specialists. Remote work is relaxing and not everyone is ready for it. SOC employees need to change their behavior when working remotely. And their managers need to monitor the performance of each analyst and when they go beyond the boundary values, respond in a timely manner. And this is not about the banal “average time of taking an incident to work” or “average time of response / localization of an incident”, but about measuring each stage or task within the framework of the developed playbook. Suppose you have a playbook for analyzing suspicious user activity and the average response time to it (from the moment a signal is received in SIEM and to freezing an account in AD, placing a node in quarantine VLAN and searching for other nodes and users,which could be related to the victim) it takes you about 28 minutes. You look at statistics on remotely worked incidents and you see that this indicator jumps a little in the range from 19 to 37 minutes, but on average it is the same 28 minutes that it was before the change in the operating mode of analysts. Everything seems to be OK.

But if you had the opportunity to monitor all the individual steps / tasks of the playbook, then you would see that instead of the traditional 1-3 minutes for identifying indicators, 1-3 minutes for checking them in the TI platform, 1-2 minutes for making a decision, 3 minutes to freeze the account, etc., your analyst immediately after receiving the alarm puts the user and his site in quarantine, or even without checking sends the incident to the next level, to L2 analysts. And, for about 9-10 minutes, it’s not at all clear what the analyst was doing. Either he went to the kitchen to pour himself coffee, or he decided to watch the news on TV, or maybe he just went for a walk, to the balcony. But your incident specific metrics are met. Therefore, when switching to remote work, it is necessary to revise the existing metrics for assessing the effectiveness of SOC.

In the described case, by the way, another solution could be the introduction of automation tools that eliminate most of the manual tasks and automate what is written in the playbook. But this could be done regardless of the remote work - SOAR platforms are just designed to automate the work of SOC analysts and not only accelerate the process of investigation and response to incidents, but also have a tool to measure the effectiveness of work with them. Therefore, by the way, Cisco has developed a new and free information security management platform - Cisco SecureX, the task of which is to automate the management of many Cisco solutions (and hundreds more from other manufacturers), including the incident response process.

Teamwork

SOC is almost always teamwork. In a typical SOC, its specialists work side by side in cramped spaces and when they are forced to switch to remote work, difficulties immediately arise for which the SOC is often not ready. By the way, this is also facilitated by the requirements of the legislation, which consider the SOC as a licensed activity in a certain room, as I mentioned above. The concept of virtual or mobile SOCs in Russia is not very accepted, especially when providing services for monitoring information security and incident response (for your own purposes, of course, you can build SOCs using any of their available architectures). Remote teamwork is a new challenge that you still need to get used to. What to look for in this aspect of the SOC.

Schedule

Review and possibly revise your shift schedule. Each analyst must know not only his role and his schedule of work, but also the role and schedule of other team members in order to be able to contact the right specialist or replace him if, for some reason, he hasn’t come to work (in addition to admission to the hospital, you can simply get under administrative arrest for 15 days for violating the rules of mandatory self-isolation / classic oxymoron / when you just went for a walk on a nearby pond). By the way, if your analyst is working in a country house and he suddenly had electricity or the Internet cut off, then of course he does not need to record a virtual “truancy,” but he must be prepared to replace it with the recovery time of his remote workstation.But in addition to the above reasons, there may be a lot of things that should be taken into account - the neighbors flooded, a fire, it is urgent to take a loved one to the hospital, etc. And all this must be taken into account first of all for analysts of the first and second line, for whom a permanent stay at the workplace is the most relevant.

Mentoring

Under conditions of a fairly high rotation of personnel at the lower levels of the SOC hierarchy, mentoring of more experienced colleagues over newcomers is widespread in them. They help with advice, look after, support. And how to do it on a remote site? How to help a beginner left to his own devices? The answer is simple - correct and constant communication !!!

Communications plan

Communications in remote work become the most important elements of its effectiveness. Moreover, communications are not only part of the investigation of a particular incident, but also “just like that”. A couple of times a day to gather for 15 minutes and exchange views, ideas; support each other in a stressful situation (and whoever says anything, but the current situation is precisely the stress that negatively affects our capabilities and abilities). Our SOC analysts have always had this, but ordinary employees have introduced the practice of “virtual coffee breaks” only recently. As experience shows, many people really lack communication with colleagues; and even introverted analysts are no exception.

If you already have a communications plan, then you should analyze it and, most likely, review it. It should include at least:

, , — e-mail , . /wiki SOC, ( ).
. , , , . , ? ? ( , IVR), , ?
, , , , « » « ». . Cisco, , Cisco Webex Teams, , . , .
- , . , , SOC, . grid card, , .
FAQ - RACI , / /use case.
, war room CSIRT , . , , , . « ?», . .
. , , SOC, , -? ? ? , , — .

?

During an investigation, the most valuable resource is time. Do not waste it. Remote investigators can duplicate each other’s work. They can repeat the same discussions in different groups and means of communication. Messengers, which are often used to communicate as part of an incident, for example, Whatsapp, are not very suitable for this task, since it is very difficult to search and work with documents and artifacts there; especially when in one chat you have too many different incidents and the confusion between them begins. Creating separate chats for each incident could solve the problem, but I cannot call this path convenient. Still, messengers are not designed for this task. In addition, it will be difficult for you to separate service chats from personal ones,and you will also become dependent on external management servers, the operation of which may be disrupted if any government agency once again begins to struggle with the conditional Telegram, or if the messenger manufacturer introduces a ban on the simultaneous posting to several groups / chats, fighting with fake news.

For effective communication within the framework of the incident, the solution you use should be able to organize chats, file sharing and remote access to the desktop. For example, using Webex Teams, you can create a separate space for each incident in it, where you can not only collect all the necessary evidence, but also communicate with all the participants involved in the incident. And due to the ability to write chatbots and integrate Webex Teams with security tools, you can immediately check and enrich the received artifacts.

If you do not use Webex Teams, then a similar idea can be implemented, for example, on the following link: a separate Slack channel for a specific incident (and a common communication channel, which can be used as a kind of index for all incidents), the Zoom room for discussion (you there must be a paid account for this and if you are not confused by the constant IB problems of this service) and the Confluence page for collecting notes. Then, the incident summary information is entered into the IRP, which can be either familiar to many Jira or specialized incident management solutions.

Whiteboarding for brainstorming

Do you have a brainstorming board in your SOC? I think it has. For remote work, you may need a Whiteboard - such a function is available in various means of communication. You can remotely draw and share ideas with colleagues. And if you used kanban boards to control team tasks, then you will need them on the remote - use, for example, Trello if you do not have a corporate solution. But when using cloud-based solutions for teamwork, do not forget to configure the appropriate access rights so that outsiders do not learn about your security problems and cannot intervene in the investigation and response process.

Remote room

You decided that SOC analysts would work remotely. And you know how to do this not only technically, but also psychologically and organizationally. But, did you ask yourself, do your SOC analysts have their own apartment or live in a hostel? Who are your first line analysts? High-class specialists working for many years at the SOC and earning their own penthouse on the Sparrow Hills? Or are these more recently students living with mom, dad and younger siblings? In your SOC, you can provide them with a workplace and three monitors to monitor “the needle in the haystack.” Does your analyst have a place for 3 or at least two monitors at home? And if his brother cuts a meter away from him in the PlayStation and distracts him with the sounds of “burning rubber” or “dying zombies”? If the analyst is unable to work from home due to lack of space,You will need to make changes to the composition and schedule of shifts.

It is also important to ensure the convenience and comfort of work for analysts. In the corporate SOC, you can spend a lot of effort and resources to choose the right lighting, noise isolation, reverb, air conditioning, comfortable chairs, etc. But at home often, alas, we do not possess such luxuries. And therefore, we need to more closely monitor how the performance of analysts has changed before (if, of course, you collected them) and after leaving for a remote site. And after analyzing them, draw the appropriate conclusions. Otherwise, the remote SOC will turn into a “pumpkin” - it seems to be there, but it does not solve its problem, since analysts are simply unable to work in their own apartments.

And do not forget that the physical security of the room in which sufficiently sensitive information circulates is also important.

SOC Architecture

In a pandemic, your security events will increase. Firstly, you will have to actively monitor your VPN cluster or remote access clusters through which users will connect to applications and data within the corporate network or who will “forward” user traffic to external cloud services. Secondly, you may need to monitor cloud environments if the IT service has decided to temporarily or already permanently transfer the processing of some data and some applications to the clouds. And finally, you have a growing zoo of various systems that are at home with users who will start generating security events and which will need to be analyzed. But that's not all.

Remote connection to SOC

SIEM, IRP / SOAR, ticketing, TI platform ... Do they have the ability to connect to them from a remote console over a secure connection (how lucky are those who use cloud SOCs / SIEMs now, where is the built-in function)? If not, then you need to think over the option with remote access via RDP / VDI or other connection methods forwarded via VPN (in this place it’s time to think about the possibility of implementing a mobile or virtual SOC). In the end, you can have a special virtual machine, LiveCD or LiveUSB for working with SOC tools; although we almost never saw the last options - it is rather done for users to work from their home computers, when it is necessary to separate corporate applications and home ones with which your relatives work.

In our opinion, VDI (but also via VPN) would be the best option from the point of view of access control and working with sensitive information, but direct access if properly configured is also a completely working option. In both cases, it is necessary not only to use multi-factor authentication, but also be sure to check the connecting PC analytics for compliance with the requirements for information security (installed patches, password protection settings, the presence of antivirus or EDR, etc.). If you have a Cisco ASA on the perimeter or a Cisco ISE is deployed inside the network, then you can arrange such a check on them. We recommend that you disable split tunneling on computers connecting to the corporate SOC. It will be much safer; and for ordinary users on a remote site too.

And do not forget to clarify whether changes in the ACL of the border equipment on the company side are required to access the holy of holies of IB? What is the general procedure for making changes to the ACL? Will it take a long approval process, and even requiring a handwritten signature on the application? Now it would be a good idea to revise all your policies and instructions with a view to their working capacity in the conditions of impossibility to collect signatures and run between bosses. And even if you have an electronic document management system, how well is it adapted to the prompt resolution of issues?

Remote Investigation on Remote PCs

Investigators are accustomed to the fact that they can come to the employee involved in the incident and remove all the logs / memory dump / and other artifacts from his computer. But how to do it remotely? Someone uses Remote Admin, familiar to IT specialists, someone RDP, someone Skype for Business or other tools. The main thing is not to forget to correctly configure the protective functionality of these tools and agree with employees about the authentication procedure for IT / IS employees connecting remotely (well, do not forget about the legal aspects, as will be discussed later). Someone uses free GRR, osquery, built-in Sysmon or Autorun Utility, or commercial EnCase (users of Cisco AMP for Endpoints do not need to worry - they have a tool like Cisco Orbital that allows remote investigation).In some cases, you can teach users to run the necessary script, which itself will collect the necessary evidence and securely transfer it to the SOC for analysis. It is important for you to decide for yourself, these tools must be installed in advance (for corporate devices there is no problem in preinstalling them on the corporate OS image with all applications) or they need to be installed in case of any problems. In the latter case, however, there is a risk that by doing so we will let the attacker know that his activity has been detected and that he can “leave” his victim, simultaneously erasing all traces left by himself. This is a difficult question, which, of course, is best solved “ashore”, and not in the process of investigating a remote incident.who himself will collect the necessary evidence and securely transfer it to the SOC for analysis. It is important for you to decide for yourself, these tools must be installed in advance (for corporate devices there is no problem in preinstalling them on the corporate OS image with all applications) or they need to be installed in case of any problems. In the latter case, however, there is a risk that by doing so we will let the attacker know that his activity has been detected and that he can “leave” his victim, simultaneously erasing all traces left by himself. This is a difficult question, which, of course, is best solved “ashore”, and not in the process of investigating a remote incident.who himself will collect the necessary evidence and securely transfer it to the SOC for analysis. It is important for you to decide for yourself, these tools must be installed in advance (for corporate devices there is no problem in preinstalling them on the corporate OS image with all applications) or they need to be installed in case of any problems. In the latter case, however, there is a risk that by doing so we will let the attacker know that his activity has been detected and that he can “leave” his victim, simultaneously erasing all traces left by himself. This is a difficult question, which, of course, is best solved “ashore”, and not in the process of investigating a remote incident.these tools must be installed in advance (for corporate devices there is no problem in preinstalling them on the corporate OS image with all applications) or they must be installed in case of any problems. In the latter case, however, there is a risk that by doing so we will let the attacker know that his activity has been detected and that he can “leave” his victim, simultaneously erasing all traces left by himself. This is a difficult question, which, of course, is best solved “ashore”, and not in the process of investigating a remote incident.these tools must be installed in advance (for corporate devices there is no problem in preinstalling them on the corporate OS image with all applications) or they must be installed in case of any problems. In the latter case, however, there is a risk that by doing so we will let the attacker know that his activity has been detected and that he can “leave” his victim, simultaneously erasing all traces left by himself. This is a difficult question, which, of course, is best solved “ashore”, and not in the process of investigating a remote incident.Of course, it is better to solve “ashore,” rather than in the process of investigating a remote incident.Of course, it is better to solve “ashore,” rather than in the process of investigating a remote incident.

I think that I will return with separate material on remote investigation and evidence collection, but for now I’ll just mention a number of questions that you need to be ready to answer in a remote investigation:

How to access home, mobile and corporate remote devices? In the zoo condition of home devices, this question is not as simple as it seems. This is especially true for mobile devices, which may also require investigation measures.
How to access the logs and data of the information security and what general logs and artifacts on remote devices do we need? How to collect them selectively?
Do you need administrative remote access to investigate and respond, for example, service accounts, sudo, SSH keys? Do I need to make changes to the ACLs on home routers for remote access (and if they are provided and managed by the service provider)?
Is it necessary to amend the whitelists of applications for the operation of investigation and evidence tools?
How to remotely install tools for investigation and evidence collection?
How to track when powered off remote devices are turned on for investigation?
How will you communicate with the victim?

The answers to these questions are highly dependent on the existing infrastructure, the availability of corporate standards for the devices used, system and application software, as well as the right for employees to use personal devices to work from home.

Outsourcing Service Providers

Perhaps in a pandemic, you decide to temporarily use the services of an outsourced SOC and an MDR provider. Moreover, you can get discounts from them now :-) But it is much more important to understand how much your current outsourcing contract or contracts take into account the remote work of your analysts? MSSP / MDR / TIP / CERT / FinCERT / GosSOPKA accept messages from your employees, from their personal phones or e-mail addresses? Do they block access not from corporate IP addresses? Is it possible to connect to dashboards from external providers not through a corporate VPN, using split tunneling?

Legal requirements

I already wrote about the impossibility of the remote operation of a commercial SOC due to the limitations of the FSTEC. But there are a number of topics that must be remembered when remotely operating SOC. Do you know a fresh joke? “My children study at home. I ask the school administration to donate money for new curtains, painting walls and furniture repairs ”:-) So in a number of countries there is a practice where employees begin to demand compensation from the employer for using the apartment and home computer as work tools. I think that in Russia this does not threaten us yet, but the example itself is quite indicative.

It is unlikely that SOC employees are offered to perform their official duties from personal computers (although ....), but ordinary employees can easily connect to corporate resources from home devices. And therefore, a legitimate question arises - on what legal basis does the employer gain access to the employee’s remote and personal computer when conducting investigations, collecting artifacts, etc.? Do you have a clause on the company's right to monitor an employee in an employment contract or in an additional agreement to it? Is there a detailed regulation on what is possible and what cannot be done within the framework of such monitoring? If not, then you are violating applicable law and the employee has the full right to send you when you try to access your personal device. The same applies to the installation of DLP agents or time tracking tools on home computers.

And do not forget that your SOC may fall within the scope of some international information security management standard (for example, in Russia several SOCs have already passed ISO 27001 certification). Consider how you will be audited with your remote SOC? You will not lead auditors to the apartments of employees. Or will you? ..

Conclusion

These are sketches about what we have to deal with when designing, auditing and developing plans for improving IS monitoring centers, including in Russia and the CIS countries. This is not to say that all these tips are not feasible or require significant financial, time and human costs. Many of these are easy to implement. The main thing is not to let these issues slip on the brakes, since ignoring them can lead not only to increase the time for investigating incidents during remote work, but even to skipping them. Also, do not think that the current pandemic is a temporary phenomenon and soon everything will work out. If the organization seriously thinks about business continuity and knows how to calculate the situation several steps forward, then we will understand that COVID-19 is just a sign; a sign that the situation may be repeated next winter.Therefore, "sledges must be prepared in the summer." In any case, the tips described above will not be superfluous in ordinary SOC life either.

Threat. And by the way, do not forget to sanitize the main SOC premises before analysts return there.

SOC on a remote site. What is worth thinking about?