Not understanding IT security can cost audiovisual designers and installers dearly. Here's what you need to know – and why.
Text:/ Derek Powell
Let’s be honest: IT Security is not a subject many audiovisual professionals know a lot about. In fact, it is a subject many of us have avoided discussing in the past, because if we even raised the topic, we were likely to be asked awkward questions that could get us into trouble. Sadly, many of those questions crop up at installation or commissioning time when we ask the IT department for an IP number to connect our control system or encoder or digital signage player to their network.
The bad news is that we can no longer avoid these conversations. And that’s just as well, because waiting until installation before discussing IT security is really bad for business.
The good news is that now AV equipment manufacturers are finally building gear that answers all those pesky security questions. Now what we have to do is learn to understand what those questions are all about.
WHY SECURITY MATTERS & THE COST
Nearly every piece of audiovisual equipment you can think of (except maybe some mics and speakers) now comes with a LAN socket on the back. Often, these network connections are the only way you can configure certain parameters and they are essential to control and monitor our increasingly complex designs. We used to request an IP number by simply saying: “It’s just a network appliance – no need for security”.
But that’s not true, and network administrators know it. You might assume that hacking an AV control system would merely allow the attacker to take control of lights, projection and sound systems but that is not the scenario that worries security specialists. Once a device with a processor on the network is compromised, hackers can use it to gain access to other systems from the inside.
Paul Zielie, Harman’s Product Manager, Enterprise Solutions has been the driving force behind a huge push to improve IT security in the AMX product range and across the wider Harman brands. His relentless zeal for spreading the message of security to the AV community has seen him deservedly named Infocomm International Educator of the year. Paul gives two case studies that starkly illuminate the issues.
Hackers who perpetrated the infamous Target (US) hack of 2014 used credentials stolen from an air conditioning contractor to access the Target’s vendor portal. That system was supposed to be separate from the network that handled financial transactions (just as AV systems are supposed to be on segregated networks). However, once inside, they moved step-by-step further into the network until they were able to access the POS terminals and steal the credit card details of 40 million customers. While the immediate cost of that single incident was a ‘mere’ $162m the lawsuits currently in train could cost many times more. That event instantly claimed the scalp of Target’s CIO and CEO and focused the minds of network administrators everywhere.
Paul goes on to relate that audiovisual manufacturers received a very specific wake-up call when hackers demonstrated they could take control of a car’s steering and brakes by remotely compromising the entertainment system.
IT managers now certainly have the power to stop any installation in its tracks, forcing a re-design that can cost your business dearly.
IMPLICATIONS AT THE COALFACE
Security isn’t just a matter for manufacturers. It needs to be understood by everyone in the industry, from sales staff to designers and installers.
Firstly, if you can’t understand the customer’s IT security standards then you may be unable to demonstrate that your product is capable of meeting these and you may not be allowed to connect to the network. At best this will compromise some of the features you promised and at worst, the system may not operate at all. But there’s more:
It is not enough to supply your clients with equipment that has the right network security features. According to Paul Zielie, in some jurisdictions an installer may also have a legal responsibility to ensure that the system is configured to operate in a secure manner. In his presentations, Paul explains that the concept of due care may be used as a test of liability for negligence, should the worst happen.
“It is expected that a ‘reasonable and prudent person’ would secure their network in a business,” he notes. “If it can be shown that the consultant or integrator organisation secures their own network then they must show the same care to their customer they exercise themselves.”
Proving due care involves showing that the necessary steps have been taken to help protect the organisation, its resources, and employees. Reviewing organisational security policies and specifying products and configurations which meet those policies show due care.
This is a tricky area and you shouldn’t rely on this as definitive legal advice in Australia. Nonetheless it is clear from what we have seen that we all need to be concerned with network security issues.
STEPS TO TAKE
All least the steps required to implement proper IT security are easy to understand. That’s because they largely mirror the best practice stages in any AV design and installation. There are five vital steps in each and every installation that requires equipment to be connected to your client’s network.
Step 1 — Needs Analysis: Once it’s established that equipment you will be supplying and installing has to be connected to the customer’s network, then you must ask for (and understand) the organisation’s IT security policies and requirements.
Step 2 — Design & Document: Design your installation using equipment that meet those IT security requirements. Prepare separate IT security documentation that show how their security requirements are met. This documentation is vital and will be used for the next steps. We’ll see more about how you write this later.
Step 3 — Sign-Off: Identify the responsible parties and get sign-off on the security documentation.
Step 4 — Configure: Working with the customer’s network administrators, confirm all network access is correctly configured and appropriate permissions set. Ensure all equipment passwords and software configuration are accurately recorded on the security documentation.
Step 5 — Handover: Deliver the completed IT security documentation to your client.
And it’s that simple! Well, it would be nice if it were. Let’s take a closer look at each stage to see what can go wrong – and how to get it right.
NEEDS ANALYSIS NEED TO KNOW
Too often in the past, audiovisual consultants and designers have ignored this step altogether. Frequently, they were secretly concerned the system mightn’t be able to be secured and if this wasn’t discovered until ‘too late’ the client’s IT people would be forced to simply make an exception and allow connection to the network anyway.
Indeed, this strategy used to work. Not anymore. As we discussed earlier, the cost of IT security failures has become so high that IT managers now certainly have the power to stop any installation in its tracks, forcing a re-design that can cost your business dearly.
The first step is to engage with your client’s IT department. You need to find out what their security policies are, so you can then show how your equipment will conform to those policies (or what you can do if that’s not possible). Many integrators dodge this step because they have no idea what sort of information they are going to get or what it may mean. So it’s time for a quick cheat sheet on the kinds of issues that will arise. At the end of the article I’ll give you a handful of web references to some really good guides to how this all works, which you should read. But first let’s set up the basic framework so you can understand the common security requirements.
Most security requirements you’ll have to meet fit into one of only three straightforward categories. If you can recognise which is which, you will be well on your way to understanding what they are and solving them. Think of these three categories as who can get onto the network; where they can go in the network; and what they can see.
Industry specialists agree that failing to change the default password in a system is the number one vulnerability in the AV community. Worse, it could leave the installer liable if bad guys gain access. So here’s Paul Zielie’s advice on best practice during integration:
- Create a job password for the installation period.
- Change all the equipment default passwords to the job password (so everyone working on the job has access to the equipment).
- As part of the commissioning documentation give the customer a list of passwords and change instructions.
ACCESS CONTROL: WHO’S WHO
It’s tempting to say this is all about passwords, and passwords are crucial, but you need to understand that there’s more to Access Control than just changing the default system password. Most IT organisations use the combination of Usernames and Passwords for these functions:
- To verify that the user is who they say they are (Authentication)
- To verify that the user has permission to do the task (Authorisation)
- To keep a record of who accessed the system (Accounting)
The provide Access Control, part of the IT security policy will be designed to make sure only certain people can get into the deep menus and make changes to the system that’s connected to their network. It also wants to make sure that if any changes are made, those changes are recorded in a log along with the identity of the user who made them.
To perform these three functions, either your AV device must be able to store lists of users, their passwords and records of what they did when logged in; or the AV system must be able to communicate on the network to an established directory that holds the records about everyone’s usernames, passwords and permissions (this is often called an LDAP or Windows Active Directory server).
In a network, every device is connected to every other device. But there are ways and means of making sure that commands from AV control systems (for example) can only reach devices like projectors that need to be controlled. We don’t want commands entered in via a touchpanel to reach secure places like where financial records are stored.
Sometimes this means building a physically separate network for the AV systems so that rogue operators don’t have a physical connecting cable to get to the secure parts of the network. However, there are better ways of separating things out. Instead of building a physically separate local area network (LAN), the network administrators can set rules in the various network routers that create separate islands in the network that are virtually separate from the rest of the LAN. These are called VLANs. VLANs are a good way of keeping AV devices on their own virtual network but still letting the right people or devices get through when needed.
An access control list (ACL) can be used to either allow or prevent devices on separate VLANs from getting through to each other. An ACL lists the devices from which a router will allow messages to pass. If a control system, say, wants to get through to a computer on another VLAN, it has to be on the list before its traffic is allowed through. ACLs are set up by the network administrators, but as part of the AV security documentation, you may be asked to create a list of which devices need to talk to each other to build that list.
Devices like computers, control systems and so on have various ways of communicating with each other across a network. These different communications connections are called ‘services’. Services are referred to by a variety of acronyms like HTTP, HTTPS, TELNET, SSH and so on. Some services are more secure than others. A device may have many services activated but some may not be needed in the particular situation. The network security part of your client’s IT security requirements might specify that if your particular installation doesn’t need a certain service, then it must be switched off. It is a bit of a stretch, but you can think of this as being like a projector that can be controlled by RS-232, IP or IR. If you are only ever going to use RS-232 control, you might want to disable the IR functions so that someone who gets hold of an IR remote doesn’t wreak havoc by taking control.
ENCRYPTION: SPEAK IN CODE
The third of our categories in the security requirements is encryption. If someone taps into the communications between AV devices, can they see what is happening or is the communication in code so that no one can understand it. For the AV world, this can refer to encryption of audio or encrypted videoconference signals so that eavesdropping on sensitive conversations is impossible. However, it may also be a requirement that network communications between devices (such as when passwords are exchanged) be encrypted so that intruders can’t steal passwords.
We have adjusted to the analogue sunset, dealt with the digital transition and made sense of the hurdles in HDCP, so I am confident IT security will eventually be just another set of lessons learned
DESIGN & DOCUMENT
Documenting the network security aspects of AV systems is one area that causes confusion. The documentation should be based on the Client’s stated policies and requirements and many network administrators will have a template form for you to complete.
If you want to get a feel for best practice in this area, download the example template that Harman has made available (web reference below). This has been put together by Paul Zielie as a companion to his excellent workbooks that cover ‘Security for Networked AV’ (also accessible through the references at the end of this article). As a minimum, look carefully at the ‘Device Inventory’ and ‘Ports & Protocols’ sheets in this Excel workbook. In the header Paul gives helpful instructions and examples for how these should be filled out.
Most major manufacturers (certainly AMX, Crestron and Extron) have already documented the information you will need for each of their products, however this isn’t usually included in the product’s specification sheets — you’ll need to look deeper into the user manuals and elsewhere. AMX publishes a ‘Network Administrator’s Guide’ with very comprehensive data and a clear discussion of security aspects. The Crestron equivalent is the ‘Crestron Secure Deployment Guide’ which is similarly detailed. For Extron products, the ‘Extron Control System Design Guide’ brings together most of the information you will need for good documentation.
The best advice, if you are uncertain, is to first engage with your client’s network administrators to find out their expectations. Find out if they have a standard form to complete or if there are any other processes to follow to allow connection to their network. Then, if you need further advice, contact your equipment suppliers. I have already noted that AMX (and Harman) are very much on the front foot in assisting integrators and users with information and advice, and I found both Crestron and Extron are also very aware of IP security issues.
Extron’s Director of Product Marketing, Joe da Silva, emphasised the importance of access control at every level, but noted that simply changing the default passwords in equipment as it is installed is a crucial and often overlooked step. “I think there’s a lot of work to do to educate our colleagues in this industry to understand the amount of responsibility and the amount of integrity with which they need to approach IT Security. I think that’s a big task for our industry,” he told me.
“We take IT security very seriously at Extron and it’s always top of mind,” he said. Joe also confirmed that Extron made it a priority to answer customer concerns about security, creating a special engineering team to handle these queries. “When a question comes up about protocols or if there are security questions about our product this group is brought in and we respond to those questions quickly and efficiently,” he stressed.
So assistance is readily available from suppliers. The message from the industry is “just ask and we are here to help”.
DOCUMENTATION SIGN OFF
Savvy professionals are already aware of the importance of having their designs signed off before commencing the installation but with the security aspects there’s one more complication. Who has (or should have) input to the approval process? Make sure you identify all the stakeholders. As well as the network administrators, seek input from the operational staff and the physical security group (if there is one). Finally, don’t forget to check if there are any external standards that must be met. Government and large corporate clients often have to meet external IT security standards (and pass audits).
IT SECURITY WEB RESOURCES
AMX by Harman has some terrifically useful training materials that explain the fundamentals of IT security practices. For the most part, they are not specific to any particular equipment and are a great way to understand the lingo, whatever brand of AV equipment you use. A great start is to read the White Papers found at: www.amx.com/security/resources.aspx (Tip: They are free, but you may have to enter your email address and provide other details to gain access – it’s well worth it!)
I also recommend you download the spreadsheet which provides an example template for documenting the security aspects of an AV system that is to be connected to the client’s network. You can find it here: www2.amx.com/AVSecurityReq
Crestron: The IP Considerations Guidelines for the IT Professional is a good starting point available for download here: www.crestron.com/downloads/pdf/product_misc/dg_ip-considerations-guide-it-professional.pdf
To obtain the ‘Crestron Secure Deployment Guide’ you’ll need to request a log-in to access Crestron’s Online Help.
Extron: ‘The Extron Control System Design Guidelines’ are available here: www.extron.com/company/article.aspx?id=designguidead
As we saw earlier, it is not enough to simply install equipment that can be made secure. If you install equipment to a network without ensuring that it is securely configured, you may still be held liable for subsequent problems. This is where the documentation (and the spreadsheets) we spoke about earlier come into their own.
You need to ensure all the settings and passwords are correct on your equipment (and recorded on the spreadsheet) and then have the client’s network administrators do their part to properly configure the network and set all necessary permissions and access control lists. This isn’t an optional step, because most likely nothing will work if it isn’t done. Do arrange a time in advance with the network people — sometimes changes can only be made according to a weekly schedule and not just whenever you happen to ask!
Hurray! We’re done! So don’t forget to hand over the security documentation (and keep a copy for yourself in case equipment changes are necessary).
As professionals we have adjusted to the analogue sunset, dealt with the digital transition and made sense of the hurdles in HDCP, so I am confident IT security will eventually be just another set of lessons learned. Possibly the hardest part is figuring out how to properly charge for all our time and effort in those extra steps we’ve just discussed.
Perhaps that’s a topic for another day.