Vulnerable AI

Just like any other computer applications, AI solutions are also vulnerable to security threats. Depending on the nature of the solution, these threats can appear in different ways. For example, they can provide end users access to information they shouldn’t have; they can run malicious software in a trusted internal network; or the AI can be modified to produce erroneous results.

As such it is important to try and recognize all possible vulnerabilities associated with AI solutions and prevent them from happening when new solutions are being designed. In this article I present a few types of vulnerabilities unique to AI solutions, alongside methods for preventing them.

Data leaks in AI

Data leaks occur when information classified as secret, or pertaining to a limited group of users, ends up being available for a larger group of people. In the case of AI solutions this can happen in two ways:

• When the AI itself contains classified information and unclassified people can access the AI.
• When the AI can connect to other information systems, and the AI has the capabilities to retrieve classified information from these systems.

The first case occurs when classified information has been used to train the AI. Imagine, for example, an AI that can predict how a hospital patient should be treated based on the patient records of other similar patients.

If these patient records still contain the personal identification details of the patients themselves, then anyone using the AI can retrieve how these people have been treated, what their illnesses were and what their names and social security numbers are. Alternatively, if the AI was created to be used by an individual doctor so that he could utilize it based on the names of his own patients, then access to the AI should be secured so that no one else should ever be able to use it.

On the other hand, if the AI can retrieve information from other systems, it may have access to data that would be intended for a limited audience. For example, consider an internal Chat GPT-like solution which can retrieve information from the company’s entire intranet. If the AI has been given read permissions to the entire intranet, then anyone utilizing the AI can retrieve any information stored in the intranet – even if in the intranet itself they wouldn’t have personal access to that data.

To prevent these data leaks, it is important to carefully specify what data is being used to train AIs, who has access to the AI and which external systems the AI can access and with what permission levels. In addition, it is important to ensure that authentication to the AI has been implemented securely.

AI executing malware

You may have seen news articles about Chat GPT being able to create code for malware. This by itself isn’t a significant security issue, since Chat GPT is simply a browser-based chat application which does not have the ability to execute the code it creates. Things become more complicated when we are creating software which is being controlled by commands given to an AI, and in which the AI decides based on these commands what the application should do.

As an example, consider an AI application which is being used to administer a server environment based on commands given to the AI. An application such as this is likely to have the ability to create its own administrative scripts and to execute them against the server environment. Initially the application would have been created with the intention of executing commands such as “archive from the folder A on a network drive all files which are over five years old.” The AI would execute this command by creating a PowerShell script which performs the tasks, and then executes the script.

This same AI could also be used to run the command “lock all files on the network drive with a ransomware-application and send extortion e-mails to the board of directors.” To prevent cases like this, it is important to limit the permissions and accesses to external systems that have been given to the AI, so that the application can only perform tasks which it is expected to perform. In addition, it should be verified that the AI is not able to circumvent its limited permissions by being able to grant itself additional user rights and network accesses.

Erroneous AI

AIs always act based on the data used for training, be it a large language model such as the GPT-AIs, an image manipulation model or an AI used for solving mathematical problems. In other words, an AI can only be as high quality as the data which had been collected for the model training is. Primarily this means that when creating new AIs, you should focus on collecting high quality training data. But this isn’t the only significant point of view.

In addition to verifying that the training data does not contain any errors, it is important to ensure that no erroneous or malicious data has been added to the training data set. We can use the earlier patient record AI as an example: If some of the diabetes patient records used in model training had been labeled as depression patient records, the resulting AI would give completely wrong guidance when it was being consulted on how to treat depression patients since it would recommend treatments for diabetes. And since the data sets used for training AIs are usually very large, it becomes difficult, if not impossible, to identify and fix errors such as these after the fact.

Due to the difficulty of fixing already collected training data sets it is important to focus on preventing these threats: Access to the training data set should be highly limited, and data collection should be either automated without human interaction, or data collection should be limited to a restricted group of people. Even in these cases it’s recommended that all collected training data be approved by a human moderator before it is added to the final training data set.

In closing

In this blog I presented three common vulnerability types related to AI solutions. AI vulnerabilities are not limited to only these three, but there are a lot more. For example, Open Worldwide Application Security Project (OWASP) is working on creating their own Top 10 list of vulnerabilities for large language models.

If you are concerned about AI security, you are considering utilizing new AI solutions, or are curious what benefits your organization can gain from AI, we will gladly help you!

At Sulava we specialize in AI solutions based on Microsoft’s latest technologies and capabilities. We have implemented multiple successful projects, in which we have utilized AI to improve internal communications, information management, employee up-skilling and well-being at work. We can help your organization to find ways to utilize AI and machine learning to improve your processes, applications, and services.

Data classification and protection must be implemented correctly so that sensitive data is not included in the data of AI solutions. Naturally, artificial intelligence also helps solve challenges related to an organization’s information security, for example, by making it easier to identify and react to threats. Sulava has long experience in implementing and utilizing these solutions for our customers.

Boost Your Work with Copilot, Azure OpenAI, and More – free webinar

Are you curious about how Artificial Intelligence can help you, and your organization? Do you want to learn about the new AI features in Microsoft 365? If yes, then join us for this free webinar where we show what is available, and how to get started with Microsoft 365 Copilot and AI.

The webinar will conclude with a summary of the key takeaways and how to get started with M365 + AI. Don’t miss this opportunity to learn more how AI can help you and your organization.