Attacking Machine Learning Systems

Feb 1, 2023 · security, coding ·

Machine learning systems are becoming increasingly popular for use in a variety of applications, ranging from medical diagnostics to autonomous vehicles and weapon systems. These systems are based on sophisticated algorithms that rely on large datasets to "learn" how to accurately make decisions. Unfortunately, these algorithms are vulnerable to attack due to their reliance on data. In this article, security expert Bruce Schneier discusses the potential dangers of attacking machine learning systems, which are often overlooked, and ways to protect them.

Schneier first outlines two main ways attackers can target machine learning systems: data manipulation and model manipulation. Data manipulation involves manipulating the data used to train the system so that it performs differently than expected. This can be done through adding malicious or biased data, or by deleting existing data. Model manipulation involves directly tampering with the model itself, either by corrupting the parameters of the model, or by adding malicious code.

These attacks can have serious consequences ranging from financial loss to physical harm and death. Schneier points out various examples of attacks, such as the targeting of facial recognition software and autonomous vehicles. He argues that the risk posed by these attacks is increasing as machine learning systems are becoming more popular and used in more sensitive areas like weapon systems.

To mitigate these risks, Schneier suggests several strategies. First, he states that organizations should employ comprehensive monitoring to detect anomalies that may indicate an attack. Second, they should also ensure that their data sets used to train the system are properly labeled and free of bias. Finally, organizations should invest in defense-in-depth security measures such as encryption to prevent unauthorized access to their models.

Overall, the article provides an important reminder that machine learning systems should not be taken lightly. Attacks on them can have serious consequences, and organizations must take steps to protect their systems. By understanding the potential threats posed by malicious actors, organizations can implement effective strategies to safeguard their machine learning systems and protect the data they rely on.