Archive for the 'Security' Category

Adversarial Machine Learning: Are We Playing the Wrong Game?

Saturday, June 10th, 2017

I gave a talk at Berkeley’s International Computer Science Institute on Adversarial Machine Learning: Are We Playing the Wrong Game? (8 June 2017), focusing on the work Weilin Xu has been doing (in collaboration with myself and Yanjun Qi) on adversarial machine learning.


Machine learning classifiers are increasingly popular for security applications, and often achieve outstanding performance in testing. When deployed, however, classifiers can be thwarted by motivated adversaries who adaptively construct adversarial examples that exploit flaws in the classifier’s model. Much work on adversarial examples, including Carlini and Wagner’s attacks which are the best results to date, has focused on finding small distortions to inputs that fool a classifier. Previous defenses have been both ineffective and very expensive in practice. In this talk, I’ll describe a new very simple strategy, feature squeezing, that can be used to harden classifiers by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different inputs in the original space into a single sample. Adversarial examples can be detected by comparing the model’s predictions on the original and squeezed sample. In practice, of course, adversaries are not limited to small distortions in a particular metric space. Indeed, it may be possible to make large changes to an input without losing its intended malicious behavior. We have developed an evolutionary framework to search for such adversarial examples, and demonstrated that it can automatically find evasive variants against state-of-the-art classifiers. This suggests that work on adversarial machine learning needs a better definition of adversarial examples, and to make progress towards understanding how classifiers and oracles perceive samples differently.

Modest Proposals for Google

Friday, June 9th, 2017

Great to meet up with Wahooglers Adrienne Porter Felt, Ben Kreuter, Jonathan McCune, Samee Zahur (Google’s latest addition from my group), and (honorary UVAer interning at Google this summer) Riley Spahn at Google’s Research Summit on Security and Privacy this week in Mountain View.

As part of the meeting, the academic attendees were given a chance to give a 3-minute pitch to tell Google what we want them to do. The slides I used are below, but probably don’t make much sense by themselves.

The main modest proposal I tried to make is that Google should take it on as their responsibility to make sure nothing bad ever happens to anyone anywhere. They can start with nothing bad ever happening on the Internet, but with the Internet pretty much everywhere, should expand the scope to cover everywhere soon.

To start with an analogy from the days when Microsoft ruled computing. There was a time when Windows bluescreens were a frequent experience for most Windows users (and at the time, this pretty much mean all computer users). Microsoft analyzed the crashes and concluded that nearly all were because of bugs in device drivers, so it wasn’t their fault and was horribly unfair for them to be blamed for the crashes. Of course, to people losing their work because of a crash, it doesn’t really matter who’s code was to blame. By the end of the 90s, though, Microsoft took on the mission of reducing the problems with device drivers, and a lot of great work came out of this (e.g., the Static Driver Verifier), with dramatic improvements on the typical end user’s computing experience.

Today, Google rules a large chunk of computing. Lots of bad things happen on the Internet that are not Google’s fault. As the latest example in the news, the leaked NSA report of Russian attacks on election officials describes a phishing attack that exploits vulnerabilities in Microsoft Word. Its easy to put the blame on overworked election officials who didn’t pay enough attention to books on universal computation they read when they were children, or to put it on Microsoft for allowing Word to be exploited.

But, Google’s name is also all over this report – the emails when through gmail accounts, the attacks phished for Google credentials, and the attackers used plausibly-named gmail accounts. Even if Google isn’t too blame for the problems that enable such an attack, they are uniquely positioned to solve it, both because of their engineering capabilities and resources, but also because of the comprehensive view they have of what happens on the Internet and powerful ability to influence it.

Google is a big company, with lots of decentralized teams, some of which definitely seem to get this already. (I’d point to the work the Chrome Security Team has done, MOAR TLS, and RAPPOR as just a few of many examples of things that involve a mix of techincal and engineering depth and a broad mission to make computing better for everyone, not obviously connected to direct business interests.) But, there are also lots of places where Google doesn’t seem to be putting serious efforts into solving problems they could but viewing them as outside scope because its really someone else’s fault (my particular motivating example was PDF malware). As a company, Google is too capable, important, and ubiquitous to view problems as out-of-scope just because they are obviously undecidable or obviously really someone else’s fault.

[Also on Google +]

Feature Squeezing: Detecting Adversarial Examples

Monday, April 10th, 2017

Although deep neural networks (DNNs) have achieved great success in many computer vision tasks, recent studies have shown they are vulnerable to adversarial examples. Such examples, typically generated by adding small but purposeful distortions, can frequently fool DNN models. Previous studies to defend against adversarial examples mostly focused on refining the DNN models. They have either shown limited success or suffer from expensive computation. We propose a new strategy, feature squeezing, that can be used to harden DNN models by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample.

By comparing a DNN model’s prediction on the original input with that on the squeezed input, feature squeezing detects adversarial examples with high accuracy and few false positives. If the original and squeezed examples produce substantially different outputs from the model, the input is likely to be adversarial. By measuring the disagreement among predictions and selecting a threshold value, our system outputs the correct prediction for legitimate examples and rejects adversarial inputs.

So far, we have explored two instances of feature squeezing: reducing the color bit depth of each pixel and smoothing using a spatial filter. These strategies are straightforward, inexpensive, and complementary to defensive methods that operate on the underlying model, such as adversarial training.

The figure shows the histogram of the L1 scores on the MNIST dataset between the original and squeezed sample, for 1000 non-adversarial examples as well as 1000 adversarial examples generated using both the Fast Gradient Sign Method and the Jacobian-based Saliency Map Approach. Over the full MNIST testing set, the detection accuracy is 99.74% (only 22 out of 5000 fast positives).

For more information, see the paper:

Weilin Xu, David Evans, Yanjun Qi. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. arXiv preprint, 4 April 2017. [PDF]

Project Site: EvadeML

Enigma 2017 Talk: Classifiers under Attack

Monday, March 6th, 2017

The video for my Enigma 2017 talk, “Classifiers under Attack” is now posted:

The talk focuses on work with Weilin Xu and Yanjun Qi on automatically evading malware classifiers using techniques from genetic programming. (See for more details and links to code and papers, although some of the work I talked about at Enigma has not yet been published.)

Enigma was an amazing conference – one of the most worthwhile, and definitely the most diverse security/privacy conference I’ve been to in my career, both in terms of where people were coming from (nearly exactly 50% from industry and 50% from academic/government/non-profits), intellectual variety (range of talks from systems and crypto to neuroscience, law, and journalism), and the demographics of the attendees and speakers (not to mention a way-cool stage setup).

The model of having speakers do on-line practice talks with their session was also very valuable (Enigma requires speakers to agree to do three on-line practice talks sessions before the conference, and from what I hear most speakers and sessions did cooperate with this, and it showed in the quality of the sessions) and something I hope other conference will be able to adopt. You actually end up with talks that fit with each other, build of things others present, and avoid unnecessary duplication, as well as, improving all the talks by themselves.

Demystifying the Blockchain Hype

Wednesday, October 26th, 2016

I gave a talk introducing the blockchain at a meetup hosted by Willow Tree Apps:
Demystifying the Blockchain Hype, 25 October 2016.

Over the past few years, explosive growth in cryptocurrencies (especially Bitcoin), has led to tremendous excitement about blockchains as a powerful tool for just about everything. Without assuming anyprevious background in cryptography, I’ll explain the cryptographic and networking technologies that make blockchains possible, explain why people are so excited about blockchains, but why you shouldn’t believe everything you hear about them.

The slides are below (I believe a recording will also be available soon).

Insecure by Default? Authentication Services in Popular Web Frameworks

Monday, August 15th, 2016

Hannah Li presented a poster at USENIX Security Symposium on how popular web frameworks perform authentication.

Insecure by Default? Authentication Services in Popular Web Frameworks
[Abstract (PDF)] [Poster (PDF)]

The work studies how different design choices made by web frameworks impact the security of web applications built by typical developers using those frameworks, with a goal of understanding the usability and performance trade-offs that lead frameworks to adopt insecure defaults, and develop alternatives that lead to better security without sacrificing the needs of easy initial development and deployment.

An exercise in password security went terribly wrong, security experts say

Friday, April 1st, 2016

PCWord has a story about CNBC’s attempt to “help” people measure their password security: CNBC just collected your password and shared it with marketers: An exercise in password security went terribly wrong, security experts say, 29 March 2016.

Adrienne Porter Felt, a software engineer with Google’s Chrome security team, spotted that the article wasn’t delivered using SSL/TLS (Secure Socket Layer/Transport Layer Security) encryption.

SSL/TLS encrypts the connection between a user and a website, scrambling the data that is sent back and forth. Without SSL/TLS, someone one the same network can see data in clear text and, in this case, any password sent to CNBC.

“Worried about security? Enter your password into this @CNBC website (over HTTP, natch). What could go wrong,” Felt wrote on Twitter. “Alternately, feel free to tweet your password @ me and have the whole security community inspect it for you.”

The form also sent passwords to advertising networks and other parties with trackers on CNBC’s page, according to Ashkan Soltani, a privacy and security researcher, who posted a screenshot.

Despite saying the tool would not store passwords, traffic analysis showed it was actually storing them in a Google Docs spreadsheet, according to Kane York, who works on the Let’s Encrypt project.

(Posted on April 1, but this is actually a real story, as hard as that might be to believe.)

Apple and the FBI

Thursday, February 25th, 2016

I’m quoted in this article on the controversy over the FBI’s requests to Apple for assistance in unlocking an iPhone used by one of the San Bernardino terrorists: Unlocking Terrorist’s iPhone Won’t Risk Your Security, Discovery News, 24 February 2016.

“Backdoors are complicated and impossible technical challenges and would risk everyone’s privacy,” Evans said. “But what the FBI is asking for is different from what Apple says the FBI is asking for.”

For the most part, I think the article gets things right. It is very misleading to conflate what the FBI has asked for here with a cryptographic backdoor that would indeed dangerously risk everyone’s privacy and security. I covered some of the technical aspects of this in my introductory computing course last week.

NDSS Talk: Automatically Evading Classifiers (including Gmail’s)

Wednesday, February 24th, 2016

Weilin Xu presented his work on Automatically Evading Classifiers today at the Network and Distributed Systems Security Symposium in San Diego, CA (co-advised by Yanjun Qi and myself). The work demonstrates an automated approach for finding evasive variants of malicious PDF files using genetic programming techniques. Starting with a malicious seed file (that is, a PDF file with the intended malicious behavior, but that is correctly classified as malicious by the target classifier), it heuristically searches for an evasive variant that preserves the malicious behavior of the seed sample but is now classified as benign. The method automatically found an evasive variant for every seed in our test set of 500 malicious PDFs for both of the target classifiers used in the experiment (PDFrate and Hidost).

Slides from the talk are below, the full paper and code is available on the website.

In addition to the results in the paper, Weilin found some new results examining gmail’s PDF malware classifier. We had hoped the classifier used by gmail would be substantially better than what we found in the research prototype classifiers used in the original experiments, and the initial cross-evasion experiments supported this. Of the 500 evasive variants found for Hidost in the original experiment, 387 were also evasive variants against PDFrate, but only 3 of them were evasive variants against Gmail’s classifier.

From those 3, and some other manual tests, however, Weilin was able to find two very simple transformations (any change to JavaScript such as adding a variable declaration, and adding padding to the file) that are effective at finding evasive variants for 47% of the seeds.

The response we got from Google about this was somewhat disappointing (and very inconsistent with my all previous experiences raising security issues to Google):

Its true, of course, that any kind of static program analysis is theoretically impossible to do perfectly. But, that doesn’t mean the dominant email provider shouldn’t be trying to do better to detect one of the main vectors for malware distribution today (and there are, we believe, many fairly straightforward and inexpensive things that could be done to do dramatically better than what Gmail is doing today).

The other new result in the talk that isn’t in the paper is the impact of adjusting the target classifier threshold. The search for evasive variants can succeed even at lower thresholds for defining maliciousness (as shown in the slide below, finding evasive variants against PDFrate at the 0.25 maliciousness threshold).

Weilin’s Summer of Code

Friday, February 5th, 2016

Google’s Open Source blog has a story by Weilin Xu about his experiences in their Summer of Code before he came to UVA: Coming to America: how Google Summer of Code helped change my life, 3 February 2016.