“Security and Human Behaviour” was an interdisciplinary workshop held at MIT in June 2008. The attendees included people with backgrounds in computer science, economics, psychology, criminology and sociology. (And one professional magician: James Randi).
Different attendees were interested in different kinds of security problem. These could be roughly divided into commercial Internet security (e.g. reducing the level of credit card fraud in online shopping) and counter-terrorism (e.g. using fMRI brain-scanners to interrogate prisoners).
As this workshop has attendees from different subject areas, we were asked to avoid unnecessary jargon and to provide explanations of technical terms that might be unfamiliar. (Example from computer security: “phishing”. Example from psychology: “cognitive dissonance”).
Someone suggested that it would be helpful if we produced a document explaining the vocabulary terms that we’re likely to use.
phishing is the criminally fraudulent process of attempting to acquire sensitive information such as usernames, passwords and credit card details, by masquerading as a trustworthy entity in an electronic communication. (http://en.wikipedia.org/wiki/Phishing)
cognitive dissonance is an uncomfortable feeling or stress caused by holding two contradictory ideas simultaneously. (http://en.wikipedia.org/wiki/Cognitive_dissonance) For a better explanation see also: Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press.
Several presenters at the workshop claimed that people’s concern about terrorism is irrationally high, while their concern about becoming a victim of fraud when using the Internet is irrationally low. It is possible to advance evolutionary explanations for this (e.g. our primate ancestors did not have to deal with Internet shopping, but did have to deal with situations where one of their group of monkeys has been killed by a member of a rival group of monkeys).
However, before we start looking for an explanation of this kind we ought to justify the claim that people are being irrational. What is the actual level of risk in – for example – Internet shopping? Maybe people are being economically rational when they ignore it. None of the presenters provided any statistics on the level of Internet fraud, although some did provide numbers on the low risk of terrorism.
A cynical economist might say that computer security specialists and counter-terrorism experts are each over-emphasizing the significance of the threat to which they are providing a solution.
Several people suggested that it would help if the next SHB workshop focussed on a specific research problem. The effective methods for reducing crime often depend on the type of crime, so it would help if we looked at a concrete example.
Several of the presenters clearly had “phishing” in mind as their motivation for looking at human factors in security. This is one possible choice for a case study.
The computer security community makes a distinction between mechanisms that provide “safety” and mechanisms that provide “security”. “Safety” mechanism protect against undesirable events that occur by accident, while “security” mechanisms protect against undesirable events that occur as a result of a deliberate (“malicious”) action by someone.
Several attendees pointed out that this distinction is not always clear-cut:
Over the twentieth century, there has been an increasing tendency to find someone to blame for adverse events that would previously have been considered accidents. (Someone is held to have been negligent).
Going further back in history, there were criminal trials of animals and even inanimate objects. And yet, by modern standards, we do not consider animals or objects to be criminally responsible.
A provocative example: in response to acts of terrorism, voters sometimes respond by increasing their support for the government in power (the Madrid bombings being an exception). Voters are, in effect, showing a revealed preference for keeping in power governments who are bad at preventing terrorism, and replacing governments who are good at it.
The more general point is that failures of complex systems may be due to complex interacting factors, and that attempting to attribute the failure to the malicious action of a specific individual may not be a helpful way of looking at it. (Compare conspiracy theories, for example).
If we’re trying to design safe/secure systems that take into account human behaviour, attempting to divide users into the malicious and the not malicious may not be the best way to proceed.
In psychology, there is a long-standing debate over whether psychiatric diagnoses are best done by psychiatrists making a subjective judgment or by statistical tests based on responses to standardized questions, such as the Minnesota Multiphasic Personality Inventory (MMPI). See: Meehl, Paul. “Clinical versus Statistical Prediction”
The presentations on detecting deception opened up a new version of this debate. A police officer trying to work out if a suspect is lying can make use of his experience and his knowledge of the context. Is this more reliable that a statistical test based on measurable factors? People sometimes think they are good at detecting deception; but are they? Statistical tests can eliminate some unconscious biases (e.g. the ethnicity of the patient/suspect making the psychiatrist/police officer more likely to think they are mentally ill/guilty).
See http://en.wikipedia.org/wiki/Base_rate_fallacy
Suppose you have a statistical test for a condition, with a certain level of false positives. If the base rate of the condition is very low, then almost all cases detected by the test will be false positives.
Nearly all airline passengers are not suicide bombers.
If you are trying to screen airline passengers for terrorists, a statistical test will give almost all false positives (unless the false positive rate is implausibly low).
The counter-argument was that it may be OK, depending on what you are going to do based on the result of the test:
http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation
Ross Anderson and Tyler Moore have shown a correlation between people’s ability to detect phishing and their scores on Simon Baron-Cohen’s EQ and SQ tests.
One possible explanation of this result is that there is a neurological difference between people that makes some better than others at detecting phishing. But there are other plausible explanations for this result, e.g. the high-SQ individuals might have read more computer magazines with articles about phishing. It was suggested that they ought to control for likely confounding variables (e.g. gender; amount of computer science education).
It was also pointed out that when you have two independent variables, it would be better to do a single regression (X = - a EQ + b SQ + c) rather than two separate regressions (X = - a EQ + c; and X = b SQ +d).
Ron Clark (a criminologist) identifies 5 types of measures that can be used to reduce crime:
Computer security (like the physical security industry) concentrates almost exclusively on the first of these – making the attack harder to carry out. Are any of the others applicable?
People might behave differently in real-life situations and contrived laboratory experiments.
The real-life deceptions that the counter-terrorism investigators are interested in have features that are hard to replicate experimentally:
Also, people may behave differently if they are being observed, or know that they are taking part in an experiment.
For example: use of MRI imaging to detect deception worked in laboratory experiments, but has been reported as not so effective in real use: extreme emotional states generate a lot of “noise” on the MRI scan that swamps the signs of deception. (Allegedly, mentally recalling emotionally charged passages from the Koran is sufficient to block interrogation by MRI scanner).
There’s a joke: in most experiments, college students are not suitable experimental subjects because they’re not typical of the population you’re really interested in. In counter-terrorism experiments, this is OK because college students are the population that most terrorists are drawn from.
A case study of security usability in a hospital.
An example of why the social context matters: A doctor instructs a nurse to operate a computer system to change a drug dosage. Computer audit and non-repudiation mechanisms would record the nurse as being responsible, but it is the doctor who is actually responsible.
End user programming: provide a toolkit, and let the domain experts finish off the system.
A new system for handling password resets. Meaningless strings (e.g. passwords!) are easily forgotten. Instead, he uses people’s preferences for one thing over another in a list of questions. To make this work, remove the questions that have low entropy (almost everyone likes TV) and if two questions are strongly correlated, remove one of them.
A study of how fear spreads after different kinds of disaster.
The level of public fear following the anthrax scare subsided when it become known that American terrorists, rather than international terrorists, were responsible.
Every point of indirection is also a point of control – and there’s a fight over who controls it. See, for example the DNS and hotels that intercept DNS queries to redirect web pages.
Correlation between ability to detect phishing and score on Simon Baron-Cohen’s SQ and EQ tests.
Bounded rationality. See Kahneman, D. Mapping Bounded Rationality. American Psychologist. 2003.
Answers to survey questions depends on the order in which questions are asked (e.g. “How happy are you?” / “Did you have a date last week?”) and, if numerical tick-boxes are provided, depends on the scale provided.
A study on people’s willingness to disclose personal information.
Two on-line surveys, asking the same questions about socially disapproved activities. One looks like an official university site, the other one looks like a joke site put up by an individual. People are more willing to disclose to the “joke” site, even though such sites in general provide less strong privacy guarantees. (But in the context of the joke site, the behaviours being asked about are not so strongly disapproved of).
Telling people that you will protect their privacy makes them less willing to disclose. (Compare James Randi – a professional magician – on how to do a card trick. You must not say “Here I have an ordinary deck of cards” because it raises the doubt that it might not be).
Audience discussion: “The irony of [a psychologist] arguing with [an economist] that there needs to be a perceived benefit in a transaction.”
People have a preference for sure gains, but risky losses. This makes security products a hard sell, because you are asking people to take a certain loss (the cost of the product, and the effort of using it) over the risky loss that a security breach might happen.
Many police forces have two items in their mission statement:
Impression management – fighting the fear of the thing rather than the thing itself – generates “fantasy documents”.
Fear can be OK – e.g. if you need to run away from a predator that is chasing you – but becomes a problem when it is dissociated from reality.
He is a photographer who previously worked on photographing nuclear weapons facilities. More recently, he has been photographing “anti-terror” training facilities. Note the “theatrical” quality of these training exercises and the brightly-coloured protective suits.
A study on the economic value of privacy.
Given a choice between a $10 anonymous card and a $12 identified card, people’s choice in influenced by which one they have been given previously.
Railway investment in 1845 compared to the Internet bubble. Both took about the same time to collapse, and railways also had press coverage about accidents.
The railways are also a good example of price discrimination (first class and second class tickets). From the vendor’s point of view, the problem with privacy is that it prevents price discrimination.
Although it was not the intended subject of the workshop, several attendees had an interest in on-line games.
During my talk, I mentioned my study of security incidents in Linden Lab’s “Second Life”.
Attacks on “World of Warcraft”.
Virtual items within the game are worth real money, so there is an incentive to steal them. There have been incidents where a keylogger was used to capture someone’s WoW password, and their virtual items within the game were stolen.
The use of computer games for training soldiers.
One of these games is designed to teach useful Arabic phrases. This is rather like many commercial language-teaching programs, except that the vocabulary and example situations are chosen to be useful for soldiers who are occupying a country.
One of the other games is “mixed reality”. When the player reaches a point in the game where they need to search a car, they leave he computer terminal and are taken outside the room to where there is a real car prepared for them to search.
“Negative training”: need to be careful that games don’t train you to do things that don’t work in real life.
“Games change your brain”. On some psychological tests of reflexes, experienced games are much faster. This is because they have learned new reflexes from game playing.
We also discussed the need for ethics committee/Institutional Review Board approval for experiments in virtual worlds. Studies that passively observe what people are doing (like my study of Second Life) probably don’t need IRB approval, but anything that tries to change peoples behaviour probably will. Ralph has had difficulty getting IRB approval for experiments within Second Life. The problem is that you don’t really know who the other players are (over Internet, they could be anyone), and it hard to convince the IRB that you really have their informed consent.
I mentioned that several Second Life players I have spoken to have expressed a strong wish not to be involved in experiments on terrorism. This isn’t what my study was about, but they were aware that other researchers are considering that kind of experiment .
Some other issues that came up during discussion:
On-line fraud is not a threatening as burglary. It’s only money that’s involved, and (perhaps) most people don’t consider their computer to be as personal as their house.
Anger affects how risk is perceived. E.g. perception of risk of terrorism after b9/11 may be lowered because people are angry about it.
The moral dimension affects how risk is perceived. Compare the 1911 flu epidemic – when disease was seen as a purely medical problem – to the Black Death.
Is there a limit to our capacity for worry? If we start worrying about some new risk, do we stop worrying about one of the older risks?Computer security talks about defense in depth, but there is also diffusion of responsibility. Suppose you have 3 policemen, any one of whom can stop a particular crime being committed. The analogy with triple-redundant systems in engineering suggests that this would make the system much safer. But – psychologically – each will be less likely to act because they think one of the other two will deal with it.