Home About Me

Long Passwords, Sparse Samples, and the Curse of Dimensionality

Many websites tell you to mix uppercase and lowercase letters and add special characters to make a password stronger. That advice is half right.

The useful half is that it pushes people away from simple passwords. Even after the CSDN password leak, plenty of people still use birthdays, phone numbers, email addresses, names, or strings like 888888 and 123456. These passwords are easy to crack not because the characters are simple but because they follow patterns. Holmes solved the Dancing Men cipher through frequency; patterned strings can be guessed from experience. People collect those patterns into dictionaries, and attackers try dictionary entries first. That is empirical but practical. How do you find such patterns? Download a leaked CSDN password set and sort through it—or, if you do not have one, think about the passwords people commonly use. Responsible sites can reject simple passwords with a script, using the same logic as an attack: check whether the password matches a simple entry in a dictionary. The dictionaries are effectively endless.

The other half is wrong, or at least incomplete. For an attacker who is not using a dictionary but simply brute-forcing, 666666 and aB#(d3 are indistinguishable. Each character has at most about twice the number of keys on a keyboard, because of the Shift key, so fewer than 100 possibilities. A six-character password therefore has $$100^6$$ possibilities, which a modern computer can run through in milliseconds. Length changes the situation. In Microsoft's password checker, if you keep typing 1, the strength starts out weak but eventually becomes best. Against brute force, a long simple password is safer than a short complex one. Microsoft also publishes principles for creating strong passwords.

But what I really want to talk about is length. Why is a long password safer? A password must be made of characters, and the kinds of characters plus the string's length are its basic properties. Abstract this: I have a set with some elements, and each element has several possible values. Anyone with a little mathematical knowledge will recognize this as a multidimensional vector. Password security is essentially the uniqueness of that vector—its resistance to being guessed. The number of dimensions is the password length; the values along each dimension are the character types. The higher the dimension, the stronger the uniqueness. Put another way, the more properties you use to describe an object, the harder it is to mistake it for something else. In Cartesian three-dimensional space, knowing a ball's coordinates may be enough to identify the object, but it does not let us describe the ball very well. In most cases, uniqueness is just a fuzzy notion we use to distinguish one thing from another; from different angles there are many criteria, and to describe a concept we can invoke countless classification features. But is that necessary?

No. This is the curse of dimensionality. The term describes the problems that arise when a mathematical space gains dimensions—often hundreds or thousands—and both analysis and organization become difficult because volume grows exponentially. An example: I go out to sample a pollutant. In a 10-square-meter area, sampling every meter gives me about 100 samples. If this were a 10-dimensional space, and I had to sample along the eight dimensions beyond length and width, I would need $$10^{10}$$ points. What does that mean? If the pollutant exists at only 100 points, in two dimensions I can still detect it and draw a concentration map—which itself adds a dimension. But in a 10-dimensional space those points are so sparse that no regular distribution is visible. That is the curse of dimensionality in action. You might say that is not so bad—no two leaves are identical—but when we try to understand the world, we mostly rely on our senses and on knowledge handed down from others. That knowledge mostly comes from summarization; inductive theories are useful because they let us apply what we have learned to later situations. This is a pragmatist's account, of course. Sensation, too, is a way of summarizing and abstracting vivid instances into knowledge. If we want to understand a thing or a class of things and ensure that the knowledge we abstract is repeatable—that is, useful; I know Russell's chicken-farm paradox, but since Gödel we can afford to be more relaxed—we need enough samples. Yet a description that is too detailed makes the work extremely complicated, pulls it away from intuition, and keeps us from reaching the sample size that statistical regularity requires. Does that make the phenomenon useless?

Not at all. The curse of dimensionality is meaningful for password design. Because most people now use uppercase letters or special characters when they set passwords, an attacker whose dictionary is incomplete does not need to treat uppercase letters or symbols as special cases. Treat each character as a dimension and each possible variation as a value along that dimension, then try them. With a large computer, that goes quickly—until the dimension gets high. Then the computation grows exponentially. Perhaps future computers will not care. But very high dimensions also burden websites, because storing data consumes server resources, and most people will not bother to create very long passwords. When DES was designed, similar concerns limited key length. These are compromises between requirements and technology; one day they may disappear. But as long as we need communication or the flow of information, encryption and decryption must keep up. Otherwise we will have to get used to life as Trisolarans.

So is that kind of detailed description ever necessary? That question leads from the curse of dimensionality to factor analysis.