Bandwidth - a measure of how much information you can serve up in a given time. Many hosting deals will specify a maximum monthly data transfer; however the actual perceived speed of a website will depend to some extent on the bandwidth of the links the server has to the internet.
Latency - the time it takes for a server to respond. Ideally your server should be as physically close as possible to your users. Latency increases with physical distance. For sites that are running (for example) games, Java applications that are in constant communication with the server, or streaming media, low latency becomes important.
Redundancy - how many physically distinct connections a server has. The more the better. Internet connections are surprisingly vulnerable to builders with JCBs. Just as people tend to overspecify the machine hardware required, they are often overly optimistic about the amount of traffic that a website will generate.