The default is INPUT which sets up the pin as an INPUT. If the pin is totally disconnected, it will randomly read HIGH and LOW. If you tie it to +5V or 0V, it will obviously read HIGH or LOW.

Internal to the Arduino, the Atmega chip has internal pullup resistors with a value around 20k ohm. (See the DigitalPins documentation for more details) These resistors can be optionally connected internally using INPUT_PULLUP. This is functionally (and electrically) equivalent to connecting a ~20k ohm resistor between the pin and +5V, the only difference is that it requires no external components and you can turn it on and off in software during the execution of your program.

So why pull-ups and not pull-downs? There are likely several reasons for it, but when wiring buttons or switches or anything "normally open", you only have to tie them to ground, you don't need to run +5V out to them. Since most boards are going to be designed with large ground pours for shielding reasons anyway, tying to ground is practically reasons.