The rules are a little more complex than this, but basically: 
you use <= to do signal assignment, which takes effect on the next delta cycle. 
You use := to do variable assignment, which takes place immediately. 
So if you have a signal, you always use <=. If you have a variable, you always use :=.
Some places where this is not quite that case that you will commonly run into, for instance, initialization, where := is used even for signals.

signal some_signal : std_logic := '0'; -- 0 initial value
...
variable some_variable : std_logic := '0'; -- 0 initial value
...
some_signal <= '1'; -- will assign 1 at the next time step (delta cycle)
...
some_variable := '1'; -- assigns 1 immediately