The latest Docker Compose allows you to access environment variables from your compose file. So you can source your environment variables, then run Compose like so:
set -a
source .my-env
docker-compose up -d

Then you can reference the variables in docker-compose.yml using ${VARIABLE}, like so:
db:
  image: "postgres:${POSTGRES_VERSION}"