When creating your ActionBarDrawerToggle, simply remove the toolbar argument. This will allow the back button to call onOptionsItemSelected() like normal instead of the hamburger menu.

ActionBarDrawerToggle(this, binding.drawer, R.string.open, R.string.close)