You can make 2 querys to get both groups of users that you want. I mean:

1) Users who dont have any role.

User::whereDoesntHave('roles')->get();

2) Users have any role, but not admin (from Adam Kozlowski answer)

$this->users()->whereHas('roles', function ($query) {
    $query->where('name','!=', 'admin');
});