Enable iCloud contacts, even if you don't use them.

You need to have iCloud enabled for groups to be able to modify groups at all—even if you're not using iCloud as your primary means for storing contacts.

My theory is that Apple doesn't use Google's contacts API to sync group information, so it needs iCloud to tack group functionality onto Google-synced contacts.

——

Update: This isn't a true fix. It's made it so some contacts can be removed from groups and so some groups can be deleted, but it's unpredictable and incomplete.