That's exactly what I did but files are blocking everyone as soon as they are labelled without even sharing the file.
The only difference to your setup is the permissions in that document library are not inheriting from the site and the Member group has no access, just a specific SharePoint group with users.
I have a call with Microsoft open so will update this thread when I have an answer.