“create mask” *removes* permissions. “force create mode” *adds* them. To quote from the man page for smb.conf
This parameter [create mask] may be thought of as a bit-wise MASK for the UNIX modes of a file. Any bit *not* set here will be *removed* from the modes set on a file when it is created.
So if you don’t want the o+r bit set use:
create mask = 0770
If you now want to force ug+rwx permissions for all files, *also* use:
force create mode = 0770
Another way of thinking about these is that “create mask” sets an upper bound on the permissions for the file and “force create mode” sets a lower bound, and the actual permissions are in between. (Except that force create mode is applied second, so if it is more permissive than the create mask, force create mode will override create mask.) Note that neither of these apply to directories. The analogous parameters there are “directory mask” and “force directory mode”.