I was hit by an apparent bug in OpenIndiana that was fixed in an older release of Solaris which doesn't fully set permissions when you use the default chmod in OI.  Because of this, I couldn't delete or modify files via the CIFS shares, and possibly the AFP shares. I say possibly because as soon as I found the fix, I shotgunned it across my data directory, and now the permissions appear to be fixed. Logging in locally via SSH was fine, but via SMB/CIFS only adding files worked.

tl;dr

After reading this post, and figuring out that the original poster's problem was the exact same issue I was having, that the extended ACEs weren't being set, and that there was no permission inheritance happening. Despite the normal ls (/usr/gnu/bin/ls) said everything was fine:

$ ls -lah  
ls: cannot access Emma: Permission denied  
total 68K  
drwxrwxrwx    9 robbie staff   9 2012-01-07 03:00 .  
drwxr-xr-x   26 root   root   28 2012-01-06 17:01 ..  
drwxrwxrwx    2 robbie staff   3 2011-10-19 03:59 .$EXTEND  
???????????   ?        ?       ? ?          ?     Emma  
drwxr-----+   3 robbie staff   4 2012-01-08 16:39 Home  
drwxr--r--+ 365 robbie staff 366 2012-01-08 18:47 Data  
drwxr--r--+   8 robbie staff   9 2012-01-08 19:35 Music

What was actually set was:

$ /usr/bin/ls -lVd *  
Emma: Permission denied  
drwxr-----+   3 robbie   staff          4 Jan 8  16:39 Home  
                  [email protected]:rwxp--aARWcCos:-------:allow  
                  [email protected]:------a-R-c--s:-------:allow  
               [email protected]:------a-R-c--s:-------:allow  
drwxr--r--+365  robbie   staff        366 Jan 8  18:47 Data  
                  [email protected]:rwxp--aARWcCos:-------:allow  
                  [email protected]:r-----a-R-c---:-------:allow  
               [email protected]:r-----a-R-c---:-------:allow  
drwxr--r--+   8 robbie   staff          9 Jan 8  19:35 Music  
                  [email protected]:rwxp--aARWcCos:-------:allow  
                  [email protected]:r-----a-R-c---:-------:allow  
               [email protected]:r-----a-R-c---:-------:allow

In the owner permissions:

[email protected]:rwxp--aARWcCos:-------:allow

This means the file owner has

read|write|execute|that it's a named pipe|no delete|no Delete child| read attributes| write Attributes|Read acl|Write acl|(I have no idea what the two 'C's are|Write owner|synchronize

And that those permissions propagate based on file and directory, and no other ways.

So I needed to change it with /usr/bin/chmod instead of the default /usr/gnu/bin/chmod. To change it to typical 700 permissions:

/usr/bin/chmod [email protected]:full_set:fd:allow,[email protected]:read_set:fd:deny,[email protected]:read_set:fd:deny Home

There may be a better way of setting the deny permissions, but this works.

After setting the extended ACL, everything worked fine.

This is all assuming that your ZFS file system is inheriting and using ACL permissions. If aclmode and aclinherit are set to discard, this shouldn't matter.

EDIT: specifying aclmode/aclinherit requirements.