With Windows, I have no problems accessing any of the SMB shares on my OI file server, with OSX, not so much. OS X's tendency to disconnect from SMB is pretty well documented, and a topic of popular conversation among people at work, and anyone I know not using a 100% Apple network.

tl;dr

Getting everything installed and working properly for <10.7 OS X clients was as simple as following the netatalk installation instructions from here, and configuring my /etc/netatalk/afpd.conf file with the following:

# default:  
--tcp -noddp -uamlist uams_dhx.so,uams_dhx2.so,uams_guest.so,uams_clrtxt.so -nosavepassword

Setting up PAM wasn't something I had to do, and my installation was in /etc/ not in /usr/local/etc/. But other than that, everything matches up to the installation tutorial. I also had to set nbmand for all my ZFS FileSystems to off. All four of the mac clients here connect without issue to the public shares.

I had some permissions issues accessing it, but I now blame that on the fact that I had to reinstall OI recently and I think the permissions from the last install were conflicting with the new one.

Time Machine on OSX 10.6, no problems either. 10.7, not so much. Getting it working with Lion was a bit more effort. The way that Lion deals with TimeMachine is different than all previous TimeMachine supported versions of OSX. This means that despite the two Snow Leopard and one Leopard working flawlessly, the Lion one doesn't even see anything as a possible TimeMachine disk.

The article here outlines how to get it all working. However, there's a lot of stuff that I didn't do, did differently, or did and then undid after it broke things. The only thing that was actually required was creating the sparsebundle file on the share after mounting them normally:

hdiutil create -size [initial size, with unit, /m/g/t, (100g)] -fs HFS+J -volname "Time Machine" [HOSTNAME]_[MACADDRESS].sparsebundle 

And then setting OSX to use non-supported network drives for TimeMachine:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Setting up the AFP shares, and getting them showing up in Bonjour is akin to creating a samba share in smb.conf for anyone who has ever done that in any other *NIX variant. All you need to do is add them to the end of your AppleVolumes.default file in /etc/netatalk like so. Setting some defaults in the file cleans up adding the individual shares:

# The line below sets some DEFAULT, starting with Netatalk 2.1.  
:DEFAULT: cnidscheme:dbd options:upriv,usedots,acl perm=0744  
  
# The "~" below indicates that Home directories are visible by default.  
# If you do not wish to have people accessing their Home directories,  
# please put a pound sign in front of the tilde or delete it.  
  
#~ cnidscheme:dbd options:usedots,invisibledots,upriv fperm:0660 dperm:0770  
/Path/To/Directory "Share Name"  
/Data/Movies "Movies"  
/Data/Home/Emma "Emma"  
# End of File

Guest access works out of the box, and connecting with a local account gives you rights to whatever the ACL says.

I had the 4 OSX clients doing their initial TimeMachine backup earlier tonight at the same time, while two people were streaming video from the same pool. No stuttering, no dropping video, no delays in browsing. I'm pretty impressed with ZFS and OpenIndiana right now.