Thursday, July 1, 2010

Implement Opensolaris Production Storage

OpenSolarisLogoXPRT[1]-SMALL.gif

 

 

 

We all know NetApp is a leading storage solution in industry. It provides reliable and performance storage array. We have been use NetApp solution for almost 2 years and never had any failure on it.

NetApp is excellent product. However it's also an expensive hardware. due to limited budget, we have to find another way to implement storage that  has similar functionality but with lower cost.

In first few days of project, I have tried Openfiler, and Nexenta, but didn't make it out. Openfiler is easy to setup, but while doing CIFS share, i encountered problem in setting windows permissions. Nexenta got pretty interface. Somehow while i change NFS configuration, it didn't applied. Finally i turned to OpenSolaris with ZFS.

I have created couple storage servers by using OpenSolaris box. They are running on production environment severing storage to Oracle database and on traffic web servers. Looks pretty stable so far. Performance is around 10 ~ 30 % slower than our current NetApp storage, but cost is , you know.

Following i will share steps that how we created those OpenSolaris storage.

Share Planning First you have plan how many shares needed in each box and what type of share each share point is. in my case, there are 2 storage box. Storage I got 2 mix share. Same share goes both NFS and CIFS protocol.  Storage II got 2 MIX share and 2 NFS share.

Production Filer Mount Structure.png

Config Network Interface Configure box DNS and running following command to refresh the change.

cp /etc/nsswitch.dns /etc/nsswitch.conf

Edit resolv.conf and add domain information in it. Ex :  "domain abc.com" "search  abc.hq"

nano /etc/resolv.conf

Make sure default routing is going to right network interface if you have 2 or above Network Interfaces.

netstat -rn

Edit host file (/etc/hosts) to contains all servers that need to access to it. Ex: 10.0.1.1 DB10.0.1.2 ApplicationServer-I 10.0.1.2 ApplicationServer-II

nano /etc/hosts
Install the Solaris CIFS service packages
Download SMB package from internet.
pkg install SUNWsmbskr pkg install SUNWsmbs
Load the smbsrv module.
add_drv smbsrv
Import the Solaris CIFS service SMF manifest.
 svccfg import /var/svc/manifest/network/smb/server.xml
Sync time with DNS server
ntpdate runner.zzf.hq
Install PAM module. Add line "other password required pam_smb_passwd.so.1 nowarn" to /etc/pam.conf
nano /etc/pam.conf
Enable CIFS service
svcadm enable smb/server
You have to reboot server after you perform previous configuration.
reboot
Configure CIFS with AD
Set up Kerberos 5. fill following information in krb5.conf:
default_realm = ZZF.HQ
[realms]
ABC.HQ = {
kdc = your ad server ip or name
admin_server = your ad server
kpasswd_server = your ad server
kpasswd_protocol = SET_CHANGE
}
nano  /etc/krb5/krb5.conf
SMB join ActiveDirectory
smbadm join -u administrator abc.hq

Create User Create Unix users

// make sure all UNIX system user ID group ID are same. useradd -u 999 oracle  useradd -u 5501 web 
Perform ID mapping on administrator and other users.
idmap add 'wingroup:administrators' 'unixgroup:sysadmin'
idmap add 'winname:administrator@zzf.hq' 'unixuser:root'
// restart service after mapping is done. i actually encountered that ID mapping doesn't always work while update mapping list.
svcadm restart smb/server; svcadm restart idmap 
Setup ZFS
Create ZFS Pool
// user format to check your disk id format  zpool create -f  laketahoe c9d0
Create Shares Points
zfs create -o casesensitivity=mixed laketahoe/mixshare-I
zfs create -o casesensitivity=mixed laketahoe/mixshare-II
zfs create -o casesensitivity=mixed laketahoe/mixshare-III
zfs create -o casesensitivity=mixed laketahoe/mixshare-IV
zfs create laketahoe/nfsshare-I
zfs create laketahoe/nfsshare-II
Share
Create NFS Share
zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@10.0.0.1/24" laketahoe/mixshare-I
zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@10.0.0.2/24" laketahoe/mixshare-II
zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@10.0.0.1/24" laketahoe/mixshare-III
zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@10.0.0.2/24" laketahoe/mixshare-IV
zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@10.0.0.1/24" laketahoe/nfsshare-I
zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@10.0.0.2/24" laketahoe/nfsshare-II

Create CIFS Share
chmod 777 /laketahoe/mixshare-I chmod 777 /laketahoe/mixshare-II chmod 777 /laketahoe/mixshare-III chmod 777 /laketahoe/mixshare-IV

zfs set sharesmb=on laketahoe/mixshare-I

zfs set sharesmb=on laketahoe/mixshare-II zfs set sharesmb=on laketahoe/mixshare-III zfs set sharesmb=on laketahoe/mixshare-IV
Mount Share
check available NFS mount point. go to machine that
 showmount -e [filer host name]
mount NFS
mount filer:/laketahoe/mixshare-I /mnt/mixshare
at this point , you should be able to see that NFS is mounted
 df -h
 
you can also mount those mix share from windows box as long you use windows administrator to set right permission on those share.
those are steps are steps that make shares work. for permission setting and detail about those configuration, following are good references to go.

References
Developer Recipes: Setting Up an OpenSolaris Storage Server in 10 Minutes or Less ~ By Dominic Kay, April 29, 2008
http://developers.sun.com/openstorage/articles/opensolaris_storage_server.html
HOW to INSTALL and SHARE a File System to CIFS Clients ~ By Brian Leonard, May 22, 2009
http://www.opensolaris.com/use/CIFS.pdf
CIFS_Service_Troubleshooting ~ wiki
http://wiki.genunix.org:8080/wiki/index.php/CIFS_Service_Troubleshooting
Getting Started With the Solaris CIFS Service ~ wiki
http://wiki.genunix.org:8080/wiki/index.php/Getting_Started_With_the_Solaris_CIFS_Service
Solaris CIFS Administration Guide ~ By Sun Microsystems, Inc, March, 2009
http://dlc.sun.com/pdf/820-2429/820-2429.pdf
How to Configure the Solaris CIFS Service in Domain Mode ~ Oracle
http://docs.sun.com/app/docs/doc/820-2429/configuredomainmodetask

Troubleshooting Guide

 

Encountering "Permission Deny" while mounting NFS

-> check nfs mount ip address and mask

zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@192.168.1.201/24" laketahoe/oradatawp



 

"NFS compound failed for server 192.168.1.17: error 27 (RPC: Received disconnect from remote)" message shown while mounting NFS

-> check nfs mount ip address and mask

zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@192.168.1.201/24" laketahoe/oradatawp

 

 

Encountering “nfs mount: mount: /local/httpd/htdocs: Device busy” message while mountng NFS

-> check nfs mount ip address and mask

zfs set sharenfs="anon=0,rw=@192.168.1.201/24,root=@192.168.1.201/24" laketahoe/oradatawp

 

 

Showing "nobody" in folder ownership on NFS mount

change following domain to map ZZF.HQ

#vi /etc/default/nfs

#vi /etc/resolv.conf

#vi /var/run/nfs4_domain

#svcadm restart nfs/mapid

No comments:

Post a Comment