Setting up Kerberised NFSv4 server on Solaris ZFS



My Environment

  • NFS Server: blackhole.dfusion.com.au (Solaris 10 – specifically 5.11, NexentaOS_20090926)
  • KDC Server: blackhole.dfusion.com.au (ie same box)
  • Domain name: dfusion.com.au

Having the NFS and KDC Servers being the same box does simplify the install because I don’t have to set up the NFS server as a KDC client.
This was for pragmatic reasons. In my ideal world the KDC server would be separate from the file server.


Dependencies

  • You need a domain name and use it consistently. For an informal (eg home) network something like "local" or "home.local” should be ok.
  • DNS (Domain Name Server) lookup must work consistently across your machines
  • NTP (Network Time Protocol) – all machines must show the same time

Useful Docs


1 Solaris KDC Server Setup


1a) Install Kerberos if needed
Under Nexenta: apt-get install sunwkdcu

NOTE: I installed the Solaris version of MIT kerberos (package sunwkdcu) as opposed to core MIT version krb5-*
Both should be near identical. Solaris version claims to have some NFS performance optimisations.


1b) Find kdcmgr script (/usr/sbin possibly) that does a lot of the legwork for you. I’ll assume it came with your package.

1c)# kdcmgr status
This has no effect except to demonstrate that the kerberos packages are installed and not yet configured (it will complain about missing config files etc. Don’t worry.)
If the command seems to indicate kerberos is already running “State: online since DATE…” then you’d better stop and find out why it is already configured and running….

1d) # /usr/lib/krb5/klookup
For my system this returns: blackhole.dfusion.com.au
If you do not get the FQDN of your server then STOP and configure DNS properly (eg /etc/resolv.conf etc)

NOTE: You need domain name resolution to work (even if it is via consistent /etc/hosts on all machines). If using /etc/hosts ensure the lines are laid out as
192.168.0.99 blackhole.dfusion.com.au blackhole
with the FQDN coming before any short aliases


1e) # kdcmgr create

NOTE: There is a man page “man kdcmgr” also you can give it command line arguments but it asks for the things it needs by default.

NOTE: On my system it appears the smf definition for kpropd is missing – an install bug somewhere no doubt.
Now I’m not using kpropd (for slave Kdcs) so I don’t care for now BUT the kdcmgr script dies. So in this case I just copied the kdcmgr script to kdcmgr_hacked and commented out the 8 lines in the kill_daemons script that refer to the kpropd and used my version for any create/destroy commands. Status works with the original script.


Answer the questions as directed below.
Realm: DFUSION.COM.AU
NOTE: Use UPPER CASE version of YOUR domain name in most cases - I think this is more for sanity to distinguish between realms and domain names, but some systems may also require it.


KDS database master key: xxxxxx
NOTE: Enter a high quality password here and don’t lose it. You are asked to enter it twice.

Administration principle: root/admin
Administration principle password: xxxxx

NOTE:
* This is the username and password you will have to type when you are administering kerberos.
* The format is principal/instance. The instance is “admin” by convention in this case.
* Some documents suggest “root/admin” others their username eg “kim/admin”. I have not yet found whether it matters. I tried both and ended up using root since I typically administer it when I’m root. For a small installation it probably doesn’t matter. With multiple administrators a person’s actual username is probably better.


The command should complete with a happy message like “Setup COMPLETE”
Any old versions of the file are renamed krb5.conf.sav. You can delete these later if you don’t need them.
A partial extract of my krb5.conf file shows the sort of things to expect:

[realms]
   DFUSION.COM.AU = {
       kdc = blackhole.dfusion.com.au
       admin_server = blackhole.dfusion.com.au
  }
 
[domain_realm]
  .dfusion.com.au = DFUSION.COM.AU


Here you see the DFUSION realm has the master key server machine (kdc) as blackhole
and the DFUSION realm is mapped to the dfusion domain name.

COMMENT: I think the kdcmgr script has a small omission/bug in it. Maybe it was delberate.
kdcmgr creates a permissions file /etc/krb5/kadm5.acl for what the relevant principals can do within kadmin.
The kdcmgr script I have sets the “root/admin” entry to “acmil”.
This omits the “d”elete permission which I later needed.
The file can be edited to include “d” eg change the entry to “adcmil” or change it to “*” meaning all permissions (no quotes).
Save and quit.
Having said this my change didn’t work but when I ran kadmin.local I could delete entries even with an unchanged acl file.


1f) The previous command also tries to start the kerberos daemons via smf (eg kadmind, krd5kdc).
Check the daemon is running:
# kdcmgr status
It should say “State: online since …”
If it says something like maintenance or disabled then look in the log file indicated.
My log file is: /var/svc/log/network-security-kadmin:default.log
NOTE: Once smf places the daemon in maintenance mode it must be restarted by first clearing the error flag:
# svcadm clear svc:/network/security/kadmin:default
# svcadm enable svc:/network/security/kadmin:default


NOTE: svcadm COMMAND kadmin also works as a shortcut service name

To check it is running (”online”):
# svcs -l svc:/network/security/kadmin:default
or
# kdcmgr status
NOTE: If you mucked up the entries to kdcmgr script you can redo the whole thing without grief.
# kdcmgr destroy


2 Solaris KDC and NFS Server Key Management


Checkpoint: KDC should now be up and running on the KDC server. This will be confirmed when we try to administer it:
There are two kadmin commands: kadmin and kadmin.local

Note: When running on the KDC box you can use kadmin.local
(using this also avoided a delete permissions permission problem that I had.)

The username/admin is the default principal if -p is omitted. So if you are root for example and used the root/admin principal you should just be able to run “kadmin” with no arguments.


2a) On the KDC server run the kerberos admin tool ("?" for help, "quit" to exit).
# kadmin -p root/admin+-
 Authenticating as principal root/admin with password.
 Password for root/admin@DFUSION.COM.AU:  xxxxxx

kadmin:  listprincs

K/M@DFUSION.COM.AU
changepw/blackhole.dfusion.com.au@DFUSION.COM.AU
kadmin/blackhole.dfusion.com.au@DFUSION.COM.AU
kadmin/changepw@DFUSION.COM.AU
kadmin/history@DFUSION.COM.AU kiprop/blackhole.dfusion.com.au@DFUSION.COM.AU
krbtgt/DFUSION.COM.AU@DFUSION.COM.AU
root/admin@DFUSION.COM.AU

These are the ones I had (after running the system for a while). Your list should be similar but don’t worry if it is different.
This lists all the principals (identities) that are defined. The current ones were added by kdcmgr, but now we must add a few more.
We are about to add the equivalent of:
host/blackhole.dfusion.com.au@DFUSION.COM.AU
nfs/blackhole.dfusion.com.au@DFUSION.COM.AU
kim@DFUSION.COM.AU

I’ve added three types of principals (identities) here:
  • host/xxxx@xxxx — the “host” principal is used to authenticate a system rather than a specific user
  • nfs/xxxx@xxxx — the "nfs" principal is used to authenticate the NFS service
  • user@xxxx — a human such as "kim". “kim” is my username on both the client box and server.

Note “kim” is different to “kim/admin”. Administration is kept separate from normal user authentication to prevent accidents, so both are needed.


Before continuing there is something important to understand. It can be summarised as follows:
ONLY perform kadmin: addprinc -randkey service/host.domain on the server that IS host.domain.
The addprinc -randkey command does two things:
  1. adds the named identity into the master Kerberos DataBase (KDB) on whatever (remote) server the KDC is located.
  2. adds the identities (random) secret password into the local keytab file (typically /etc/krb5/krb5.keytab )
Only the server providing the specified service should know its secret password. BTW It is random simply to save humans thinking up secure passwords they won't ever need to use.
In the case of human users, the password is NOT stored in the local keytab file, instead it is stored in the master KDB.
NOTE: It is important on which server the addprinc -randkey commands are typed.


So for example:
  • host/myhost.... should only be created on myhost (so the secret password gets put into myhost:/etc/krb5/krb5.keytab)
  • nfs/nfshost.... should only be created on nfshost (so the secret password gets put into nfshost:/etc/krb5/krb5.keytab)

Thus on each server (and possibly each client but some clients don't seem to use "host/") you probably ought to create a:
  • host/myhost.... should only be created on myhost (so the secret password gets put into myhost:/etc/krb5/krb5.keytab)
and on the NFS server:
  • nfs/nfshost.... (the secret password gets put into nfshost:/etc/krb5/krb5.keytab)

If your KDC and NFS host happen to be the same machine then of course they will just be done together.

At some stage you may also need to create machine specific root identities just as you did for host/*. The distinction between use of host/ and root/ within applications is unclear (to me at least):
  • root/myhost.... [-randkey] - representing the machine root (or root user). Human password or machine ???

To do the above on each server, I just login to the relevant server and do:
$ kadmin -p  kim/admin
kadmin:   addprinc ...
kadmin:   ktadd ...
kadmin: quit


The admin tool reads the krb5.conf file to find out how to contact the kdc. so that needs to be correct.

If you cannot run kadmin on the relevant server for some reason, there are fiddly ways around creating a server specific keytab file and then moving it to the server but it's harder.


To add these entities do the following within the kadmin tool command line:
Example of my system
kadmin:  addprinc -randkey host/blackhole.dfusion.com.au
kadmin:  ktadd host/blackhole.dfusion.com.au

kadmin:  addprinc -randkey nfs/blackhole.dfusion.com.au
kadmin:  ktadd nfs/blackhole.dfusion.com.au

kadmin: addprinc kim
 (Enter kim’s desired kerberos password) 
kadmin: listprincs
kadmin: quit


2b) Having done this we can check the list of _local_ private keys (generated by -randkey) using either:
  • klist - simple list tool
  • ktutil - keytable editor tool
Refer to man pages for usage.
# klist -ke

KVNO Principal
---- --------------------------------------------------------------------------
   3 host/blackhole.dfusion.com.au@DFUSION.COM.AU (AES-256 CTS mode with 96-bit SHA-1 HMAC) 
   3 host/blackhole.dfusion.com.au@DFUSION.COM.AU (AES-128 CTS mode with 96-bit SHA-1 HMAC) 
   3 host/blackhole.dfusion.com.au@DFUSION.COM.AU (Triple DES cbc mode with HMAC/sha1) 
   3 host/blackhole.dfusion.com.au@DFUSION.COM.AU (ArcFour with HMAC/md5) 
   3 host/blackhole.dfusion.com.au@DFUSION.COM.AU (DES cbc mode with RSA-MD5) 
   3 nfs/blackhole.dfusion.com.au@DFUSION.COM.AU (AES-256 CTS mode with 96-bit SHA-1 HMAC) 
   3 nfs/blackhole.dfusion.com.au@DFUSION.COM.AU (AES-128 CTS mode with 96-bit SHA-1 HMAC) 
   3 nfs/blackhole.dfusion.com.au@DFUSION.COM.AU (Triple DES cbc mode with HMAC/sha1) 
   3 nfs/blackhole.dfusion.com.au@DFUSION.COM.AU (ArcFour with HMAC/md5) 
   3 nfs/blackhole.dfusion.com.au@DFUSION.COM.AU (DES cbc mode with RSA-MD5)

Remember in my case the KDC and NFS server are combined so I only have one server keytab file

Shows the keytab entries (those added via ktadd earlier). There are multiple entries per principal, which brings us to the next Kerberos twist :-).
The machine/service identity passwords generated earlier with -randkey are actually stored multiple times, each one being encrypted with a different encryption cipher (eg des-cbc-crc). Typically there will be around five different ciphers in use.
For now just accept the fact. Similarly the first column KVNO output by keylist can also be important but generally not after a clean new install.

3 Solaris NFS Server (Kerberised ZFS) Configuration


3a) On the NFS server: edit /etc/nfssec.conf
Find the three lines:
krb5 390003 kerberos_v5 default – # RPCSEC_GSS
krb5i 390004 kerberos_v5 default integrity # RPCSEC_GSS
krb5p 390005 kerberos_v5 default privacy # RPCSEC_GSS

Make sure they are uncommented (remove any leading #)
NOTE: krb5 is plain authentication. krb5i adds integrity (a checksum) to ensure nfs data is not tampered with intransit, krb5p is integrity and privacy (it encrypts the nfs data so it can’t be snooped on the wire). Additional security typically adds a performance penalty.


3b) Optional (I haven’t tried this)
Find the line:
default 1 - - - # default is AUTH_SYS

The default nfs security is (only) AUTH_SYS. It seems that to ensure high security and avoid accidents, the “1″ should be changed to 390003.


3c) Configure NFSv4 to use Kerberos
For non ZFS the relevant command is:
share -F nfs -o OPTIONS FILESYSTEM
For ZFS the command is:
zfs set sharenfs=OPTIONS FILESYSTEM
Or one can use sharemgr

where OPTIONS is the same in both cases. See the man pages for share and share_nfs for the options as well as the Sun “System Administration Guide: Security Services” Manual.

The options of immediate interest are:
sec=opts
Security. Default is "sys". Set to say "krb5" or an ordered list eg "krb5
rw[=opts]
read/write access. Default is rw for everyone. Refer man page. eg
ro[=opts]
read only access (to all if specified, else just to specified list


NFS Shared ZFS example
zfs get sharenfs     # to see the current state of play

zfs set sharenfs=sec=krb5:krb5i:krb5p,rw=@192.168 tank/backup

zfs get sharenfs [[tank/backup]    # to see the changes
NAME         PROPERTY  VALUE                             SOURCE
tank/backup  sharenfs  sec=krb5:krb5i:krb5p,rw=@192.168  local

sharemgr show -vp     # another way to show what is actually shared
default nfs=()
zfs
    zfs/tank nfs=()
    ....

    zfs/tank/backup nfs=() smb=(rw="@192.168") nfs:krb5=(rw="@192.168") nfs:krb5i=(rw="@192.168") nfs:krb5p=(rw="@192.168")
	  tank_backup=/tank/backup


Note: To permit traditional insecure unix permission mating (AUTH_UNIX / AUTH_SYS) during the testing/Kerberos transition phase use something like: sec=krb5:sys or sec=krb5:....:sys or sec=sys:krb5:... but remove it later if security is a concern.


Note. Although this article isn't about Samba, it's worth mentioning I also used something like:
zfs set sharesmb=rw=@192.168 tank/backup to create a samba share called tank_backup.




The original document is available at http://dfusion.com.au/wiki/tiki-index.php?page=Setting+up+Kerberised+NFSv4+server+on+Solaris+ZFS