System administration of CKM Debian Linux Cluster

1. System Allocations
2. BIOS settings
3. Making system secure
4. Adding user to the system
5. Compiling kernel
6. Setting up X11
7. Setting up PAM
8. Setting up email (exim)
9. Setting up OpenAFS
10. Setting up kerberized CVS
11. Mandatory seurity notice
12. Setting up DVD player
13. Adding USB support
14. Adding support for lm-sensors
15. Changing sizes of partitions
16. Setting up applications for File Manager
17. Setting up UPS/UPD from scratch
18. Recovering backup files

System Allocations / Locations / Addresses / Accounts

• Name----IP Address------Location--------Accounts
• ckm00 131.225.54.187 wh13e-1330 ckm sysadmin kushnir
• ckm01 131.225.54.189 wh13e-1336 ckm sysadmin pcooper
• ckm02 131.225.54.190 wh13w-1358 ckm sysadmin hwhite
• ckm03 131.225.54.192 wh13e-1328 ckm sysadmin bellanto
• ckm04 131.225.54.193 wh13e-1330 ckm sysadmin coleman
• ckm05 131.225.54.194 wh13e-1334 ckm sysadmin hogann
• ckm06 131.225.54.195 wh13w-1361 ckm sysadmin caroline
• ckm07 131.225.54.196 wh13e-1336 ckm sysadmin ramberg
• ckm08 131.225.54.197 wh13e-1328 ckm sysadmin kobilar mastner
• ckm09 131.225.54.199 wh13e-1332 ckm sysadmin tsch
• ckm10 131.225.54.198 wh13e-1330 ckm sysadmin
• pc781b 131.225.53.167 wh13w-13?? selex guest
• pc781c 131.225.53.168 wh13w-13?? selex guest
• pc781d 131.225.53.169 wh13w-13?? selex guest
• pc781e 131.225.53.170 wh13w-13?? selex guest

BIOS settgins

ckmxx

• Boot order Floppy - CDROM - IDE0
• ECC Error checking Correct + Scrub
• Supervisior Password Set for System (so that users can not play with BIOS)
• State after AC failure Last state. This is power management option.

Making system secure.

• Install /etc/hosts.allow /etc/hosts.deny
• Adjust /etc/ssh/ssh_config.

  
  Host *
     ForwardAgent yes
     ForwardX11 yes
  
  ... snip ...
  # Kerberos options
  # Ticket forwarding is disabled by default as forwarding tickets
  # to an untrusted system can be very bad.
  # enable for systems you trust.
  KerberosAuthentication yes
  KerberosTGTPassing no
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes
  

• Adjust /etc/ssh/sshd_config

Adding user to the system.

• Find out user UID at Fermilab. This is a unique number that should be the same across all Fermilab computers. You can extract it from /etc/passwd file on fnppd.fnal.gov computer for example.

  caroline:!:2658:1706:Caroline Milstene:/afs/fnal/files/home/room1/caroline:/bin/csh
             ^^^^ 

• Create a user account. GID should be the as for CKM group 5460. Ask user about his preferable shell (or guess it looking in fsgi03 or fnppd)
  adduser --shell /bin/tcsh --uid 2658 --gid 5460 caroline
• Modify file /etc/email-addresses to force outgoing mails have an official persons email address. We do not want people to reply to username@ckmxx.fnal.gov type of addresses.

  
  	    ckm06:~# cat /etc/email-addresses
  	    # This is /etc/email-addreses. It is part of the exim package
  	    .... snap ....
  	    caroline: caroline@fnal.gov
          

• Add user into audio group modifying /etc/group file

  
  	    ckm06:~# cat /etc/group
  	    .... snap ....
  	    audio:x:29:sysadmin,ckm,caroline
  	    .... snap ....
         

Compiling kernel.

Kernel should be compiled on ckm00 computer in //usr/src/kernel-source-2.4.16 directory. Command make-kpkg is very helpful

1. Make a cleanup
   # make-kpkg clean
2. Prepare kernel configuration. You may want to start from already existing configurations:
   • config.ckm00 - for ckm00 computer
   • config.ckmxx - for ckm01..ckm10 computers
   • config.pc781x - for pc781b..pc781e computers
   
   # cp config.ckmxx .config
   # make menuconfig
3. Compile the kernel and prepare Debian package with the kernel image
   # make-kpkg --revision=ckmxx.1.1 kernel_image
4. Compile the kernel and prepare Debian package with the modules images (openafs, lm-sensors and so on)
   # make-kpkg modules_image
5. Goto upper directory and copy relevant *.deb files to /usr/ckm/debian directory
6. Install kernel-image on all computers
   # slogin ckm01
   # dpkg -i /usr/ckm/debian/kernel-image-2.4.16_ckmxx.1.1_i386.deb

Hint: If during compilation some header files are missing you may need to install some lib*-dev.deb pacakges. You can use Debain Packages contents search page to find appropriate Debian package.

Setting up X11

1. Standard /etc/X11/Xsession from Debian distribution is fine
2. To run wincenter we have to allow Xserver to listen to TCP Modify /etc/X11/xdm/Xservers (we start X only through xdm) and take out -nolisten tcp .
3. Syncronize default X application. Those are programs that X will launch at the beginnig. These applications are not more than symbolic links. To see and modify them run:
   # update-alternatives --display x-window-manager
   # update-alternatives --config x-window-manager

   application	choice
   x-session-manager	/usr/bin/gnome-session
   x-window-manager	/usr/bin/enlightenment
   x-terminal-emulator	/usr/X11R6/bin/xterm

Setting up PAM.

The idea of this setup is to allow user to use both Kerberos and local passoword to login into the system. Getting AFS token is also important if user uses kerberos passowrd.

• Following packages should be installed:

  Package name	Brief Description
  libpam-krb5	PAM module for MIT Kerberos
  libpam-modules	Pluggable Authentication Modules for PAM
  libpam-openafs	PAM Module to get AFS tokens and set up PAG
  libpam-runtime	Runtime support for the PAM library
  libpam0g	Pluggable Authentication Modules library

• Modify /etc/pam.d/login and /etc/pam.d/xdm . First one is responsible for TTY logins (rarely used), the second one for XDM logins.

  
  	       Original  
  	      > auth       required     pam_unix.so nullok
  	       Replacement 
  	      < auth       sufficient   pam_krb5.so 
  	      < auth       sufficient   pam_unix.so nullok try_first_pass
  	      < auth       required     pam_deny.o
  	      < session    optional     pam_openafs_session.so 
            

• Here are login and xdm files just for reference.
• ToDo list It would be quite useful to modify xscreensaver so that one can type Kerberos password to unlock the screen and update Kerberos and AFS tokens. At the moment this xscreensaver would let you in, but would not create/update kerberos ticket

Setting up email (exim).

Fermilab Run eximconfig and choose option 1 (Internet site). Hit return for all questions except for Replace /etc/aliases file (keep original file).

Adjust file /etc/email-addresses so that outgoing mail has format user@fnal.gov

IHEP Run eximconfig and choose option 2 (Internet site using smarthost). Hit return for most questions except for smarthost (choose mx.ihep.su) and Replace /etc/aliases file (keep original file).

Adjust file /etc/email-addresses so that outgoing mail has format user@mx.ihep.su

Setting up OpenAFS.

Following packages also have to be installed:

libpam-openafs-session	To generate AFS token during login
openafs-client	To access AFS file system
openafs-krb5	Kerberos 5 support in OpenAFS
openafs-modules-2.4.16	Kernel OpenAFS modules

It is usefull to limit number of Cells that you can access from a single computer. Reasonable set is in /etc/openafs/CellServDB

Setting up kerberized CVS.

1. Get debian package cvs-sourceand go to some working directory # apt-get source cvs
2. There will be a directory tree with applied patches and all ready to be compiled. Go there
3. There is a directory debian inside. Edit file debian/rules . Somewhere inside it there is a configuration line. Add flag --with-gssapi

   
           ./configure --prefix=/usr  --sysconfdir=/etc \
                   --mandir=\$${prefix}/share/man \
                   --infodir=\$${prefix}/share/info --with-gssapi
   	                                         ^^^^^^^^^^^^^
   

4. Compile the tree
   # debian/rules configure
   # debian/rules build
   # debian/rules binary

   You may need some extra packages ( dbs and development libraries mostly, for example libkrb5-dev) for successfull compilation. Install them. Basically configure will complain and you would see what is absent. Run debian/rules clean before each try.

5. As a result you will get debian package in the upper directory. Install it
   
   cd ..
   dpkg -i cvs_1.11.1p1-3_i386.deb

   In configuration stage don't forget to enable pserver . This port is used for kerberos authentication.

6. Restart inetd service
   /etc/init.d/inetd restart
7. Check that cvs-pserver is allowed in /etc/hosts.allow
8. Define correct setting for CVSROOT in /etc/environment
   CVSROOT=:gserver:ckm00.fnal.gov:/usr/ckm/cvsroot

Mandatory Security notice.

Setting up DVD player

1. Creadte symbolic link that points to DVD ROM IDE device
   # ln -s /dev/hdd /dev/dvd
2. Install packages ogle and ogle-gui While instaling it you would get message that you need to run /usr/share/doc/ogle/examples/install-css.sh . Do it after packages are installed.
   # apt-get install ogle ogle-gui
   # /usr/share/doc/ogle/examples/install-css.sh

1. Find out which USB adapter you are using OHCI or UHCI. You can do it carefully reading output of lspci -v | grep -i usb
   00:07.4 USB Controller: Advanced Micro Devices [AMD] AMD-765 [Viper] USB (rev 07) (prog-if 10 [OHCI]) So in this example it is clearly OHCI. In make menuconfig stage of kernel configuration go USB support --> OHCI . Put it in as a module if you think that it is going to be rarely used. Compile it in otherwise. As most of our mice are USB it is worse compiling in.
2. Add USB mouse support. It carries encyptic name Human Interface Device (HID). This option is not seen under USB support, unless you turn on Input core support in the main menu. Compile in Mouse support in that submenu.
   Now go back to USB support and you will see that Human Interface device options showed up. Choose only USB Human Interface Device (full HID) support
3. Compile in Preliminary USB device filesystem
4. You may also choose USB Scanner support as a module.
5. Go ahead and compile the kernel.
6. Install packages usbmgr usbutils usbview
7. Create USB devices and INPUT devices (USB mouse)

   
   # cd /dev
   # ./MAKEDEV usb
   # ./MAKEDEV input
   

   Check that mouse works: do cat /dev/input/mice and move the mouse.

8. Make X11 aware of the USB Mouse adding to /etc/X11/XF86Config-4

   
   Section "InputDevice"
        Identifier  "USB Mice"
        Driver      "mouse"
        Option      "Protocol"   "IMPS/2"
        Option      "Device"     "/dev/input/mice"
        Option      "ZAxisMapping"   "4 5"
        Option      "Buttons"        "5"
   EndSection
   
    ... snap ...
   
   Section "ServerLayout"
       Identifier  "Simple Layout"
       Screen 	"Screen 1"
       InputDevice "USB Mice"    	"CorePointer"
       InputDevice "Keyboard1"   	"CoreKeyboard"
   EndSection
   
   

Adding support for lm-sensors

lm-sensors can provide information about CPU temperature and FAN rotation speeds. Can be quite useful. Check official lm-sensors page for details.

Note pc781[b-e] do not have sensors installed on motherboard.

1. Compile lm-sensors and i2c drivers. You would do it on ckm00 computer. You need packages lm-sensors-source and i2c-source. Usually you would do it each time you compile kernel.

   
   # cd /usr/src/kernel-source-2.4.17
   # make-kpkg clean
   # make-kpkg --revision=ckmxx.1.2 kernel_image
   # make-kpkg modules_image
   

2. Move newly created debina packages to /usr/ckm/debian . Slogin to ckmxx computer and install compiled modules

   
   # dpkg -i /usr/ckm/debian/i2c-2.4.17_2.6.2-1+ckmxx.1.2_i386.deb 
   # dpkg -i /usr/ckm/debian/lm-sensors-2.4.17_2.6.2-2+ckmxx.1.2_i386.deb 
   

3. Install packages to support lm-sensors: lm-sensors sensord
4. Insert necessary modules. You can try sensors-detect but it sucks. For ckmxx it gave me completely wrong answer. You are better off carefully reading lspci -v and reading Motherboard manual. Or search the Web :)

   computer	ckm00	ckmxx	pc781x	oka08
   motherboard	Tyan Thunder K7 S2462N	Gigabyte GA-7DXR	FIC VL-601
   monitoring chip	W83627HF	VT82C686A	PII4X
   modules	i2c-amd756	i2c-viapro
   	i2c-isa	i2c-isa
   	w83781d	via686a
   /etc/sensors.conf		sensors.conf

Changing sizes of partitions

Useful tool is called GNU parted (package parted). It is very easy to use. Check official web page details. You are usually going to resize partitions. There is a catch : you can move the "tail" of partition not its "beginning". I used this tool several times to create additional Linux partitions, creating dual boot partitions and so on. No problems so far.

1. Backup data! You don't want to be screwed.
2. Unmount disk (all partitions). If it is shared with NFS use:
   # /etc/init.d/etc/init.d/nfs-kernel-server stop
3. Run parted . Type help and read.
4. Restart NFS if necessary
   # /etc/init.d/etc/init.d/nfs-kernel-server start
5. Mount all partitions that belong to this disk.

Setting up applications for File Manager

If you open File manager and click on a file icon certain applications should start to process those files (acroread, gv, mozilla...). Sometimes choices in official distribution are odd - xpdf for example. Tweak the file /usr/share/mime-info/mc.keys . Changes will be used by all users (unless they overwrote them for their account).