Chapter 2: Server Administration, Security, and Monitoring
Master essential server management tasks including user administration, permissions, system monitoring, backup strategies, and security hardening for production environments
1.1 Linux User Administration
User management is fundamental to server security, enabling proper access control, accountability, and resource isolation in multi-user environments.
Every process runs under a user identity (UID) and group identity (GID). The root user (UID 0) has unrestricted system access. Regular users have limited permissions. Groups enable collective permission assignment.
Essential User Management Commands:
# Create a new user with home directory and default shell sudo useradd -m -s /bin/bash jsmith # Set user password (prompts interactively) sudo passwd jsmith # Create user with specific UID, GID, and comments sudo useradd -u 1501 -g developers -c "John Smith" -m jsmith # Add user to supplementary groups sudo usermod -aG sudo,docker jsmith # Modify user shell sudo usermod -s /bin/zsh jsmith # Lock/unlock user account sudo usermod -L jsmith # Lock sudo usermod -U jsmith # Unlock # Delete user and home directory sudo userdel -r jsmith # View user information id jsmith getent passwd jsmith finger jsmith # if installed
Group Management:
# Create new group sudo groupadd webteam # Add user to group sudo usermod -aG webteam jsmith # Change user's primary group sudo usermod -g webteam jsmith # Remove user from group sudo gpasswd -d jsmith webteam # List group members getent group webteam members webteam # if memcached-tools installed # Delete group sudo groupdel webteam
Best Practice: Always use -aG (append) with usermod to add users to groups. Using -G alone replaces all supplementary groups, potentially removing critical access!
1.2 Windows User and Group Management
Windows Server uses Active Directory (AD) for centralized user and group management in enterprise environments, with local accounts for standalone servers.
Local User Management (PowerShell):
# Create new local user New-LocalUser -Name "jsmith" -Password (ConvertTo-SecureString "P@ssw0rd!" -AsPlainText -Force) -FullName "John Smith" -Description "Developer Account" # Add user to local group Add-LocalGroupMember -Group "Administrators" -Member "jsmith" # Enable/disable account Enable-LocalUser -Name "jsmith" Disable-LocalUser -Name "jsmith" # Set password to never expire Set-LocalUser -Name "jsmith" -PasswordNeverExpires $true # List local users and groups Get-LocalUser Get-LocalGroup Get-LocalGroupMember -Group "Administrators" # Remove user Remove-LocalUser -Name "jsmith"
Active Directory Basics:
- Domain Controllers (DCs): Servers hosting AD database and authentication services
- Organizational Units (OUs): Containers for organizing users, groups, and computers
- Group Policy Objects (GPOs): Centralized configuration and security settings
- Security Groups: Used for permission assignment (e.g., "FileServer-ReadOnly")
- Distribution Groups: Used for email lists, not permissions
AD Best Practice: Follow the AGDLP principle: Add user accounts to Global groups, add Global groups to domain Local groups, assign Permissions to domain Local groups. This simplifies permission management at scale.
2.1 Linux File Permissions
Linux permissions use a three-tier model (owner/group/others) with read, write, and execute bits, plus special bits for advanced control.
Permission Basics:
| Symbol | Octal | Permission | For Files | For Directories |
|---|---|---|---|---|
| r | 4 | Read | View file contents | List directory contents |
| w | 2 | Write | Modify file contents | Create/delete files in directory |
| x | 1 | Execute | Run as program/script | Enter/traverse directory |
Understanding ls -l Output:
# Example output: # -rwxr-xr-- 1 owner group 4096 Jan 15 10:30 script.sh # │ └─┬─┘ └─┬─┘ └─┬─┘ # │ │ │ └── Others: r-- (read only) # │ │ └── Group: r-x (read + execute) # │ └── Owner: rwx (full access) # └── File type: - = regular file, d = directory, l = symlink # View permissions ls -l /path/to/file stat /path/to/file # Change permissions (symbolic mode) chmod u+x script.sh # Add execute for owner chmod g-w file.txt # Remove write for group chmod o=r file.txt # Set others to read-only chmod a+rX directory/ # Add read + execute (dirs only) for all # Change permissions (octal mode) chmod 755 script.sh # rwxr-xr-x chmod 644 file.txt # rw-r--r-- chmod 600 ~/.ssh/id_rsa # Owner read/write only (SSH key) # Change ownership chown user:group file.txt chown -R user:group /directory/ # Recursive # Change group only chgrp developers project/
Special Permission Bits:
| Bit | Symbol | Octal | Purpose | Example |
|---|---|---|---|---|
| SUID | s/S | 4000 | Run file with owner's permissions | /usr/bin/passwd |
| SGID | s/S | 2000 | Run with group permissions; new files inherit group | Shared project directories |
| Sticky Bit | t/T | 1000 | Only file owner can delete in directory | /tmp directory |
Security Warning: Avoid SUID/SGID on scripts and unnecessary binaries. These are common privilege escalation vectors. Audit with: find / -type f \( -perm -4000 -o -perm -2000 \) -exec ls -l {} \;
2.2 Access Control Lists (ACLs)
ACLs extend beyond traditional Unix permissions, allowing granular access control for multiple users and groups on individual files and directories.
ACL Management Commands:
# View ACLs on a file getfacl /path/to/file # Grant read access to specific user setfacl -m u:jsmith:r /path/to/file # Grant read-write to group setfacl -m g:developers:rw /path/to/project # Set default ACL for new files in directory setfacl -d -m u:jsmith:rw /shared/team-folder # Remove specific ACL entry setfacl -x u:jsmith /path/to/file # Remove all ACLs (revert to base permissions) setfacl -b /path/to/file # Recursive ACL application setfacl -R -m g:auditors:r /var/log/ # Example ACL output: # file: /shared/report.pdf # owner: admin # group: finance # user::rw- # user:jsmith:r-- # group::r-- # mask::r-- # other::---
Windows NTFS Permissions vs Linux ACLs:
| Feature | Windows NTFS | Linux ACLs |
|---|---|---|
| Granularity | Very fine-grained (13+ permissions) | Moderate (rwx + extensions) |
| Inheritance | Built-in, configurable | Via default ACLs |
| Management | GUI + PowerShell + icacls | Command-line: getfacl/setfacl |
| Audit | Integrated with Event Log | Requires auditd configuration |
| Effective Permissions | Easy GUI view | Calculate manually or use tools |
Enable ACLs: Ensure filesystem supports ACLs. For EXT4, mount with acl option (usually default). Verify with: mount | grep acl or check /etc/fstab.
2.3 Sudo and Privilege Escalation
sudo allows controlled privilege escalation, enabling users to run specific commands as root or other users without sharing the root password.
Configuring /etc/sudoers Safely:
CRITICAL: Always edit sudoers with visudo command, which validates syntax before saving. A syntax error can lock you out of sudo access!
# Open sudoers file safely sudo visudo # Basic sudo rule syntax: # user/host = (runas_user) [NOPASSWD:] commands # Allow user jsmith to run any command with password jsmith ALL=(ALL:ALL) ALL # Allow user to run specific commands without password jsmith ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx, /usr/bin/tail -f /var/log/nginx/* # Allow group 'developers' to run commands as www-data %developers ALL=(www-data) /usr/bin/php, /usr/bin/composer # Alias examples for maintainability Cmnd_Alias WEBMGMT = /usr/bin/systemctl restart nginx, /usr/bin/systemctl reload apache2 Cmnd_Alias LOGVIEW = /usr/bin/tail -f /var/log/*, /usr/bin/journalctl -u * %webadmins ALL=(ALL) WEBMGMT %auditors ALL=(ALL) LOGVIEW # Require password re-authentication after 15 minutes Defaults timestamp_timeout=15 # Log all sudo commands to syslog Defaults logfile="/var/log/sudo.log" Defaults log_input, log_output # Test sudo configuration sudo -l -U jsmith # List allowed commands for user sudo -v # Validate/refresh timestamp sudo -k # Kill timestamp (force re-auth)
Best Practices for sudo:
- Principle of Least Privilege: Grant only the minimum commands needed
- Avoid NOPASSWD: Require password except for automated tasks
- Use Groups: Manage permissions via groups, not individual users
- Enable Logging: Audit all privileged command execution
- Regular Audits: Review sudoers file quarterly
Monitor sudo usage: Check logs with grep sudo /var/log/auth.log (Debian/Ubuntu) or grep sudo /var/log/secure (RHEL/CentOS). Consider centralized logging with rsyslog or journald forwarding.
3.1 Real-time System Monitoring
System monitoring provides visibility into server health, performance bottlenecks, and resource utilization for proactive management.
Essential Monitoring Commands:
# CPU and process monitoring top # Interactive process viewer htop # Enhanced top with colors/search (install first) vmstat 1 # Virtual memory stats, update every 1s mpstat -P ALL 1 # Per-CPU utilization (sysstat package) # Memory monitoring free -h # Human-readable memory usage cat /proc/meminfo # Detailed memory information # Disk I/O monitoring iostat -x 1 # Extended I/O stats per device iotop # Per-process I/O usage (requires root) # Network monitoring iftop # Bandwidth usage per connection nethogs # Bandwidth per process ss -tunap # Socket statistics (replaces netstat) # System load and uptime uptime # Load averages: 1min, 5min, 15min w # Who is logged in + load # Quick system summary glances # All-in-one monitoring dashboard
Understanding Load Averages:
Load average represents the average number of processes in runnable or uninterruptible state. Rule of thumb: Load < CPU cores = healthy; Load > 2× cores = investigate. A single-core system with load 1.0 is at 100% capacity.
3.2 Logging Fundamentals
System logs provide an audit trail for troubleshooting, security analysis, and compliance reporting.
Linux Logging Architecture:
- rsyslog/syslog-ng: Traditional syslog daemons for log collection and routing
- journald: systemd's binary logging system with structured metadata
- Log files: Text logs in
/var/log/(auth.log, syslog, kern.log, etc.) - Log rotation:
logrotateprevents logs from consuming all disk space
Essential Log Management Commands:
# View systemd journal logs journalctl -xe # Recent errors with context journalctl -u nginx.service # Logs for specific service journalctl -f # Follow logs (like tail -f) journalctl --since "2024-01-15 10:00:00" --until "2024-01-15 11:00:00" journalctl -p err # Show only error-level messages # View traditional log files tail -f /var/log/syslog # Follow system log tail -f /var/log/auth.log # Authentication attempts grep "Failed password" /var/log/auth.log # Find failed logins less +F /var/log/nginx/access.log # Follow with less # Search logs efficiently journalctl | grep -i "error" grep -r "connection refused" /var/log/ # Log rotation configuration cat /etc/logrotate.conf ls /etc/logrotate.d/ # Manually rotate logs sudo logrotate -f /etc/logrotate.conf # Check log file sizes du -sh /var/log/* | sort -hr | head -10
Windows Event Logs:
# View recent system events Get-EventLog -LogName System -Newest 20 # Filter by event type Get-EventLog -LogName Security -EntryType FailureAudit | Select-Object -First 10 # Search for specific event ID Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4625} -MaxEvents 5 # Export logs for analysis Get-WinEvent -LogName Application -MaxEvents 100 | Export-Csv app-events.csv # View event log names Get-WinEvent -ListLog * | Where-Object {$_.RecordCount -gt 0} | Select-Object LogName, RecordCount # Real-time log monitoring Get-WinEvent -LogName System -Oldest -MaxEvents 1 -Oldest | ForEach-Object { while ($true) { Get-WinEvent -LogName System -MaxEvents 1 Start-Sleep -Seconds 2 } }
Centralized Logging: For multiple servers, implement centralized logging with ELK Stack (Elasticsearch, Logstash, Kibana), Graylog, or cloud solutions like AWS CloudWatch Logs or Azure Monitor.
4.1 Backup Fundamentals and 3-2-1 Rule
Backup strategy is your last line of defense against data loss from hardware failure, human error, ransomware, or disasters.
Keep 3 copies of your data (1 primary + 2 backups), on 2 different media types (e.g., disk + tape/cloud), with 1 copy stored offsite or offline. This protects against local disasters, media failure, and ransomware.
Backup Types Comparison:
| Type | What It Backs Up | Storage Used | Restore Speed | Best For |
|---|---|---|---|---|
| Full | All selected data | High | Fastest (single source) | Weekly baseline, small datasets |
| Incremental | Changes since last backup (any type) | Lowest | Slowest (chain of backups) | Daily backups, limited storage |
| Differential | Changes since last full backup | Medium | Moderate (full + one differential) | Balance of speed and storage |
| Synthetic Full | Creates new full from previous full + incrementals | Medium | Fast | Enterprise backup software |
Test Restores! A backup is only as good as your ability to restore from it. Schedule quarterly restore tests to verify backup integrity and document recovery procedures.
4.2 Linux Backup Tools and Techniques
rsync for Efficient File Backups:
# Basic rsync backup (local) rsync -avh --delete /source/data/ /backup/data/ # Options explained: # -a = archive mode (preserves permissions, timestamps, symlinks) # -v = verbose output # -h = human-readable numbers # --delete = remove files in backup not in source # Remote backup over SSH rsync -avzh -e ssh /local/data/ user@backup-server:/remote/backup/ # Exclude patterns rsync -av --exclude={'*.tmp','cache/','.git/'} /source/ /backup/ # Bandwidth limit (10 Mbps) rsync -av --bwlimit=10000 /source/ user@remote:/backup/ # Create timestamped backup directory rsync -av /data/ /backups/data-$(date +%Y-%m-%d)/ # Dry run (test without making changes) rsync -av --delete --dry-run /source/ /backup/
tar for Archiving and Compression:
# Create compressed archive tar -czvf backup-$(date +%F).tar.gz /important/data # Options: # -c = create archive # -z = gzip compression # -j = bzip2 compression (better ratio, slower) # -J = xz compression (best ratio, slowest) # -v = verbose # -f = specify filename # Exclude files during archive tar -czvf backup.tar.gz --exclude='*.log' --exclude='tmp/' /data # Extract archive tar -xzvf backup.tar.gz -C /restore/location # List archive contents without extracting tar -tzvf backup.tar.gz # Backup with permissions preservation sudo tar -czpf backup.tar.gz --acls --xattrs /etc /home
Automated Backups with cron:
# Edit crontab crontab -e # Daily backup at 2 AM 0 2 * * * /usr/local/bin/backup-script.sh # Weekly full backup on Sunday at 3 AM 0 3 * * 0 /usr/local/bin/full-backup.sh # Rotate backups: keep last 7 daily, 4 weekly # Add to backup script: find /backups/daily -type f -mtime +7 -delete find /backups/weekly -type f -mtime +28 -delete # Email backup status (requires mailutils) # At end of backup script: if [ $? -eq 0 ]; then echo "Backup completed successfully" | mail -s "Backup OK" admin@example.com else echo "Backup FAILED" | mail -s "Backup ERROR" admin@example.com fi
Encryption: For sensitive data, encrypt backups with gpg or openssl. Example: tar -czf - /data | gpg -c --cipher-algo AES256 -o backup.tar.gz.gpg. Store encryption keys separately from backups!
4.3 Windows Backup Solutions
Windows Server Backup (WBAdmin):
# Install Windows Server Backup feature Install-WindowsFeature Windows-Server-Backup # One-time backup to D: drive wbadmin start backup -backupTarget:D: -include:C: -quiet # Schedule daily backup at 11 PM wbadmin enable backup -schedule:23:00 -backupTarget:D: -include:C: -quiet # View backup history wbadmin get versions # Recover specific file from backup wbadmin recover version:01/15/2024-23:00 -items:C:\data\important.doc -recursive -quiet # Delete old backups (keep last 3) wbadmin delete backup -keepVersions:3 -quiet # PowerShell alternative using Shadow Copy # Create shadow copy and copy files $shadow = (Get-WmiObject Win32_ShadowCopy).Create("C:\\", "ClientAccessible") $shadowID = $shadow.ShadowID # ... (additional code to access shadow copy)
Third-Party and Cloud Options:
- Veeam Agent: Free for Windows servers, image-level backups, cloud tiering
- AWS Backup: Centralized backup for EC2, EBS, RDS with lifecycle policies
- Azure Backup: Integrated with Recovery Services vaults, application-consistent
- Datto/R1Soft: Enterprise CDP (Continuous Data Protection) solutions
Cloud Backup Tip: Use cloud storage (S3, Azure Blob) with versioning and lifecycle policies for offsite backups. Enable MFA Delete to protect against accidental or malicious deletion.
5.1 SSH Hardening (Linux/Unix)
SSH is the primary remote access method for Linux servers; proper hardening prevents unauthorized access and brute-force attacks.
Secure /etc/ssh/sshd_config:
# Backup original config sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup # Edit configuration sudo nano /etc/ssh/sshd_config # Essential hardening settings: # Disable root login (use sudo instead) PermitRootLogin no # Use key-based authentication only PasswordAuthentication no PubkeyAuthentication yes # Limit users/groups allowed to connect AllowUsers admin jsmith # OR AllowGroups sshusers sudo # Change default port (reduces automated attacks) Port 2222 # Protocol and ciphers (modern defaults usually sufficient) Protocol 2 KexAlgorithms curve25519-sha256,diffie-hellman-group-exchange-sha256 Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com # Limit authentication attempts MaxAuthTries 3 LoginGraceTime 60 # Disable unused features X11Forwarding no AllowTcpForwarding no PermitTunnel no # Logging LogLevel VERBOSE # Apply changes sudo sshd -t # Test config syntax sudo systemctl restart sshd # IMPORTANT: Keep current session open until you verify new connection works!
Additional SSH Security Measures:
- Fail2Ban: Automatically ban IPs after failed login attempts
- SSH Keys: Use Ed25519 keys with passphrase:
ssh-keygen -t ed25519 -C "admin@server" - Key Management: Store public keys in
~/.ssh/authorized_keyswith restrictive permissions (chmod 600) - Two-Factor Auth: Add Google Authenticator or Duo for MFA
Lockout Prevention: Always test SSH configuration changes in a separate terminal before closing your current session. Keep a console access method (IPMI, VNC) available in case of misconfiguration.
5.2 Firewall Configuration
Linux: iptables/nftables and UFW:
# UFW (Uncomplicated Firewall) - Ubuntu/Debian sudo ufw enable sudo ufw default deny incoming sudo ufw default allow outgoing # Allow essential services sudo ufw allow 2222/tcp # Custom SSH port sudo ufw allow 80/tcp # HTTP sudo ufw allow 443/tcp # HTTPS sudo ufw allow from 192.168.1.0/24 to any port 3306 # MySQL from LAN # Enable logging sudo ufw logging medium # Check status sudo ufw status verbose # firewalld (RHEL/CentOS) sudo firewall-cmd --permanent --add-service=ssh sudo firewall-cmd --permanent --add-service=http sudo firewall-cmd --permanent --add-service=https sudo firewall-cmd --reload # Create custom zone for internal services sudo firewall-cmd --permanent --new-zone=internal sudo firewall-cmd --permanent --zone=internal --add-source=192.168.1.0/24 sudo firewall-cmd --permanent --zone=internal --add-service=mysql sudo firewall-cmd --reload
Windows Firewall with PowerShell:
# Enable firewall profiles Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled True # Create inbound rule for custom SSH port New-NetFirewallRule -DisplayName "Allow SSH" -Direction Inbound -LocalPort 2222 -Protocol TCP -Action Allow # Allow HTTP/HTTPS New-NetFirewallRule -DisplayName "Allow Web" -Direction Inbound -LocalPort 80,443 -Protocol TCP -Action Allow # Restrict RDP to specific subnet New-NetFirewallRule -DisplayName "RDP Restricted" -Direction Inbound -LocalPort 3389 -Protocol TCP -Action Allow -RemoteAddress "192.168.1.0/24" # Block specific IP New-NetFirewallRule -DisplayName "Block Malicious IP" -Direction Inbound -RemoteAddress "203.0.113.42" -Action Block # View rules Get-NetFirewallRule | Where-Object {$_.Enabled -eq $true} | Select-Object DisplayName, Direction, Action # Export rules for documentation Get-NetFirewallRule | Export-Csv firewall-rules.csv -NoTypeInformation
Test Firewall Rules: After changes, verify connectivity with telnet server-ip port, nc -zv server-ip port, or online port scanners. Document all rules and review quarterly.
5.3 System Updates and Patch Management
Regular patching is the most effective defense against known vulnerabilities. Establish a predictable update schedule with testing procedures.
Linux Package Updates:
# Debian/Ubuntu (APT) sudo apt update # Refresh package index sudo apt list --upgradable # Show available updates sudo apt upgrade -y # Install updates (non-destructive) sudo apt full-upgrade -y # Install updates (may remove packages) sudo apt autoremove -y # Clean unused dependencies # RHEL/CentOS/Rocky (DNF) sudo dnf check-update sudo dnf update -y sudo dnf autoremove -y # Security-only updates (Debian/Ubuntu) sudo unattended-upgrades --dry-run # Test config # Configure /etc/apt/apt.conf.d/50unattended-upgrades for auto security updates # Schedule updates via cron (example: Sundays at 4 AM) # Add to /etc/cron.weekly/security-updates: #!/bin/bash sudo apt update && sudo apt upgrade -y --only-upgrade 'linux-image-*' 'linux-headers-*' sudo systemctl reboot # Optional: auto-reboot for kernel updates # Verify kernel version after reboot uname -r
Windows Update Management:
# Check for updates Install-Module PSWindowsUpdate -Force Get-WindowsUpdate # Install updates with auto-reboot Get-WindowsUpdate -AcceptAll -Install -AutoReboot # Install only security updates Get-WindowsUpdate -Category "Security Updates" -AcceptAll -Install # Configure Windows Update via Group Policy (recommended for enterprises) # Path: Computer Configuration → Administrative Templates → Windows Components → Windows Update # PowerShell: Configure update settings Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU" -Name "NoAutoUpdate" -Value 0 Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU" -Name "AUOptions" -Value 4 # Auto download and schedule install # View update history Get-HotFix | Select-Object -First 10 Get-WindowsUpdateLog # Generates readable update log
Patch Management Strategy: Implement a staged rollout: 1) Test updates in non-production, 2) Deploy to dev/staging, 3) Schedule production maintenance window, 4) Verify functionality, 5) Document changes. Use WSUS (Windows) or Spacewalk/Satellite (Linux) for centralized control.
Key Takeaways
- User Management: Implement least-privilege access using groups, sudo, and ACLs. Regularly audit user accounts and remove inactive ones.
- Permissions: Master Linux rwx permissions and ACLs for granular control. Use Windows NTFS permissions with AGDLP principle for scalable AD management.
- Monitoring: Combine real-time tools (htop, iostat) with centralized logging (journald, ELK) for proactive issue detection. Set up alerting for critical metrics.
- Backups: Follow the 3-2-1 rule with automated, tested backups. Use rsync/tar for Linux, WBAdmin/Veeam for Windows, and encrypt sensitive data.
- Security Hardening: Disable password SSH logins, configure firewalls with default-deny policies, and maintain a disciplined patch management schedule.
- Documentation: Document all configurations, backup procedures, and recovery steps. Runbooks save time during incidents.
Next Steps: Create a server hardening checklist based on CIS Benchmarks, implement automated configuration management with Ansible/Puppet, and establish a monitoring dashboard with Grafana/Prometheus. Practice disaster recovery drills quarterly!