This guide covers security best practices in Linux server management with a trait towards Ansible. It is a good start for anyone considering their infra security and contains inspiration for future improvment ideas for this repo and beyond.
- Security Philosophy
- Ansible Security
- SSH and Authentication
- User and Access Management
- System Hardening
- Network Security
- Secrets Management
- Monitoring and Auditing
- Compliance and Standards
- Incident Response
- Security Checklist
This project implements multiple layers of security controls:
- Physical Security: Secure data centers and hardware
- Network Security: Firewalls, network segmentation, VPNs
- Host Security: System hardening, access controls, monitoring
- Application Security: Secure configurations, least privilege
- Data Security: Encryption at rest and in transit
- Operational Security: Secure procedures, incident response
- Never Trust, Always Verify: Authenticate and authorize every connection
- Least Privilege Access: Grant minimum necessary permissions
- Assume Breach: Design systems to limit damage from compromises
- Continuous Monitoring: Monitor all activities and maintain visibility
# BAD: Hardcoded sensitive valuesdatabase_password: "plaintext_password"# GOOD: Use Ansible Vaultdatabase_password: "{{ vault_database_password }}"# GOOD: Environment variables for CI/CDapi_key: "{{ lookup('env', 'API_KEY') }}"# Use become only when necessary
- name: Install packagepackage:
name: nginxbecome: true # Only when root is required# Validate inputs
- name: Create useruser:
name: "{{ username }}"when: - username is defined
- username | length > 0
- username is match('^[a-z][a-z0-9_-]*$')# Use no_log for sensitive operations
- name: Set passworduser:
name: "{{ username }}"password: "{{ password | password_hash('sha512') }}"no_log: true# Validate file permissions
- name: Create config filetemplate:
src: config.j2dest: /etc/app/configowner: appgroup: appmode: '0600'# Restrict access# ansible.cfg - Secure connection settings[defaults]host_key_checking = Truessh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=yestimeout = 30[ssh_connection]ssh_args = -o StrictHostKeyChecking=yes -o PasswordAuthentication=nocontrol_path_dir = ~/.ansible/cp# Create new encrypted file
ansible-vault create inventories/production/group_vars/all/vault.yml
# Encrypt existing file
ansible-vault encrypt sensitive_vars.yml
# Edit encrypted file
ansible-vault edit inventories/production/group_vars/all/vault.yml
# Decrypt for viewing (avoid this in production)
ansible-vault view inventories/production/group_vars/all/vault.yml# inventories/production/group_vars/all/vault.yml (encrypted)
---
vault_database_password: "SecureDBPassword123!"vault_api_key: "sk-1234567890abcdef"vault_ssl_private_key: | -----BEGIN PRIVATE KEY----- MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC... -----END PRIVATE KEY-----# inventories/production/group_vars/all/vars.yml (plain text)
---
database_password: "{{ vault_database_password }}"api_key: "{{ vault_api_key }}"ssl_private_key: "{{ vault_ssl_private_key }}"# Use password file (secure on control machine)echo'MySecureVaultPassword'>~/.vault_pass
chmod 600 ~/.vault_pass
export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass
# Use key script for integration
cat > vault_key.sh << 'EOF'#!/bin/bash# Retrieve vault password from secure key management systemvault kv get -field=password secret/ansible/vaultEOF
chmod +x vault_key.sh
export ANSIBLE_VAULT_PASSWORD_FILE=./vault_key.sh# Generate strong SSH key pair
ssh-keygen -t ed25519 -C "username@hostname-$(date +%Y%m%d)"# For older systems requiring RSA
ssh-keygen -t rsa -b 4096 -C "username@hostname-$(date +%Y%m%d)"# Use key passphrases
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_project -C "project_access"# ~/.ssh/config
Host production-servers
HostName %h.example.com
User deploy
IdentityFile ~/.ssh/id_ed25519_production
IdentitiesOnly yes
StrictHostKeyChecking yes
UserKnownHostsFile ~/.ssh/known_hosts_production
Host bastion-server
HostName bastion.example.com
User admin
Port 2222
IdentityFile ~/.ssh/id_ed25519_admin
ServerAliveInterval 60
ServerAliveCountMax 3# SSH configuration enforced by hardening rolesshd_config_settings:
Protocol: 2PermitRootLogin: noPasswordAuthentication: noChallengeResponseAuthentication: noUsePAM: yesX11Forwarding: noPrintMotd: noClientAliveInterval: 600ClientAliveCountMax: 0MaxAuthTries: 3MaxSessions: 2LoginGraceTime: 30AllowGroups: ssh-usersThis playbook implements TOTP-based 2FA using Google Authenticator PAM module.
- First Login: User authenticates with SSH key, then sees QR code for 2FA enrollment
- Enrollment: User scans QR code with authenticator app (Aegis, Google Authenticator, Authy, Bitwarden, etc..., etc.)
- Subsequent Logins: User provides SSH key + 6-digit TOTP code
2FA is enabled globally via the ENABLE_2FA variable:
ENABLE_2FA: trueSecrets are stored centrally in /var/lib/google-authenticator/ (root-owned, mode 0700).
- Token reuse prevention: Each code can only be used once (
-dflag) - Rate limiting: Max 3 attempts per 30 seconds (
-r 3 -R 30) - Time window: Accepts 3 codes to handle clock skew (
-w 3) - Emergency codes: 5 one-time backup codes generated
If a user loses their 2FA device or needs to re-enroll:
# Remove the user's 2FA secret file
sudo rm /var/lib/google-authenticator/username
# On next SSH login, the user will be prompted to set up 2FA again# /etc/ssh/sshd_config additions
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive# Secure user creation with users_add roleusers_add_userlist:
- username: "john_doe"public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... john@secure-laptop"admin: false # Principle of least privilegenon_admin_allowed_commands: "NOPASSWD: /usr/bin/systemctl status *, /usr/bin/journalctl -f"# Specific, limited sudo commands only# PAM password quality settings (managed by hardening role)pwquality_settings:
minlen: 14# Minimum password lengthminclass: 3# Minimum character classesmaxrepeat: 2# Maximum repeated charactersmaxsequence: 3# Maximum sequence lengthdcredit: -1# Require digitsucredit: -1# Require uppercaselcredit: -1# Require lowercaseocredit: -1# Require special characters# Individual sudoers files (/etc/sudoers.d/username)# Full admin access
john_admin ALL=(ALL) ALL
# Limited access examples where needed
backup_user ALL=(root) NOPASSWD: /usr/bin/rsync, /bin/tar
monitor_user ALL=(root) NOPASSWD: /usr/bin/systemctl status *, /usr/bin/journalctl
web_user ALL=(www-data) NOPASSWD: /usr/bin/systemctl reload nginx
# Restrict dangerous commands where needed
operator_user ALL=(ALL) ALL, !/bin/su, !/usr/bin/sudo, !/bin/bash, !/bin/sh# Service accounts with minimal privileges
- name: Create application service accountuser:
name: app_servicesystem: yesshell: /bin/false # No shell accesshome: /var/lib/app # Dedicated home directorycreate_home: yesbecome: true
- name: Set application directory permissionsfile:
path: /var/lib/appowner: app_servicegroup: app_servicemode: '0750'# Limited accessThe project implements CIS (Center for Internet Security) benchmarks through the konstruktoid.hardening role:
- Remove unnecessary software packages
- Configure network parameters
- Configure logging and auditing
- Configure access, authentication and authorization
- Configure system maintenance
- Additional access restrictions
- Advanced auditing configuration
- Enhanced logging
- Network security configurations
# Additional hardening beyond CIS baselinesecurity_extensions:
# Kernel securitykernel_parameters:
kernel.dmesg_restrict: 1kernel.kptr_restrict: 2kernel.unprivileged_bpf_disabled: 1# Network securitynetwork_parameters:
net.ipv4.tcp_syncookies: 1net.ipv4.conf.all.rp_filter: 1net.ipv6.conf.all.disable_ipv6: 1# File system securityfilesystem_parameters:
fs.suid_dumpable: 0fs.protected_hardlinks: 1fs.protected_symlinks: 1# Secure mount options (managed by hardening role)
/tmp tmpfs defaults,nodev,nosuid,noexec 0 0
/var/tmp tmpfs defaults,nodev,nosuid,noexec 0 0
/dev/shm tmpfs defaults,nodev,nosuid,noexec 0 0# Find files with dangerous permissions
find / -perm -4000 -type f 2>/dev/null # SUID files
find / -perm -2000 -type f 2>/dev/null # SGID files
find / -perm -002 -type f 2>/dev/null # World-writable files
find / -nouser -o -nogroup 2>/dev/null # Orphaned files# Basic UFW configuration via hardening roleufw_configuration:
default_incoming: denydefault_outgoing: allowdefault_routed: denyufw_rules:
- rule: allowport: '22'proto: tcpsrc: '192.168.1.0/24'# Restrict SSH to specific networks
- rule: allowport: '80'proto: tcp
- rule: allowport: '443'proto: tcp
- rule: denyport: '23'# Deny telnetproto: tcp# Disable unused network protocolsecho'install dccp /bin/true'>> /etc/modprobe.d/blacklist-rare-network.conf
echo'install sctp /bin/true'>> /etc/modprobe.d/blacklist-rare-network.conf
echo'install rds /bin/true'>> /etc/modprobe.d/blacklist-rare-network.conf
# Network monitoring
netstat -tuln | grep LISTEN # Check listening ports
ss -tuln # Modern alternative to netstatProduction Network: 10.0.1.0/24
- Web servers: 10.0.1.10-50
- Database servers: 10.0.1.60-80
- Application servers: 10.0.1.90-120
Management Network: 10.0.2.0/24
- Bastion hosts: 10.0.2.10-20
- Monitoring: 10.0.2.30-40
- Backup systems: 10.0.2.50-60
DMZ Network: 10.0.3.0/24
- Load balancers: 10.0.3.10-20
- Reverse proxies: 10.0.3.30-40
- Public: Information that can be shared publicly
- Internal: Information for internal use only
- Confidential: Sensitive business information
- Restricted: Highly sensitive information requiring special handling
secrets_classification:
public:
- Application configurations (non-sensitive)
- Public SSL certificates
- Documentationinternal:
- Internal API endpoints
- Service discovery configurations
- Non-production credentialsconfidential:
- Database passwords
- API keys
- SSL private keys
- Production credentialsrestricted:
- Root certificates
- Master encryption keys
- Recovery codes#!/bin/bash# Example secrets rotation scriptrotate_database_password() {
local new_password=$(openssl rand -base64 32)# Update password in vault
ansible-vault edit inventories/production/group_vars/all/vault.yml
# Apply to systems
ansible-playbook -i production update-db-password.yml \
--ask-vault-pass \
-e "new_db_password=${new_password}"# Verify connectivity
ansible production -m shell -a "mysql -u app -p${new_password} -e 'SELECT 1'"
}
# Schedule rotation (crontab)# 0 2 1 * * /opt/scripts/rotate_secrets.sh >> /var/log/rotation.log 2>&1# Vault integration example
- name: Retrieve secret from Vaulturi:
url: "{{ vault_url }}/v1/secret/data/myapp"method: GETheaders:
X-Vault-Token: "{{ vault_token }}"register: vault_response
- name: Use secret in tasktemplate:
src: config.j2dest: /etc/app/configvars:
db_password: "{{ vault_response.json.data.data.password }}"no_log: true# /etc/audit/rules.d/audit.rules# Monitor authentication
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
# Monitor privilege escalation
-w /bin/su -p x -k privilege_escalation
-w /usr/bin/sudo -p x -k privilege_escalation
# Monitor network configuration
-w /etc/sysconfig/network -p wa -k network_config
-w /etc/hosts -p wa -k network_config
# Monitor file access
-w /etc/ssh/sshd_config -p wa -k ssh_config
-w /var/log/ -p wa -k log_access# Centralized logging configurationrsyslog_configuration:
# Forward logs to central serverforward_logs: truelog_server: "log-collector.example.com:514"# Local log retentionlog_retention_days: 90# Log rotationlog_rotation:
daily: truecompress: truemax_age: 30# AIDE (Advanced Intrusion Detection Environment)
aide --init # Initialize database
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
aide --check # Check for changes# Tripwire alternative - simple checksum monitoring
find /etc /bin /sbin /usr/bin /usr/sbin -type f -exec sha256sum {} \;> /var/lib/checksums.db# Example monitoring tasks
- name: Check for failed SSH attemptsshell: grep "Failed password" /var/log/auth.log | tail -10register: failed_sshchanged_when: false
- name: Alert on suspicious activitymail:
subject: "Security Alert: Failed SSH attempts on {{ inventory_hostname }}"body: "{{ failed_ssh.stdout }}"to: security@example.comwhen: failed_ssh.stdout_lines | length > 5# Unattended upgrades configurationunattended_upgrades:
enabled: trueorigins_patterns:
- "origin=Ubuntu,archive=${distro_codename}-security"
- "o=Ubuntu,a=${distro_codename}-updates"package_blacklist:
- kernel* # Exclude kernel updates (require manual reboot)automatic_reboot: falseemail_notifications: "admin@example.com"# Local vulnerability assessment
lynis audit system # Security audit tool
chkrootkit # Rootkit detection
rkhunter --check # Rootkit hunter# Network vulnerability scanning
nmap -sV --script vuln target_host- SOC 2: Service Organization Control 2
- ISO 27001: Information Security Management
- PCI DSS: Payment Card Industry Data Security Standard
- GDPR: General Data Protection Regulation
- HIPAA: Health Insurance Portability and Accountability Act
controls_mapping:
access_control:
frameworks: [SOC2, ISO27001, PCI_DSS]implementations:
- SSH key-based authentication
- Multi-factor authentication
- Principle of least privilege
- Regular access reviewsencryption:
frameworks: [SOC2, ISO27001, PCI_DSS, GDPR]implementations:
- TLS 1.3 for data in transit
- AES-256 for data at rest
- SSH encryption for management
- Ansible Vault for secretslogging_monitoring:
frameworks: [SOC2, ISO27001, PCI_DSS]implementations:
- Centralized log collection
- Real-time monitoring
- Audit trail maintenance
- Security event alerting- Security Policies: High-level security requirements
- Procedures: Step-by-step security operations
- Standards: Technical security configurations
- Guidelines: Security best practices and recommendations
- Asset inventory and classification
- Risk assessment and treatment
- Incident response procedures
- Business continuity planning
- Security awareness training records
- Vulnerability management reports
- Establish incident response team
- Create communication channels
- Prepare investigation tools
- Document escalation procedures
# Quick incident detection commands
who # Current users
last # Login history
ps aux | grep -E "(nc|netcat)"# Check for suspicious processes
netstat -tuln | grep LISTEN # Check listening ports
find /tmp -name ".*" -type f # Hidden files in tmp# Emergency containment actions# Block suspicious IP
ufw deny from suspicious_ip
# Disable compromised user account
usermod -L compromised_user
# Kill suspicious processes
pkill -f suspicious_process
# Isolate system from network
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP# Recovery playbook example
- name: Emergency security responsehosts: compromised_systemstasks:
- name: Reset compromised passwordsuser:
name: "{{ item }}"password: "{{ lookup('password', '/tmp/new_passwords/' + item) | password_hash('sha512') }}"loop: "{{ compromised_users }}"no_log: true
- name: Update all packagespackage:
name: "*"state: latest
- name: Restart security servicessystemd:
name: "{{ item }}"state: restartedloop:
- ssh
- ufw
- auditd# System information collection
hostname > /tmp/incident_data/hostname.txt
uname -a > /tmp/incident_data/system_info.txt
date > /tmp/incident_data/collection_time.txt
# Network information
netstat -tuln > /tmp/incident_data/network_connections.txt
arp -a > /tmp/incident_data/arp_table.txt
route -n > /tmp/incident_data/routing_table.txt
# Process information
ps aux > /tmp/incident_data/processes.txt
lsof > /tmp/incident_data/open_files.txt
# Log collection
cp /var/log/auth.log /tmp/incident_data/
cp /var/log/syslog /tmp/incident_data/
cp /var/log/audit/audit.log /tmp/incident_data/- Network segmentation implemented
- Firewall rules configured and tested
- VPN access configured for remote management
- Physical security measures in place
- Backup and recovery procedures tested
- Operating systems fully patched
- Unnecessary services disabled
- Strong authentication mechanisms enabled
- File system permissions properly configured
- Audit logging enabled and configured
- Secure configuration templates validated
- Secrets properly encrypted with Ansible Vault
- Service accounts configured with minimal privileges
- Security scanning completed
- Penetration testing performed
- Review security alerts and logs
- Monitor failed authentication attempts
- Check system resource usage
- Verify backup completion
- Update threat intelligence feeds
- Review user access and permissions
- Analyze security metrics and trends
- Update security tools and signatures
- Conduct vulnerability scans
- Review and update documentation
- Conduct security risk assessment
- Review and test incident response procedures
- Perform access certification reviews
- Update security awareness training
- Conduct tabletop exercises
- Comprehensive security audit
- Penetration testing assessment
- Review and update security policies
- Business continuity plan testing
- Third-party security assessments
security_metrics:
technical:
- Mean time to patch (MTTP)
- Number of critical vulnerabilities
- Failed authentication attempts
- Security incidents per month
- System uptime and availabilityoperational:
- Security training completion rate
- Incident response time
- Access review completion rate
- Policy compliance percentage
- Security tool effectivenessbusiness:
- Cost of security incidents
- Customer trust metrics
- Regulatory compliance status
- Security ROI measurements
- Business continuity metricsThis section documents where our configuration (this repo's defaults) deviates from standard CIS benchmarks and the operational rationale behind these decisions.
| Setting | CIS/Standard | Our Setting | Rationale | Risk Assessment |
|---|---|---|---|---|
| SSH Login Grace Time | 30s | 60s | User requires additional time to type complex passwords, reducing authentication failures | Low Risk: Minimal exposure increase, improves user experience |
| Session Timeout | 300-600s | 900s (15 min) | User frequently annoyed by re-authentication when switching tasks, reducing productivity | Low Risk: Acceptable trade-off for operational efficiency |
All critical security controls remain at CIS standards:
- ✅ Strong SSH ciphers and MACs
- ✅ No root login permitted
- ✅ No password authentication (key-based only)
- ✅ No X11/TCP forwarding
- ✅ Proper audit logging
- ✅ UFW firewall with default deny
- ✅ System hardening via konstruktoid.hardening
Security is an ongoing process requiring continuous attention, monitoring, and improvement. This guide provides a comprehensive foundation for implementing and maintaining security in your linux-server-management infrastructure.
- Implement Defense in Depth: Multiple layers of security controls
- Follow Least Privilege: Grant minimal necessary access
- Automate Security: Use automation for consistent security implementation
- Monitor Continuously: Maintain visibility into system activities
- Plan for Incidents: Prepare for security events before they occur
- Stay Current: Keep systems patched and security knowledge updated