The Complete Guide to SSH: Advanced Topics

This guide covers advanced SSH features and deep technical concepts. It assumes you already understand the basics of SSH.

🚀 New to SSH? Start with the Beginner's Guide to SSH first. It covers installation, basic connections, key generation, file transfers, and essential security practices in ~20 minutes.

This advanced guide explores:

• The cryptographic foundations of SSH (Diffie-Hellman key exchange)
• Advanced port forwarding and tunnelling techniques
• SSH multiplexing for performance optimisation
• Jump hosts and complex network topologies
• SSH certificates for enterprise key management
• Production-ready configuration patterns
• Advanced troubleshooting and debugging

Prerequisites: You should be comfortable with basic SSH connections, key-based authentication, and common SSH commands before diving into this guide.

Deep Dive: SSH Cryptography

For basic SSH concepts, see the Beginner's Guide. This section covers the mathematical foundations.

The Three-Layer Security Architecture

SSH combines three cryptographic techniques:

1. Symmetric Encryption - Fast bulk data encryption (AES, ChaCha20)
2. Asymmetric Encryption - Secure key exchange and authentication (RSA, ED25519)
3. Hashing - Message integrity verification (SHA-2, HMAC)

Diffie-Hellman Key Exchange: The Magic Explained

The Diffie-Hellman algorithm is the cornerstone of SSH's security. It allows two parties to create a shared secret key over a public channel without ever transmitting the key itself.

The Paint Mixing Analogy:

Imagine Alice and Bob want to create a shared secret colour without anyone seeing it:

1. They publicly agree on a common base colour (yellow)
2. Alice picks a secret colour (red) and mixes it with yellow → orange
3. Bob picks a secret colour (blue) and mixes it with yellow → green
4. They exchange their mixtures publicly (orange and green)
5. Alice adds her secret (red) to Bob's mixture (green) → brown
6. Bob adds his secret (blue) to Alice's mixture (orange) → brown
7. They both end up with the same colour (brown), but no one else can recreate it!

The Mathematical Reality:

1. Public parameters agreed upon:
   - Prime number p (very large, 2048+ bits)
   - Generator g (primitive root modulo p)

2. Private key generation:
   Alice: picks private a (random)
   Bob: picks private b (random)

3. Public key computation:
   Alice: A = g^a mod p  →  sends A to Bob
   Bob:   B = g^b mod p  →  sends B to Alice

4. Shared secret computation:
   Alice: s = B^a mod p = (g^b)^a mod p = g^(ab) mod p
   Bob:   s = A^b mod p = (g^a)^b mod p = g^(ab) mod p

Result: Both have the same secret s = g^(ab) mod p

Why It's Secure:

The "discrete logarithm problem" makes it computationally infeasible to derive the private keys (a or b) from the public keys (A or B), even though the attacker knows g, p, A, and B.

🔐 With modern parameters (2048-bit primes), breaking Diffie-Hellman would take billions of years even with today's supercomputers.

Visual Connection Flow

Modern Cipher Recommendations (2025)

Preferred algorithms for maximum security:

# ~/.ssh/config
Host *
    # Key exchange algorithms (order of preference)
    KexAlgorithms curve25519-sha256,diffie-hellman-group-exchange-sha256

    # Host key algorithms
    HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

    # Ciphers (symmetric encryption)
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com

    # MACs (message authentication)
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

💡 Performance vs Security: ChaCha20-Poly1305 is faster on systems without AES hardware acceleration. AES-GCM is faster with hardware support (most modern CPUs).

---

Advanced Port Forwarding & Tunneling

Basic port forwarding covered in the Beginner's Guide. This section focuses on advanced patterns and real-world scenarios.

Local Port Forwarding: Advanced Patterns

Basic syntax:

ssh -L [bind_address:]local_port:remote_host:remote_port username@ssh_server

Pattern 1: Access Internal Database Through Jump Host

# Forward local 5432 to the database.internal:5432 through bastion
ssh -L 5432:database.internal:5432 user@bastion.company.com

# Now connect locally
psql -h localhost -p 5432 -U dbuser production_db

Pattern 2: Multiple Port Forwards in One Connection

# Forward multiple services simultaneously
ssh -L 5432:db.internal:5432 \
    -L 6379:redis.internal:6379 \
    -L 8080:api.internal:80 \
    user@bastion.company.com

Pattern 3: Bind to Specific Interface

# Make the forwarded port accessible to other machines on your network
# WARNING: Security implications! Use with caution.
ssh -L 0.0.0.0:8080:internal-web:80 user@gateway.com

Remote Port Forwarding: Advanced Patterns

Basic syntax:

ssh -R [bind_address:]remote_port:local_host:local_port username@ssh_server

Pattern 1: Expose Local Development to Remote Team

# Your local dev server (localhost:3000) is accessible on the remote server port 8080
ssh -R 8080:localhost:3000 user@shared-server.com

# Team accesses: http://shared-server.com:8080

Pattern 2: Bypass Firewall Restrictions (Reverse Tunnel)

# Your local SSH (port 22) is accessible from the remote server on port 2222
# Useful when you're behind a NAT/firewall
ssh -R 2222:localhost:22 user@public-server.com

# Later, from anywhere:
ssh -p 2222 your_username@public-server.com  # Actually connects to your local machine

Pattern 3: GatewayPorts for Public Access

# On remote server's /etc/ssh/sshd_config:
# GatewayPorts yes  # Allow binding to non-localhost addresses

# Now remote port is publicly accessible
ssh -R 0.0.0.0:8080:localhost:3000 user@server.com

Dynamic Port Forwarding: SOCKS Proxy Patterns

Basic syntax:

ssh -D [bind_address:]local_port username@ssh_server

Pattern 1: Browse Through Remote Server

# Create SOCKS proxy on port 9090
ssh -D 9090 user@server.com

# Configure browser:
# Network Settings → Manual Proxy → SOCKS Host: localhost, Port: 9090

Pattern 2: Route Specific Applications

# Use proxychains to route specific commands
ssh -D 9090 -N user@server.com  # -N = don't execute remote command

# In another terminal:
proxychains curl https://ifconfig.me  # Shows remote server's IP

Pattern 3: Multi-Hop SOCKS Proxy

# Chain through multiple servers
ssh -D 9090 user@bastion.com
# Then through bastion, connect to internal:
ssh -o ProxyCommand="nc -X 5 -x localhost:9090 %h %p" user@internal.server

Persistent Tunnels with AutoSSH

Keep tunnels alive automatically:

# Install autossh
brew install autossh  # macOS
apt install autossh   # Ubuntu

# Keep tunnel alive, restart on failure
autossh -M 0 -N -L 5432:db.internal:5432 user@bastion.com

# For systemd service (Linux):
cat <<EOF | sudo tee /etc/systemd/system/ssh-tunnel.service
[Unit]
Description=SSH Tunnel to Database
After=network.target

[Service]
User=youruser
ExecStart=/usr/bin/autossh -M 0 -N -L 5432:db.internal:5432 user@bastion.com
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now ssh-tunnel.service

---

SSH Multiplexing

Reuse a single TCP connection for multiple SSH sessions. This dramatically speeds up subsequent connections and reduces authentication overhead.

Enable Multiplexing

Add to ~/.ssh/config:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

What this does:

• ControlMaster auto: Automatically creates a master connection
• ControlPath: Where to store the control socket (must create directory first)
• ControlPersist 600: Keep the master connection open for 10 minutes after last use

Setup:

mkdir -p ~/.ssh/sockets
chmod 700 ~/.ssh/sockets

Performance Gains

# First connection (normal speed)
time ssh server.com exit
# real    0m2.847s

# Subsequent connections (using multiplexing)
time ssh server.com exit
# real    0m0.341s  ← 8x faster!

Managing Multiplex Connections

# List active multiplex connections
ssh -O check server.com

# Close a multiplex master
ssh -O exit server.com

# Forward a port through existing multiplex connection (no new TCP connection!)
ssh -O forward -L 8080:localhost:80 server.com

Advanced: Background Master Connection

# Start a background master connection
ssh -fNM server.com

# -f: Background the process
# -N: Don't execute remote command
# -M: Master mode

# All subsequent ssh/scp/rsync commands to server.com now reuse this connection
ssh server.com  # Instant connection!
scp file.txt server.com:~  # Instant transfer!

---

Jump Hosts & ProxyJump

Access servers behind firewalls or in private networks using intermediate "jump" hosts.

Basic ProxyJump

# Connect to internal server through bastion
ssh -J bastion.com user@internal.private

# Multiple hops
ssh -J bastion1.com,bastion2.com user@deeply-nested.server

ProxyJump in SSH Config

Host bastion
    HostName bastion.company.com
    User admin
    IdentityFile ~/.ssh/bastion_key

Host internal-*
    User developer
    IdentityFile ~/.ssh/internal_key
    ProxyJump bastion

# Now simply:
ssh internal-database
ssh internal-webserver

ProxyJump vs ForwardAgent

ProxyJump (Recommended):

• More secure: keys never exposed to bastion host
• Cleaner: no agent forwarding needed
• Faster: uses multiplexing through a jump host

ForwardAgent (Legacy):

• Security risk: compromised bastion can use your keys
• Required for some legacy setups

Advanced: Different Keys Per Hop

Host bastion
    HostName bastion.company.com
    User admin
    IdentityFile ~/.ssh/bastion_key

Host production-db
    HostName db.internal.company.com
    User dbadmin
    IdentityFile ~/.ssh/production_db_key
    ProxyJump bastion

Each hop uses its own key, which is much more secure than agent forwarding.

ProxyCommand for Complex Scenarios

For scenarios where ProxyJump isn't sufficient:

Host complex-internal
    HostName 10.0.1.50
    User admin
    ProxyCommand ssh bastion -W %h:%p
    # -W: Forwards stdin/stdout through the jump host

---

SFTP: Interactive File Transfers

SFTP provides an FTP-like interface over SSH.

# Connect
sftp user@server.com

# Common commands
sftp> ls                    # List remote files
sftp> lls                   # List local files
sftp> pwd                   # Remote working directory
sftp> lpwd                  # Local working directory
sftp> cd /var/www          # Change remote directory
sftp> lcd ~/downloads      # Change local directory

# Upload
sftp> put local_file.txt
sftp> put -r local_folder/  # Recursive upload

# Download
sftp> get remote_file.txt
sftp> get -r remote_folder/  # Recursive download

# Create/remove directories
sftp> mkdir new_folder
sftp> rmdir old_folder

# Batch operations
sftp> mget *.log           # Download all .log files
sftp> mput *.txt           # Upload all .txt files

sftp> bye                  # Exit

Batch SFTP Scripts

# Create batch file
cat <<EOF > sftp_commands.txt
cd /var/www/html
get index.html
put updated_file.txt
bye
EOF

# Execute batch
sftp -b sftp_commands.txt user@server.com

---

SSH Certificates (Enterprise Key Management)

Note: SSH certificates are different from SSL/TLS certificates. They solve the problem of managing SSH keys at scale.

The Problem SSH Certificates Solve

Traditional approach:

• Add your public key to 100 servers → tedious
• Remove someone's access → update 100 servers
• Key rotation → massive operational burden

SSH Certificates:

• Central authority signs short-lived certificates
• Servers trust the CA, not individual keys
• Certificates expire automatically
• Revocation is trivial

Setting Up SSH CA

1. Generate CA Key (do this on a secure host):

ssh-keygen -t ed25519 -f ca_key -C "SSH CA"
# Store ca_key securely (offline/HSM in production)
# Distribute ca_key.pub to all servers

2. Configure Servers to Trust CA:

# On each server: /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/ca_key.pub

# Copy CA public key
sudo cp ca_key.pub /etc/ssh/
sudo systemctl restart sshd

3. Sign User Keys (issue certificates):

# Sign user's public key, valid for 1 day
ssh-keygen -s ca_key \
    -I "john.doe" \
    -n john,ubuntu,ec2-user \
    -V +1d \
    id_ed25519.pub

# This creates: id_ed25519-cert.pub

Parameters:

• -s ca_key: Sign with CA private key
• -I "john.doe": Certificate identity (for logging)
• -n john,ubuntu,...: Allowed usernames on remote systems
• -V +1d: Valid for 1 day (can use +8h, +30d, etc.)

4. User Connects with Certificate:

# User has both files:
# ~/.ssh/id_ed25519
# ~/.ssh/id_ed25519-cert.pub

ssh user@any-server  # Certificate used automatically

Certificate Inspection

# View certificate details
ssh-keygen -L -f id_ed25519-cert.pub

Host Certificates (Prevent TOFU)

Servers can also use certificates to prove their identity:

# Sign server's host key
ssh-keygen -s ca_key -I server01 -h -n server01.company.com -V +365d /etc/ssh/ssh_host_ed25519_key.pub

# Client trusts CA for host certificates
# ~/.ssh/known_hosts or /etc/ssh/ssh_known_hosts
@cert-authority *.company.com ssh-ed25519 AAAAC3... (CA public key)

---

Production SSH Config Patterns

Streamlined, production-ready SSH configuration examples.

# ~/.ssh/config - Production SSH Configuration
# Permissions: chmod 600 ~/.ssh/config

# ============================================================================
# DEFAULT SETTINGS (applies to all hosts)
# ============================================================================
Host *
    # Security
    IdentitiesOnly yes
    StrictHostKeyChecking ask
    HashKnownHosts yes

    # Performance: Multiplexing
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

    # Keep-alive
    ServerAliveInterval 60
    ServerAliveCountMax 3

    # Speed optimisations
    Compression yes
    GSSAPIAuthentication no

    # Modern cryptography (2025)
    KexAlgorithms curve25519-sha256,diffie-hellman-group-exchange-sha256
    HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# ============================================================================
# PATTERN 1: Production with Bastion + Internal Network
# ============================================================================
Host bastion
    HostName bastion.company.com
    User admin
    IdentityFile ~/.ssh/bastion_ed25519
    # No ForwardAgent for security

Host prod-*
    User deploy
    IdentityFile ~/.ssh/production_ed25519
    ProxyJump bastion
    StrictHostKeyChecking yes
    # Match pattern: prod-web, prod-api, prod-db, etc.

# ============================================================================
# PATTERN 2: Database with Port Forwarding
# ============================================================================
Host db-tunnel
    HostName db-bastion.company.com
    User dba
    IdentityFile ~/.ssh/db_key
    LocalForward 5432 postgres.internal:5432
    LocalForward 6379 redis.internal:6379
    # After connecting, use localhost:5432 and localhost:6379

# ============================================================================
# PATTERN 3: GitHub/GitLab with Multiple Accounts
# ============================================================================
Host github-work
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_work_ed25519
    IdentitiesOnly yes

Host github-personal
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_personal_ed25519
    IdentitiesOnly yes

# Usage:
# git clone git@github-work:company/repo.git
# git clone git@github-personal:myuser/repo.git

# ============================================================================
# PATTERN 4: Wildcard for Ephemeral Cloud Instances
# ============================================================================
Host *.aws.internal
    User ubuntu
    IdentityFile ~/.ssh/aws_deployment_key.pem
    ProxyJump bastion
    # Disable strict checking for auto-scaled instances
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
    LogLevel ERROR

Setup:

# Create sockets directory
mkdir -p ~/.ssh/sockets
chmod 700 ~/.ssh ~/.ssh/sockets

# Set permissions
chmod 600 ~/.ssh/config
chmod 600 ~/.ssh/id_*
chmod 644 ~/.ssh/*.pub

Test your configuration:

# View computed config for a host
ssh -G prod-web

# Test connection verbosely
ssh -v bastion

Modular Config with Include Directive

For complex setups, split your config:

# ~/.ssh/config
Include ~/.ssh/config.d/*

Host *
    # Global defaults here
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

Then organise:

~/.ssh/
├── config               # Main config with Include
├── config.d/
│   ├── work.conf       # Work-related hosts
│   ├── personal.conf   # Personal servers
│   ├── github.conf     # Version control
│   └── aws.conf        # Cloud providers
└── sockets/            # Multiplexing sockets

---

Advanced Security Hardening

Basic security practices (key-based auth, permissions, etc.) are covered in the Beginner's Guide. This section covers advanced hardening for production environments.

Two-Factor Authentication (2FA/MFA)

Add an extra authentication layer using Google Authenticator or similar.

1. Install PAM module:

sudo apt install libpam-google-authenticator  # Ubuntu/Debian

2. Configure sshd to use 2FA:

# /etc/ssh/sshd_config
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

# /etc/pam.d/sshd (add this line)
auth required pam_google_authenticator.so

3. Set up for user:

google-authenticator
# Follow prompts, scan QR code with authenticator app

4. Restart SSH:

sudo systemctl restart sshd

Now users need both their SSH key AND a time-based OTP.

Fail2Ban for Intrusion Prevention

Automatically ban IPs after failed login attempts:

# Install
sudo apt install fail2ban

# Configure /etc/fail2ban/jail.local
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

# Start
sudo systemctl enable --now fail2ban

# Check bans
sudo fail2ban-client status sshd

Advanced sshd_config Hardening

# /etc/ssh/sshd_config - Production Hardening

# Disable root and password auth
PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Limit authentication attempts
MaxAuthTries 3
MaxSessions 5

# Whitelist specific users/groups
AllowUsers deploy ops-team
# Or use groups:
AllowGroups ssh-users sudo

# Limit key types to modern algorithms only
PubkeyAcceptedKeyTypes ssh-ed25519,rsa-sha2-512,rsa-sha2-256

# Disable X11 and other forwarding (if not needed)
X11Forwarding no
AllowTcpForwarding no  # Or 'local' for local forwarding only
AllowStreamLocalForwarding no
PermitTunnel no

# Logging
LogLevel VERBOSE  # Or INFO for less detail
SyslogFacility AUTH

# Idle timeout
ClientAliveInterval 300
ClientAliveCountMax 2

# Modern ciphers and MACs only (match client config)
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
KexAlgorithms curve25519-sha256,diffie-hellman-group-exchange-sha256

# Test before restarting
# sudo sshd -t && sudo systemctl restart sshd

⚠️ Critical: Always keep a backup SSH session open when modifying sshd_config to avoid lockouts!

SSH Audit Logging with Custom Script

Log all SSH commands for audit compliance:

# /etc/profile.d/ssh-audit.sh
if [ -n "$SSH_CONNECTION" ]; then
    export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local6.debug "$(whoami) [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//" )"'
fi

# Configure rsyslog to capture
# /etc/rsyslog.d/ssh-audit.conf
local6.*    /var/log/ssh-audit.log

# Restart rsyslog
sudo systemctl restart rsyslog

All commands executed via SSH are now logged to /var/log/ssh-audit.log.

Port Knocking

Hide SSH from port scanners using port knocking:

# Install knockd
sudo apt install knockd

# /etc/knockd.conf
[openSSH]
    sequence    = 7000,8000,9000
    seq_timeout = 5
    command     = /sbin/iptables -I INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
    tcpflags    = syn

[closeSSH]
    sequence    = 9000,8000,7000
    seq_timeout = 5
    command     = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
    tcpflags    = syn

# Client knocks the ports in sequence
knock server.com 7000 8000 9000
ssh user@server.com
knock server.com 9000 8000 7000  # Close after done

Hardware Security Keys (YubiKey)

Use FIDO2/U2F hardware tokens for SSH:

# Generate SK-resident key (stored on YubiKey)
ssh-keygen -t ecdsa-sk -O resident -O verify-required

# Key is now on the hardware token
# Requires physical touch to authenticate
# Copy public key to servers as usual

Benefits:

• Private key can't be stolen (lives on hardware)
• Requires physical presence
• One token for all servers

---

Advanced Troubleshooting

Basic issues (permission denied, connection timeouts, host key verification) are covered in the Beginner's Guide. This section covers complex scenarios.

Debugging with Maximum Verbosity

# Triple verbose shows every detail
ssh -vvv user@host 2>&1 | tee ssh-debug.log

# Key sections to look for:
# - "debug1: Offering public key" - which keys are tried
# - "debug1: Authentication succeeded" - what worked
# - "debug1: Remote protocol version" - server SSH version
# - "debug1: kex: algorithm" - negotiated encryption

Multiplexing Connection Issues

Problem: Can't reuse existing connection.

# Check if master is running
ssh -O check user@host

# If stale, manually clean
rm ~/.ssh/sockets/*

# Prevent stale sockets
# ~/.ssh/config
ControlPersist 10m  # Shorter timeout

ProxyJump Through Multiple Bastions

Problem: Complex network topology with multiple jump hosts.

# Chain multiple jumps
ssh -J bastion1,bastion2,bastion3 user@final-destination

# Different users/keys per hop
# ~/.ssh/config
Host bastion1
    HostName bastion1.company.com
    User admin1
    IdentityFile ~/.ssh/key1

Host bastion2
    HostName bastion2.internal
    User admin2
    IdentityFile ~/.ssh/key2
    ProxyJump bastion1

Host production
    HostName prod.deep.internal
    User deploy
    IdentityFile ~/.ssh/prod_key
    ProxyJump bastion2

Cipher/Algorithm Negotiation Failures

Error: no matching cipher found or no matching key exchange method found

Cause: Client and server can't agree on algorithms (common with very old or very new systems).

Debug:

ssh -vvv user@host 2>&1 | grep -i "kex\|cipher\|mac"

Solution 1: Temporarily allow legacy algorithms (client side)

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 \
    -oHostKeyAlgorithms=+ssh-rsa \
    user@legacy-server

Solution 2: Update server to support modern algorithms (preferred)

# /etc/ssh/sshd_config
KexAlgorithms curve25519-sha256,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1

SELinux Blocking SSH on Non-Standard Port

Problem: Changed SSH port, but connections still fail (CentOS/RHEL).

# Check SELinux logs
sudo ausearch -m avc -ts recent | grep sshd

# Allow SSH on port 2222
sudo semanage port -a -t ssh_port_t -p tcp 2222

# Verify
sudo semanage port -l | grep ssh

Broken Pipe / Connection Reset

Problem: Connection drops mid-session with "broken pipe" or "connection reset by peer".

Causes & Solutions:

1. NAT/Firewall timeout

   # Client: ~/.ssh/config
   ServerAliveInterval 30
   ServerAliveCountMax 3
   
   # Server: /etc/ssh/sshd_config
   ClientAliveInterval 30
   ClientAliveCountMax 3
   

2. MTU issues (especially with VPNs/tunnels)

   # Test with smaller packets
   ping -M do -s 1400 remote-host
   
   # If it fails, lower the MTU on the network interface
   sudo ip link set dev eth0 mtu 1400
   

3. Unstable connection → Use Mosh

   # Mosh survives IP changes and roaming
   mosh user@host
   

Debugging Server-Side Issues

Problem: Connection works, but something's wrong server-side.

# Server logs (Ubuntu/Debian)
sudo tail -f /var/log/auth.log | grep sshd

# Server logs (CentOS/RHEL)
sudo tail -f /var/log/secure | grep sshd

# Check sshd config syntax
sudo sshd -t

# Run sshd in debug mode (test port)
sudo /usr/sbin/sshd -d -p 2222

# Then connect: ssh -p 2222 user@localhost

Certificate Debugging

Problem: SSH certificate not working.

# Verify certificate is valid
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

# Check:
# - Valid after/before dates
# - Principals match your username
# - Critical Options don't restrict you

# Test with verbose output
ssh -vvv user@host 2>&1 | grep -i cert

# Common issue: cert file not found
# Ensure both private key and *-cert.pub are in ~/.ssh/

Analysing Slow Connections

# Time different stages
time ssh -o ConnectTimeout=5 user@host exit

# Profile the connection
ssh -v user@host 2>&1 | awk '/debug1:/{print strftime("%T"), $0}'

# Common culprits:
# - DNS lookups (UseDNS yes on server)
# - GSSAPI auth attempts (GSSAPIAuthentication yes)
# - Slow key exchange (old hardware, bad entropy)

---

SSH Escape Sequences

During an active SSH session, press Enter, then ~, then one of these keys:

Example: Add port forward to existing session

~C
ssh> -L 8080:localhost:80
Forwarding port.

This is incredibly useful when you forgot to set up a tunnel initially!

---

Conclusion

You've now explored the advanced capabilities of SSH—from the mathematical foundations of Diffie-Hellman to production-grade security hardening and complex troubleshooting scenarios.

What You've Mastered:

• Deep Cryptography: Understanding how Diffie-Hellman key exchange actually works
• Advanced Tunnelling: Local, remote, and dynamic port forwarding with complex patterns
• Performance: SSH multiplexing for faster connections and reduced overhead
• Network Complexity: ProxyJump for navigating multi-hop topologies
• Enterprise Scale: SSH certificates for managing keys across large infrastructures
• Security Hardening: 2FA, fail2ban, audit logging, and hardware tokens
• Complex Troubleshooting: Cipher negotiations, SELinux, multiplexing issues, and more

Applying These Skills:

1. Audit your infrastructure: Review your current SSH setup against the hardening guidelines
2. Implement multiplexing: Dramatically speed up your daily workflow
3. Set up certificates: If managing 10+ servers, SSH CAs will save you time
4. Harden production: Deploy fail2ban, 2FA, and audit logging on critical systems
5. Document patterns: Create team SSH config templates using these patterns

Advanced Resources:

• OpenSSH Security Advisory - Stay updated on vulnerabilities
• RFC 4251-4256 - SSH Protocol Specifications
• SSH Audit - Automated SSH security scanner
• OpenSSH Source Code - For the truly curious
• NIST SP 800-115 - Technical security testing

Production Checklist:

• ✅ All servers use key-based auth only (passwords disabled)
• ✅ Modern ciphers enforced (ED25519, ChaCha20-Poly1305)
• ✅ fail2ban or equivalent deployed
• ✅ 2FA enabled for privileged access
• ✅ SSH audit logging configured
• ✅ Multiplexing enabled for performance
• ✅ Jump hosts configured for internal network access
• ✅ Regular key rotation policy in place

Remember: Security is not a one-time configuration—it's an ongoing process. Subscribe to security advisories, audit regularly, and stay curious.

Happy (advanced) connecting! 🔐