Introduction

“My logs show nothing suspicious.” That’s because your logs are lying. Standard Application Logs (STDOUT) only show what the developer chose to print. Rootkits and sophisticated malware operate in the kernel shadows, intercepting syscalls and modifying logs before they touch the disk.

This lesson explores Deep Observability: Tracing the actual physics of the Kernel and ensuring log integrity with Cryptography.

---

The Linux Audit System (Auditd)

auditd listens to the kernel’s internal event stream. It is not an application log; it is a Syscall Recorder.

The Physics of a Trace

When you run cat /etc/passwd:

1. Syscall: openat(AT_FDCWD, "/etc/passwd", O_RDONLY)
2. Kernel Hook: The Audit subsystem pauses the CPU thread.
3. Filter: Checks rules. Does this match a monitored path?
4. Record: Writes event to audit.log (Binary format) before the file is even opened.

Code: Configuring Audit Checks

# /etc/audit/rules.d/audit.rules

# 1. Watch critical file writes (Permissions Physics)
-w /etc/passwd -p wa -k identity_theft

# 2. Watch execution of sensitive binaries
-a always,exit -F arch=b64 -S execve -k command_exec

# 3. Lock the configuration (Immutable Mode)
-e 2

Physics Note: Once -e 2 is set, auditd cannot be stopped or changed until Reboot. This prevents an attacker who gets Root from disabling auditing.

---

eBPF: The X-Ray Machine

Even auditd can be bypassed by advanced rootkits that modify the syscall table. eBPF (Extended Berkeley Packet Filter) is the ultimate source of truth. It runs sandboxed code inside the kernel to observe functions as they execute.

Tracing a “Fileless” Attack

Malware often executes purely in memory (no disk file). Traditional AV scanning files on disk sees nothing. eBPF traces the mmap and mprotect syscalls used to create executable memory pages.

# Using bpftrace to watch for executable memory allocation
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_mprotect /args->prot & PROT_EXEC/ { printf("Process %s is making memory executable!\n", comm); }'

---

Log Integrity: The Merkle Tree

Hackers delete logs. How do you prove a log entry wasn’t deleted? Merkle Chains.

The Physics: Log Entry N includes the Hash of Log Entry N-1. $Hash(N) = SHA256(Data_N + Hash(N-1))$

If an attacker deletes Entry 50, the Hash of Entry 51 breaks. The chain snaps. To delete one log, they must recalculate the hashes of ALL subsequent logs. If you stream these hashes to a separate “Write-Only” server (or Blockchain), modification becomes mathematically impossible.

---

Practice Exercises

Exercise 1: auditd setup (Beginner)

Task: Install auditd. Watch /tmp/testfile. Action: Modify the file. Observation: Read /var/log/audit/audit.log and decode the hex-encoded typical output using ausearch.

Exercise 2: The Immutable Flag (Intermediate)

Task: TBD (Dangerous). Concept only: chattr +i vs auditd -e 2. Constraint: Do not actually lock your production machine.

Exercise 3: eBPF Spy (Advanced)

Task: Use execsnoop (from bcc-tools). Action: Run it in one terminal. Run ls in another. Observation: See the ls execution even if you hide the process from ps.

---

Knowledge Check

1. What is the difference between syslog and auditd?
2. Why is -e 2 critical in audit rules?
3. How does eBPF bypass syscall hooking rootkits?
4. What mathematical structure prevents log deletion?
5. Does auditd impact system performance?

Answers

1. Depth. Syslog is application-level text. Auditd is kernel-level syscall tracing.
2. Immutability. It prevents changes until reboot.
3. Internal Instrumentation. It hooks kernel functions directly (kprobes), not only the syscall table interface.
4. Merkle Chain (Hash Chain).
5. Yes. Inspecting every syscall adds context switch overhead. Use filters wisely.

---

Summary

• Auditd: The Kernel’s Flight Recorder.
• eBPF: The X-Ray for deeper visibility.
• Immutability: -e 2 locks the door.
• Integrity: Hashing chains prevent history rewriting.

---

---

Pro Version: See the full research: Security Architecture for Trading Firms