Linux 內核自帶了一個性能分析工具叫 perf。它能做函數級和指令級的熱點采樣，也能配合 tracepoint 采集系統調用、網絡事件、文件系統操作等內核事件。因為代碼就在內核源碼樹里，算得上是 Linux 平臺上最順手的性能工具了。
perf 基于內核的性能計數器子系統，硬件層面利用 CPU 的 PMU（Performance Monitoring Unit），軟件層面依賴內核的 tracepoint 和軟件計數器。
大致原理是：每隔一個固定時間，CPU 產生一個中斷，記下當前跑的是哪個進程、哪個函數，累加對應的計數。多采幾次就知道 CPU 時間主要花在了哪里。
整體架構分兩層：
sudo apt install linux-tools-common linux-tools-generic linux-tools-`uname -r`
perf --help
usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
The most commonly used perf commands are:
annotate Read perf.data and display annotated code
archive Create archive with object files with build-ids
bench General framework for benchmark suites
buildid-cache Manage build-id cache.
buildid-list List the buildids in a perf.data file
c2c Shared Data C2C/HITM Analyzer.
config Get and set variables in a configuration file.
data Data file related processing
diff Read perf.data files and display the differential profile
evlist List the event names in a perf.data file
ftrace simple wrapper for kernel's ftrace functionality
inject Filter to augment the events stream with additional information
kallsyms Searches running kernel for symbols
kmem Tool to trace/measure kernel memory properties
kvm Tool to trace/measure kvm guest os
list List all symbolic event types
lock Analyze lock events
mem Profile memory accesses
record Run a command and record its profile into perf.data
report Read perf.data (created by perf record) and display the profile
sched Tool to trace/measure scheduler properties (latencies)
script Read perf.data and display trace output
stat Run a command and gather performance counter statistics
test Runs sanity tests.
timechart Tool to visualize total system behavior during a workload
top System profiling tool.
version display the version of perf binary
probe Define new dynamic tracepoints
trace strace inspired tool
幾個常用命令的簡要說明：
perf list
perf list 'sched:*'
perf stat command # 統計命令的 CPU 計數器
perf stat -d command # 詳細統計
perf stat -p PID # 統計指定進程
perf stat -a sleep 5 # 全系統 5 秒
perf stat -e cycles,instructions,cache-references,cache-misses,bus-cycles -a sleep 10
perf stat -e L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores command
perf stat -e LLC-loads,LLC-load-misses,LLC-stores,LLC-prefetches command
perf stat -e raw_syscalls:sys_enter -I 1000 -a # 每秒系統調用數
perf record -F 99 command # 99Hz 采樣命令
perf record -F 99 -p PID -g -- sleep 10 # 采樣 + 調用棧
perf record -F 99 -ag -- sleep 10 # 全系統 99Hz，10 秒
perf record -e L1-dcache-load-misses -c 10000 -ag -- sleep 5 # 按 Cache Miss 采樣
perf record -e cycles:k -a -- sleep 5 # 只采內核態
perf record -e cycles:u -a -- sleep 5 # 只采用戶態
perf top -F 49
perf top -F 49 -ns comm,dso
perf record -e sched:sched_process_exec -a
perf record -e context-switches -a
perf record -e 'ext4:*' -o /tmp/perf.data -a
perf record -e vmscan:mm_vmscan_wakeup_kswapd -ag
perf probe --add tcp_sendmsg # 添加探測點
perf probe -d tcp_sendmsg # 刪除
perf probe 'tcp_sendmsg%return' # 探測返回值
perf probe -V tcp_sendmsg # 查看可用變量
perf probe -L tcp_sendmsg # 查看可用行號
perf probe -x /lib64/libc.so.6 malloc # 用戶態探測
perf probe -l # 列出當前探測點
perf report # TUI 模式
perf report -n # 顯示采樣計數
perf report --stdio # 文本輸出
perf script # 列出所有事件
perf script --header -F comm,pid,tid,cpu,time,event,ip,sym,dso
perf annotate --stdio # 反匯編注解
END
作者：仲一Linux
