perf record -a --call-graph dwarf -p 29052
perf report --call-graph
perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > perf.svg

       The overhead can be shown in two columns as Children and Self when
       perf collects callchains. The self overhead is simply calculated by
       adding all period values of the entry - usually a function (symbol).
       This is the value that perf shows traditionally and sum of all the
       self overhead values should be 100%.

       The children overhead is calculated by adding all period values of
       the child functions so that it can show the total overhead of the
       higher level functions even if they don’t directly execute much.
       Children here means functions that are called from another (parent)
       function.

sudo perf report --no-children

./FlameGraph/flamegraph.pl --inverted --reverse

sudo perf script -l #查看当前支持的内置脚本

perf record -a -g -F 99 sleep 60
perf script report flamegraph

sudo perf stat -d perf_test
sudo perf list #可以查看支持的event列表 block/ext4/syscalls/net/sched/kmem/task
sudo perf stat -e 'syscalls:sys_enter_*'  command #查看执行的系统调用
sudo perf stat -e 'sched:*' -p PID #查看某个进程的调度事件

-e：指定性能事件
-a：显示在所有CPU上的性能统计信息
-C：显示在指定CPU上的性能统计信息
-p：指定进程PID
-t：指定线程TID
-K：隐藏内核统计信息
-U：隐藏用户空间的统计信息

# Sample CPU stack traces for the PID, using dwarf (dbg info) to unwind stacks, 
# at 99 Hertz, for 10 seconds:
perf record -F 99 -p PID --call-graph dwarf sleep 10
perf script

sudo perf record -e sched:sched_stat_sleep -e sched:sched_switch \
    -e sched:sched_process_exit -a -g -o perf.data.raw sleep 1
sudo perf inject -v -s -i perf.data.raw -o perf.data
sudo perf script -f comm,pid,tid,cpu,time,period,event,ip,sym,dso,trace | awk '
    NF > 4 { exec = $1; period_ms = int($5 / 1000000) }
    NF > 1 && NF <= 4 && period_ms > 0 { print $2 }
    NF < 2 && period_ms > 0 { printf "%s\n%d\n\n", exec, period_ms }' | \
    ./stackcollapse.pl | \
    ./flamegraph.pl --countname=ms --title="Off-CPU Time Flame Graph" --colors=io > offcpu.svg

sudo perf probe --add tcp_sendmsg #--add是可选参数，默认值
sudo perf probe -l #查看当前增加的tracepoints
sudo perf probe -d tcp_sendmsg #删除tracepoint

sudo perf probe -V tcp_sendmsg --externs #查看tcp_sendmsg的可用参数
sudo perf probe -L tcp_sendmsg #查看可供probe代码行

sudo perf probe 'tcp_sendmsg %ax %dx %cx' #增加一个tracepoint 并记录寄存器值
sudo perf probe 'tcp_sendmsg size sk->__sk_common.skc_state' #记录当时可用参数值
sudo perf probe 'tcp_sendmsg%return $retval' #返回时的tracepoint，记录返回值

sudo perf record -e probe:tcp_sendmsg --filter 'size > 0 && skc_state != 1' -a

# Add a tracepoint for the user-level malloc() function from libc:
perf probe -x /lib64/libc.so.6 malloc

# Add a tracepoint for this user-level static probe (USDT, aka SDT event):
perf probe -x /usr/lib64/libpthread-2.24.so %sdt_libpthread:mutex_entry

[admin@i32f09086.sqa.eu95 /home/admin]
$sudo perf probe -x binary_path -V tc_new
Available variables at tc_new
        @<tc_new+0>
                size_t  size

[admin@i32f09086.sqa.eu95 /home/admin]
$sudo perf probe -x binary_path  --add tc_delete -v
Open Debuginfo file:binary_path
Try to find probe point from debuginfo.
Probe point found: tc_delete+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/uprobe_events write=1
Added new event:
Writing event: p:probe_pangu/tc_delete binary_path:0x240f910
Failed to write event: Invalid argument
  Error: Failed to add events. Reason: Invalid argument (Code: -22)

$sudo perf probe -x binary_path   --add "tc_new size"
Added new event:
  probe_pangu:tc_new   (on tc_new in binary_path with size)

You can now use it in all perf tools, such as:

  perf record -e probe_pangu:tc_new -aR sleep 1

$sudo perf probe -l
  probe_pangu:tc_new   (on tc_new@src/tcmalloc.cc in binarypath with size)

$sudo perf record -e probe_pangu:tc_new -aR sleep 1

sudo perf probe -x \
binary_path --funcs \
--no-demangle --filter '*' >rg_funcs  #列出所有函数

PROBE SYNTAX
       Probe points are defined by following syntax.

           1) Define event based on function name
            [EVENT=]FUNC[@SRC][:RLN|+OFFS|%return|;PTN] [ARG ...]

           2) Define event based on source file with line number
            [EVENT=]SRC:ALN [ARG ...]

           3) Define event based on source file with lazy pattern
            [EVENT=]SRC;PTN [ARG ...]

       EVENT specifies the name of new event, if omitted, it will be set the name of the probed function. Currently, event group name is set as probe. FUNC specifies
       a probed function name, and it may have one of the following options; +OFFS is the offset from function entry address in bytes, :RLN is the relative-line
       number from function entry line, and %return means that it probes function return. And ;PTN means lazy matching pattern (see LAZY MATCHING). Note that ;PTN
       must be the end of the probe point definition. In addition, @SRC specifies a source file which has that function. It is also possible to specify a probe point
       by the source line number or lazy matching by using SRC:ALN or SRC;PTN syntax, where SRC is the source file path, :ALN is the line number and ;PTN is the lazy
       matching pattern. ARG specifies the arguments of this probe point, (see PROBE ARGUMENT).
       
PROBE ARGUMENT
       Each probe argument follows below syntax.

           [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE]

       NAME specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var→field, var.field2), local
       array with fixed index (e.g. array[1], var→array[0], var→pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this
       argument will be set as the last member name if you specify a local data structure member (e.g. field2 for var→field1.field2.) $vars special argument is also
       available for NAME, it is expanded to the local variables which can access at given probe point. TYPE casts the type of this argument (optional). If omitted,
       perf probe automatically set the type based on debuginfo. You can specify string type only for the local variable or structure member which is an array of or a
       pointer to char or unsigned char type.

       On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid.
       
FILTER PATTERN
           The filter pattern is a glob matching pattern(s) to filter variables.
           In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".

       e.g. With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar". With --filter "!foo* & *bar", perf probe -V shows variables
       which don’t start with "foo" and end with "bar", like "fizzbar". But "foobar" is filtered out.