Scribd
Upload
29 views

Unit 5 - Linux System Performance

This document discusses six facets of Linux performance analysis: observability, methodologies, benchmarking, profiling, tracing, and tuning. It provides examples of various Linux tools that can be used for observability including uptime, top, vmstat, iostat, free, strace, tcpdump, netstat, slabtop, pcstat, and perf_events. It also discusses different methodologies that can be used for performance analysis such as the Linux Performance Analysis in 60 Seconds method, the USE method, CPU profiling method, resource analysis, and workload analysis.

Uploaded by

sharmashivi0122
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
29 views

Unit 5 - Linux System Performance

This document discusses six facets of Linux performance analysis: observability, methodologies, benchmarking, profiling, tracing, and tuning. It provides examples of various Linux tools that can be used for observability including uptime, top, vmstat, iostat, free, strace, tcpdump, netstat, slabtop, pcstat, and perf_events. It also discusses different methodologies that can be used for performance analysis such as the Linux Performance Analysis in 60 Seconds method, the USE method, CPU profiling method, resource analysis, and workload analysis.

Uploaded by

sharmashivi0122
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 27

Systems Performance

Agenda
A brief discussion of 6 facets of Linux performance:

1. Observability
2. Methodologies
3. Benchmarking
4. Profiling
5. Tracing
6. Tuning

Audience: Everyone (DBAs, developers, operations, …)


1. Observability
How do you measure these?
Linux Observability Tools
Observability Tools
• Tools showcase common metrics
– Learning Linux tools is useful even if you never use them:
the same metrics are in GUIs
• Netflix usually use these metrics via:
– Netflix Atlas: cloud-wide monitoring
– Netflix Vector: instance analysis
• Linux has many tools
– Plus many extra kernel sources of data that lack tools, are
harder to use, and are practically undocumented
• Some tool examples…
uptime
• One way to print load averages:
$ uptime
07:42:06 up 8:16, 1 user, load average: 2.27, 2.84, 2.91

•A measure of resource demand: CPUs + disks


– Other OSes only show CPUs: easier to interpret
• Exponentially-damped moving averages
• Time constants of 1, 5, and 15 minutes
– Historic trend without the line graph
• Load > # of CPUs, may mean CPU saturation
– Don’t spend more than 5 seconds studying these
top (or htop)
• System and per-process interval summary:

$ top - 18:50:26 up 7:43, 1 user, load average: 4.11, 4.91, 5.22


Tasks: 209 total, 1 running, 206 sleeping, 0 stopped, 2 zombie
Cpu(s): 47.1%us, 4.0%sy, 0.0%ni, 48.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.2%st
Mem: 70197156k total, 44831072k used, 25366084k free, 36360k buffers
Swap: 0k total, 0k used, 0k free, 11873356k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5738 apiprod 20 0 62.6g 29g 352m S 417 44.2 2144:15 java
1386 apiprod 20 0 17452 1388 964 R 0 0.0 0:00.02 top
1 root 20 0 24340 2272 1340 S 0 0.0 0:01.51 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
[…]

• %CPU is summed across all CPUs


• Can miss short-lived processes (atop won’t)
• Can consume noticeable CPU to read /proc
htop
vmstat
iostat
free
• Main memory usage:

$ free -m
total used free shared buffers cached
Mem: 3750 1111 2639 0 147 527
-/+ buffers/cache: 436 3313
Swap: 0 0 0

• buffers: block device I/O cache


• cached: virtual page cache
strace
• System call tracer:
$ strace –tttT –p 313
1408393285.779746 getgroups(0, NULL) = 1 <0.000016>
1408393285.779873 getgroups(1, [0]) = 1 <0.000015>
1408393285.780797 close(3) = 0 <0.000016>
1408393285.781338 write(1, "LinuxCon 2014!\n", 15LinuxCon 2014!
) = 15 <0.000048>

• Eg, -ttt: time (us) since epoch; -T: syscall time (s)
• Translates syscall args
– Very helpful for solving system usage issues
• Currently has massive overhead (ptrace based)
– Can slow the target by > 100x. Use extreme caution.
tcpdump
• Sniff network packets for post analysis:
$ tcpdump -i eth0 -w /tmp/out.tcpdump
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C7985 packets captured
8996 packets received by filter
1010 packets dropped by kernel
# tcpdump -nr /tmp/out.tcpdump | head
reading from file /tmp/out.tcpdump, link-type EN10MB (Ethernet)
20:41:05.038437 IP 10.44.107.151.22 > 10.53.237.72.46425: Flags [P.], seq 18...
20:41:05.038533 IP 10.44.107.151.22 > 10.53.237.72.46425: Flags [P.], seq 48...
20:41:05.038584 IP 10.44.107.151.22 > 10.53.237.72.46425: Flags [P.], seq 96...
[…]

• Study packet sequences with timestamps (us)


• CPU overhead optimized (socket ring buffers), but can
still be significant. Use caution.
netstat
• Various network protocol statistics using -s:
• A multi-tool:
-i: interface stats
-r: route table
default: list conns
• netstat -p: shows
process details!
• Per-second interval
with -c
slabtop
• Kernel slab allocator memory usage:
pcstat
• Show page cache residency by file:
# ./pcstat data0*
|----------+----------------+------------+-----------+---------|
| Name | Size | Pages | Cached | Percent |
|----------+----------------+------------+-----------+---------|
| data00 | 104857600 | 25600 | 25600 | 100.000 |
| data01 | 104857600 | 25600 | 25600 | 100.000 |
| data02 | 104857600 | 25600 | 4080 | 015.938 |
| data03 | 104857600 | 25600 | 25600 | 100.000 |
| data04 | 104857600 | 25600 | 16010 | 062.539 |
| data05 | 104857600 | 25600 | 0 | 000.000 |
|----------+----------------+------------+-----------+---------|

• Uses the mincore(2) syscall. Useful for database


performance analysis.
Perf_events
• Provides the "perf" command
• In Linux source code: tools/perf
– Usually pkg added by linux-tools-common, etc.
• Multi-tool with many capabilities
– CPU profiling
– PMC profiling
– Static & dynamic tracing
• Covered later in Profiling & Tracing
Where do you start?...and stop?
2. Methodologies
Anti-Methodologies
• The lack of a deliberate methodology…
• Street Light Anti-Method:
– 1. Pick observability tools that are
• Familiar
• Found on the Internet
• Found at random

– 2. Run tools
– 3. Look for obvious issues
• Drunk Man Anti-Method:
– Tune things at random until the problem goes away
Methodologies
• Linux Performance Analysis in 60 seconds
• The USE method
• CPU Profile Method
• Resource Analysis
• Workload Analysis
• Others include:
– Workload characterization
– Drill-down analysis
– Off-CPU analysis
– Static performance tuning
– 5 whys
– …
Linux PerfAnalysisin 60s
1. uptime
2. dmesg | tail
3. vmstat 1
4. mpstat -P ALL 1
5. pidstat 1
6. iostat -xz 1
7. free -m
8. sar -n DEV 1
9. sar -n TCP,ETCP 1
10. top
Linux Perf Analysis in 60s
The USE method
USE method for hardware

You might also like