linux:performance [Starlab knowledge base]

[Show page]
[Old revisions]
[[unknown link type]]
[
]
This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
Original Source: http://www.ahammer.ch/104
RH https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/index.html
HardInfo http://hardinfo.berlios.de/HomePage


=====Linux Performance=====

The following summary is a braindump of many, many pages from the internet and
the tipps and tricks they descriped.

IO Benchmarsks: bonnie++ - IOMeter 

RedHat 6 Deployment notes for high performance and
Performance Tuning Guide


Get to know your hardware first!

The Hardinfo tool might help.


IO
Check out your system. Does it wait for data?
# top
if the %wa column is that high - you have a problem ;-)




ext3/ext4 filesystem
disables some ext3 unused features and reduce the amount of journaling data
noatime,nodiratime,noacl,data=writeback,commit=15

check this our on a running system without reboot...later you can add the
mountoptions to /etc/fstab
# mount -o remount,noatime,nodiratime,noacl,commit=15 /
# mount -o remount,noatime,nodiratime,noacl,commit=15 /home

be carefully with the journal option. It can only be changed during startup. Test it
on a non-productive system first!

another way is to mount the /tmp directory as the TMPFS filesystem.
tmpfs /tmp  tmpfs defaults,size=256M,noatime,noacl,nodiratime   0   0




ext4 has additional options
# mount -o remount,barrier=0 /home




the last bit of performance can be achieved by disabling the journaling,
but be aware that the checkdisk may take forever...
# mkfs -V -t ext4 /dev/sdb1
# tune2fs -c0 -i0 -m0 /dev/sdb1
# tune2fs -o journal_data_writeback /dev/sdb1
# tune2fs -O ^has_journal /dev/sdb1
/dev/sdb1 /mountpoint ext4 defaults,data=writeback,noatime,nodiratime,noacl,barrier=0 0 0

you can check the active features
# tune2fs -l /dev/sdb1



On a busy server the directory index needs to be clean up from time to time. Check if the Options is really active. The filesystem needs to be OFFLINE!
# tune2fs -O dir_index /dev/sda5
# umount /dev/sda5 
# e2fsck -D -f /dev/sda5

Free up more space to use

reduces the root-reserved space on this volume from 5% to 2%.
tune2fs -m2 /dev/sda1 



50% of your disk space used but only 2% of the possible inodes?

Then you should optimize your formating....
mkfs -t ext4 /dev/vg00/lvol0 -i 65536

This reduces the inodes about 8 times. The default settings give 1 inode every 8192 bytes.
Now only every 64K.



ext3 basics and tuning - 
ext4 on SSD -
linux on flash


IO scheduler

 redhat manual

 Using fast SSD or external SAN Storage - free up CPU time by minimizing
 the I/O optimizsation during the scheduling.
# echo "noop" > /sys/block/<device>/queue/scheduler
return to default
# echo "cfq" > /sys/block/<device>/queue/scheduler

depending on the workload - deadline might boost the throughput too - as recommended
by Oracle.


IDE/Sata parameters
Use hdparm to optimize the disk usage
# hdparm -i /dev/sda   Parameter auslesen
# hdparm -I /dev/sda   more infos
# hdparm -Tt /dev/sda  Geschwindigkeit testen


disable the power- and/or accustic management (if supported)
# hdparm -B 255 /dev/sda
 setting Advanced Power Management level to disabled
 APM_level	= off
# hdparm -M 254 /dev/sda
 setting acoustic management to 254

Configuring Disk Read-Ahead
Set read-ahead cache to 512KB
/sbin/blockdev --setra 1024 /dev/cciss/c0d0
/sbin/blockdev --setra 1024 /dev/cciss/c0d1



NIC &TCP/IP

Check the nic settings first - with
ethtool
# ethtool eth0
If it 1000MB/s and Fullduplex ... the go on and check the NIC offload functions. Go on with
the driver in use
# ethtool -i eth0
google the driver in use for more tuning settings or 
# modinfo <drivername>
an INTEL e1000 drivers can tweak the descriptions values from the default 256 to the present max. value 4096.
# ethtool -G rx 4096 tx 4096
# ethtool -g eth0

Now continue with the offload engine
# ethtool --show-offload eth0
Change the settings
--K --offload  Changes the offload parameters of the specified ethernet device.
rx on|off     Specifies whether RX checksumming should be enabled.
tx on|off     Specifies whether TX checksumming should be enabled.
sg on|off     Specifies whether scatter-gather should be enabled.
tso on|off    Specifies whether TCP segmentation offload should be enabled.
ufo on|off    Specifies whether UDP fragmentation offload should be enabled
gso on|off    Specifies whether generic segmentation offload should be enabled

Another NIC option is the transmit queue length.
# ifconfig eth0 | grep "txqueuelen"
ifconfig eth0 txqueuelen 2000

TCP Segmentation offload


The TCP/IP stack can be modified for bigger buffers and more possible window ranges.
# sysctl net.core.rmem_max
# sysctl net.core.wmem_max
# sysctl -w net.core.rmem_max=16777216
# sysctl -w net.core.wmem_max=16777216
On SAP Systems
# sysctl -w net.ipv4.ip_local_port_range="1024 65000"
# sysctl -w net.core.rmem_default=262144
# sysctl -w net.core.wmem_default=262144
# sysctl -w net.core.rmem_max=262144
# sysctl -w net.core.wmem_max=262144
# sysctl net.ipv4.tcp_rmem
# sysctl net.ipv4.tcp_wmem
# sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
# sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

Depending on the kernel in use, be sure that the following auto-tuning settins are active.
# sysctl net.ipv4.tcp_window_scaling
# sysctl net.ipv4.tcp_timestamps
# sysctl net.ipv4.tcp_sack

If possible from the hardware slots and BIOS DO NOT SHARE IRQ amoung
disk- and network controller(s).


Increase - if possible - the MTU ~9000 (aka. Jumbo Frames)
# ping -s 8400 -M do <ip address>
c:> ping -f -l 8000 <ip address>

Some CISCO commands - depending on the model
config
    mtu 9216
enable
    system mtu 8000

and verify
show port jumbo
show system mtu



syslog
Tell syslog not to sync the disc after every entry. Add an - to the configuration.
*.info;mail.none;authpriv.none;cron.none		-/var/log/messages
If IO is really an issue - you can try disable some rarly used log files - such as 
debug, cron or maillog

Get rid of the -- MARK -- entries in syslog
use -m 0 option




Need more IO? Try changing to RAID-1 (e.g. from RAID-5) or even RAID-10 or get a
SAN-Storage...

 



Wikipedia ext3 - 
hdparm - 
syslog - 
Lesswatts DISK -
iozone.org Filesystem benchmark


memory management
tune some kernel parameters
Using sysctl.conf to modify some kernel memory options - try it only after studying the online documentation and measuring the results!
vm.dirty_writeback_centisecs=1500
vm.swappiness=1
vm.vfs_cache_pressure=50

oracle suggests the following values
# less filesystemcache [oracle disables it on some files anyway] in favors of loaded programs
vm.swappiness=10
# start flushing out dirty memory pages at 3% of the system memory using pdflush
vm.dirty_background_ratio=3
# start flusing out dirty filesystem cache pages at 15% of the system memory
vm.dirty_ratio=15
# after 5sec a dirty page has to be written to disk
vm.dirty_expire_centisecs=500
# wake up pdflush every second to check if there are any dirty pages to flush
vm.dirty_writeback_centisecs=100

to make the change permanent - add them to /etc/sysctl.conf and activate them
# sysctl -p


vm-tuning
 - 
pdflush theory -
Kernel vm.txt


disable unused services and processes
The classic phrase, disable all unneeded services - I think you have heard it already.
$ who -r
$ ps ax

Check out the startup procedures (on Fedora/Centos/RedHat Enterprise)
# chkconfig --list

Disable unused services...
# chkconfig SERVICENAME off
# service SERVICENAME stop



Most of the distributions start several login consoles during /etc/inittab. More than
2 should not really being needed.
c1:1235:respawn:/sbin/agetty 38400 tty1 linux
c2:1235:respawn:/sbin/agetty 38400 tty2 linux
#c3:1235:respawn:/sbin/agetty 38400 tty3 linux
#c4:1235:respawn:/sbin/agetty 38400 tty4 linux
#c5:1235:respawn:/sbin/agetty 38400 tty5 linux
#c6:12345:respawn:/sbin/agetty 38400 tty6 linux

let init reload the new inittab
# telinit q
if the services do not stop by itself - kill them...


cron jobs
Especially if your computer runs 24x7 - keep an eye on the cronjobs [monthly, weekly, daily and the rest]. Time them not to run on the same time as the backup.


Try to use a tickless - kernel. Besides much better powersaving, the kernel doesnt
need to poll its internal todolist and with the much improved schedules in 2.6 the whole
system should be more responsive.


do your own compilations
compiler options
I you compile your own binaries - use the compiler switches to optimize the binaries
onto your own instrastructure.
$ cat /proc/cpuinfo
CFLAGS="-march=prescott -O3 -pipe -fomit-frame-pointer"

in more recent gcc versions there is a generic way without the need to know the exact type of cpu
CFLAGS="-march=native -O3 -pipe -fomit-frame-pointer"

in doubt - check the CPUs capability - either using the kernel or gcc itself
# cat /proc/cpuinfo | grep flags
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 
3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt 
lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw 
ibs skinit wdt npt lbrv svm_lock nrip_save
# gcc -c -Q -march=native --help=target
The following options are target specific:
  -m128bit-long-double        		[disabled]
  -m32                        		[disabled]
  -m3dnow                     		[disabled]
  -m3dnowa                    		[disabled]
  -m64                        		[enabled]
  -m80387                     		[enabled]
  -m96bit-long-double         		[enabled]
...




GNU Compiler Collection Summary -
more safe CFLAGS
 
Usage samples 
apache /
php

Shared vs. static build - Tutorial 



do not forget to check if the binaries can be stripped
# strip /opt/rrdtools/bin/*

enable only used modules

Doing your own compilation you get the chance to enable only the modules needed and
disable the rest.

This will reduce the size of the binary and let you build a lean system
$ ./configure --enable-ssl --disable-userdir --disable-imagemap ...

consult the online help for the possible configurations or do ...
$ ./configure --help



kernel
The most useful hit ever - build your own kernel. Try to include the necessary drivers
als buildin - not as module. strip all not used parts, such as ATM, ISDN...
# cat /proc/version
# cat /proc/cpuinfo
# cat /proc/modules

To start with - look at the file /boot/config-$(uname -r) of the installed distribution. 
# make localmodconfig        on certain systems you need to update the initrd
or
# make localyesconfig        to add all modules static in the kernel

produces a new config with only the present active modules configured - this gives you a very
minimal kernel!
# make -j 4 bzImage modules
# make modules_install headers_install install



If you need to crosscompile a kernel...
[target]  # lsmod > module_ziel
[compile] # make LSMOD=module_ziel localmodconfig
[compile] # make bzImage
copy the whole /usr/src/linux tree to the target server
[target]  # make modules_install install



the kernel optimization knowhow is based on an article from the german
cT magazin 05/2011 on page 182


applications
apache
get an overview of the apache at hand
# ./apachectl -v
# apachectl -M                     dump all modules
# apachectl -S                     dump all virtual hosts

And do not forget - as always after modifying the httpd.conf or any included files - run a
synatex check and reload the service graceful.
# apachectl -t
# apachectl -k graceful



Multi-Processing Modules (MPMs)

Using linux the preferred and traditional MPM is called prefork.
<IfModule mpm_prefork_module>_
StartServers         2
MinSpareServers      1
MaxSpareServers      1
MaxClients          10
MaxRequestsPerChild  0
</IfModule>


The default settings start at least 5 processes with another 5 spare processes. On a system
with only few users ... way to much. 

Newer version work more efficient with the worker MPM
./configure --with-mpm=worker
<IfModule mpm_worker_module>
StartServers          2
MaxClients          150
MinSpareThreads      10
MaxSpareThreads      50 
ThreadsPerChild      25
MaxRequestsPerChild   0
</IfModule>



Reduce IO and network traffic by preventing, caching and compression

disable .htaccess usage
AllowOverride None

on a website there are always some static files such as icons, images, styles and scripts.
Why not cache them on the browsers! This will reduce the load on the server and allows faster display.
<Directory /some/path/to/scripts_images_styles>
 ExpiresActive On
 ExpiresByType image/x-icon "access plus 20 minutes"
 ExpiresByType image/gif "access plus 20 minutes"
 ExpiresByType image/jpg "access plus 20 minutes"
 ExpiresByType image/jpeg "access plus 20 minutes"
 ExpiresByType image/png "access plus 20 minutes"
 ExpiresByType text/css "access plus 10 minutes"
 ExpiresByType text/javascript "access plus 20 minutes"
 ExpiresByType text/x-javascript "access plus 20 minutes"
 ExpiresByType application/javascript "access plus 20 minutes"
 ExpiresByType application/x-javascript "access plus 20 minutes"
 ExpiresDefault "now"
</Directory>

reduce the amount of traffic to transfer by compressing it before sending
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE image/x-icon 
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/x-javascript
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

A simple online web page compression / deflate / gzip test tool - 
How To Optimize Your Site With GZIP Compression


Logging

Apache logs most activities. Not all are needed.... Check out the options
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn
LogFormat "%h %l %u %t "%r" %>s %b" common
CustomLog "logs/access_log" common

to fit your requirements.



disable the logging completely use
CustomLog /some/path combined env=DOES_NOT_EXIST


httpd.apache.org

mysql

Mysql can be quite a beast - especially if you host several Typo3 CMS on a server with a DB volume of over 3GByte. A look at its numbers is worth a while. 
# mysql --version

Before you start - make sure you have a working backup - just in case ;-)
$ mysqldump --user=<name> --password=<pwd> --all-databases --opt --result-file=all.sql
or
$ mysqldump --user=<name> --password=<pwd> <db> --opt --result-file=<db>.sql


Start with this presentation (german)

Use the tuning-primer.sh 
(website) script to optimize some key options. (sample)

Next would be to disable the binary log. It is used for cluster replication and
point in time restores. If you use neither nor - disable it
#log-bin

A reset might be handy before disabling to reduce the logfile-size
reset master;

Another area are the DB-Engines. The newer InnoDB module adds a lot of sql functionality
to mysql for the price of performance. 

If you do not need the additional sql - disable it and stick to the traditional, fast Isamchk. 
disable all #innodb_... entries in my.cnf
and add
skip-innodb

Be aware that InnoDB tables have to be
converted!
alter table tblName ENGINE=MYISAM;

innoDB tutorial


Is mysql still hogging your CPU? Try cleaning up the db content. Sometime there are
cache tables, outdated indexes or fragmentated databases to clean up.

First get a picture of the tables available.
show table status like index_fulltext;

If you have a lot of tables - you might use the
showtablesize.sh to automatically print out
all tables in the database given. [show table status reference]

Sometimes, people forget the maintenance tasks - so the sysadmin has to do the cleanup. e.g. for index_ and cache_ tables in typo3, use the truncate function to clean them
[truncate reference]
truncate table index_fulltext;

After all the cleanup optimize the tables to regain full performance. 

BE AWARE that during optimizing the tables are LOCKED by mysql.
optimize table index_fulltext;

[optimize table reference]



additional performance can be achieved by optimizing the SQL code - or to be more precise
use additional attribute indexes to tune the SQL internal operation. 

e.g. for Cacti
CREATE INDEX `data_template_data_id` ON `data_input_data` (`data_template_data_id`);
...
CREATE INDEX `host_id_rrd_next_step` ON poller_item (`host_id`,`rrd_next_step`);




MySql 5.0 Reference - 
mysqltuner.com alternative checkscript -
optimize Tables with cronjob - 
some handy sql statements - 
mysql basic troubleshooting


php

configuration

be sure to enable persistent mysql connections...
mysql.allow_persistent = On
mysql.max_persistent = -1
mysql.max_links = -1
mysql.connect_timeout = 60
mysql.trace_mode = Off



misc...
startup time
If not needed, disable the kernel chooser prompt during LILO/GRUB and put the
kernel in Quiet-Mode. 
#prompt
append = " quiet"

Boot Time
# vi lilo.conf
compact

Also try to "compact" BIOS read calls, as there are the slowest in
the world....


disable some default ext3 check during startup - but do not forget to do them
manually during a service window!
# tune2fs -c 0 -i 0 /dev/vg00/lvol1

Tracing the most powerconsuming tasks with 
powertop.

- Kernel Options

- X11 Device Option: Option "NoDRI" reduces the RADEON Interrups

- USB Autosuspend : usbcore.autosuspend=1

- CPU Governor ondemand for both cpus..

- setterm in rc.M
linux/performance.txt · Last modified: 2013/05/14 12:23 by admin