Finally, a more detailed writeup on NFS configuration which I was hoping to blog. Finally it is done.
Do read Configuring NFS Server for Performance
Showing posts with label Performance. Show all posts
Showing posts with label Performance. Show all posts
Configure TCP for faster connections and transfers
On a default Linux Box, the TCP settings may not be optimise for “bigger” available network bandwidth connections and transfer available for 100MB+. Currently, most TCP settings are optimise for 10MB settings. For more information, on how to configure, read my Linux Cluster Blog Configure TCP for faster connections and transfers
Switching off CPU Throttling on CentOS or Fedora
Under CentOS and Fedora, you can switch off CPU Throttling or "Dynamic Frequency Scaling" to maximise your CPU performance. For more information of CPU Throttling, you can also read Dynamic Frequency Scaling from Wikipedia. Just type the command
For Debian-based hardware, you may want to take a look at
Looking at Xorg High CPU Usage Issue for netbook
# /usr/bin/cpufreq-selector -g performance
For Debian-based hardware, you may want to take a look at
Looking at Xorg High CPU Usage Issue for netbook
Looking at Xorg High CPU Usage Issue for netbook
If you are suffering from excessive slowness from your netbook or your old and slow notebook, you may want to tweak the CPU frequency governor to performance mode instead of the general mode. How do you do that? If you are using Linux Mint, you have to install a utilities called cpufrequtils
Step 1: Install CPU Frequency Utilities
# apt-get install cpufrequtils
Step 2: Query the CPU frequency governors
# cpfreq-info
Step 3: Force the CPU to use performance mode
# cpufreq-set -g performance
If this does not resolve the issues, you really have to look at other possibilities that is impacting your Xorg. Sometimes it could be your video drivers not the CPU governor issues....
Step 1: Install CPU Frequency Utilities
# apt-get install cpufrequtils
Step 2: Query the CPU frequency governors
# cpfreq-info
Step 3: Force the CPU to use performance mode
# cpufreq-set -g performance
If this does not resolve the issues, you really have to look at other possibilities that is impacting your Xorg. Sometimes it could be your video drivers not the CPU governor issues....
GPFS Tuning Parameters
GPFS Tuning Parameters is a good wiki information resource written by IBM for GPFS Tuning. Just parroting some of the useful tips I have learned
To view the configuration parameters that has been changed from the default
To view the active value of any of these parameters you can run
To change any of these parameters use mmchconfig. For example to change the pagepool setting on all nodes.
1. Consideration to modify the PagePool
A. Sequential I/O
The default pagepool size may be sufficient for sequential IO workloads, however, a recommended value of 256MB is known to work well in many cases. To change the pagepool size
If the file system blocksize is larger than the default (256K), the pagepool size should be scaled accordingly. For example, if 1M blocksize is used, the default 64M pagepool should be increased by 4 times to 256M. This allows the same number of buffers to be cached.
B. Random I/O
The default pagepool size will likely not be sufficient for Random IO or workloads involving a large number of small files. In some cases allocating 4GB, 8GB or more memory can improve workload performance.
C. Random Direct IO
For database applications that use Direct IO, the pagepool is not used for any user data. It's main purpose in this case is for system metadata and caching the indirect blocks of the database files.
D. NSD Server
Assuming no applications or Filesystem Manager services are running on the NSD servers, the pagepool is only used transiently by the NSD worker threads to gather data from client nodes and write the data to disk. The NSD server does not cache any of the data. Each NSD worker just needs one pagepool buffer per operation, and the buffer can be potentially as large as the largest filesystem blocksize that the disks belong to. With the default NSD configuration, there will be 3 NSD worker threads per LUN (nsdThreadsPerDisk) that the node services. So the amount of memory needed in the pagepool will be 3*#LUNS*maxBlockSize. The target amount of space in the pagepool for NSD workers is controlled by nsdBufSpace which defaults to 30%. So the pagepool should be large enough so that 30% of it has enough buffers.
For more information
To view the configuration parameters that has been changed from the default
mmlsconfig
To view the active value of any of these parameters you can run
mmfsadm dump config
To change any of these parameters use mmchconfig. For example to change the pagepool setting on all nodes.
mmchconfig pagepool=256M
1. Consideration to modify the PagePool
A. Sequential I/O
The default pagepool size may be sufficient for sequential IO workloads, however, a recommended value of 256MB is known to work well in many cases. To change the pagepool size
mmchconfig pagepool=256M [-i]
If the file system blocksize is larger than the default (256K), the pagepool size should be scaled accordingly. For example, if 1M blocksize is used, the default 64M pagepool should be increased by 4 times to 256M. This allows the same number of buffers to be cached.
B. Random I/O
The default pagepool size will likely not be sufficient for Random IO or workloads involving a large number of small files. In some cases allocating 4GB, 8GB or more memory can improve workload performance.
mmchconfig pagepool=4000M
C. Random Direct IO
For database applications that use Direct IO, the pagepool is not used for any user data. It's main purpose in this case is for system metadata and caching the indirect blocks of the database files.
D. NSD Server
Assuming no applications or Filesystem Manager services are running on the NSD servers, the pagepool is only used transiently by the NSD worker threads to gather data from client nodes and write the data to disk. The NSD server does not cache any of the data. Each NSD worker just needs one pagepool buffer per operation, and the buffer can be potentially as large as the largest filesystem blocksize that the disks belong to. With the default NSD configuration, there will be 3 NSD worker threads per LUN (nsdThreadsPerDisk) that the node services. So the amount of memory needed in the pagepool will be 3*#LUNS*maxBlockSize. The target amount of space in the pagepool for NSD workers is controlled by nsdBufSpace which defaults to 30%. So the pagepool should be large enough so that 30% of it has enough buffers.
For more information
Phoronix Test Suite
- 120+ Test Profiles
- 50+ Test Suites
- Extensible (XML-based) Testing Architecture
- Optional Linux-based LiveDVD/USB Testing Distribution (PTS Desktop Live)
- Automated Test Installation
- Dependency Management Support
- Module-based Plug-In Architecture
- Integrated Results Viewer
- PNG, JPG, Adobe SWF, SVG Graph Rendering Support
- Autonomous Batch Mode Support
- Global Database For Result Uploads, Benchmark Comparisons
- HTML Documentation Covering Test Profiles, Module Framework
- Installed Software, Hardware Detection
- System Monitoring Support
- GTK2 Graphical User Interface + Command-Line Interface
- Runs On Linux, OpenSolaris, Mac OS X, & BSD Operating Systems
NFS share on Linux client not immediately visible to other NFS clients
If you are using NFS as the shared file system, you may encounter this issue where NFS share on Linux client not immediately visible to other NFS clients. This is due to caching parameters which you must take note of on the NFS Client side. These are
- acregmin=n. The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. The default is 3 seconds.
- acregmax=n. The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. The default is 60.
- acdirmin=n. The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. The default is 60
- acdirmax=n. The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. The default is 60
- actimeo=n. When you wish to sets all of acregmin, acregmax, acdirmin, and acdirmax to the same value.
Speeding up boot time by Optimising Physical Memory and Swap
If you have a lot of physical memory, you may want to reduce your swappiness to increase performance. The performance is boosted as the kernel can cache data in memory for faster access and reduce the amount of data being swapped in and out of the swap space.
Alternatively, if you have very little memory (For example: 256MB or below), you may want to increase swappiness to boost performance.
To change the swappiness, You have to edit the /etc/sysctl.conf. /etc/sysctl.conf is a file tocontain sysctl which configure kernel parameters at runtime.
# vim /etc/sysctl.conf
At the bottom of the file /etc/sysctl.conf, add the following parameters
---------------------------------
vm.swappiness=5
---------------------------
Reboot and you will notice the difference. You may want to use conky to monitor the memory/swap usages
Alternatively, if you have very little memory (For example: 256MB or below), you may want to increase swappiness to boost performance.
To change the swappiness, You have to edit the /etc/sysctl.conf. /etc/sysctl.conf is a file tocontain sysctl which configure kernel parameters at runtime.
# vim /etc/sysctl.conf
At the bottom of the file /etc/sysctl.conf, add the following parameters
---------------------------------
vm.swappiness=5
---------------------------
Reboot and you will notice the difference. You may want to use conky to monitor the memory/swap usages
Speeding up boot time by running boot processes in parallel for Linux Mint
If you are using Linux Mint and have multi-core computer at home and wishes to speed up boot time by running processes in parallel, you can tweaked Grub to take advantage of parallelism.
Step 1: Edit /etc/init.d/rc and change CONCURRENCY from none to shell
# vim /etc/init.d/rc
---------------------------
CONCURRENCY=shell
---------------------------
You should notice a performance improvement in your boot-time.
Step 1: Edit /etc/init.d/rc and change CONCURRENCY from none to shell
# vim /etc/init.d/rc
---------------------------
CONCURRENCY=shell
---------------------------
You should notice a performance improvement in your boot-time.
Dstat a versatile resource statistics tool
Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.
Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).
Some of the features includes:
- Combines vmstat, iostat, ifstat, netstat information and more
- Shows stats in exactly the same timeframe
- Enable/order counters as they make most sense during analysis/troubleshooting
- Modular design
- Written in python so easily extendable for the task at hand
- Easy to extend, add your own counters (please contribute those)
- Includes about 10 external plugins to show how easy it is to add counters
- Can summarize grouped block/network devices and give total numbers
- Can show interrupts per device
- Very accurate timeframes, no timeshifts when system is stressed
- Shows exact units and limits conversion mistakes
- Indicate different units with different colors
- Show intermediate results when delay > 1
- Allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs
Make sure you have rpmforge repository installed.
# yum install dstat
Some of the useful commands are as followed:
# dstat --help
# dstat --full
# dstat --vmstat
# dstat --mem
# dstat --load
Remove the timeout during boot time
When you boot, there is a small count down before continuing with the boot process. This was originally introduced to ensure that older hardware load modules in time for the kernel to boot. Nowsaday, it is used for users to select their kernel modules, to dual boot etc.
However, if you have a modern machine and you wish to boot fast, you can edit
# /boot/grub/menu.lst
----------------------------------
timeout 0
----------------------------------
(Change timeout from 5 to 0)
You just save 5 seconds from your boot time
However, if you have a modern machine and you wish to boot fast, you can edit
# /boot/grub/menu.lst
----------------------------------
timeout 0
----------------------------------
(Change timeout from 5 to 0)
You just save 5 seconds from your boot time
Testing the Infiniband Interconnect Performance with Intel MPI Benchmark
This writeup focuses on verifying the performance of the Infiniband Interconnects or RDMA/iWARP Interconnects with Intel MPI Benchmark. For more information, do look at my Linux Cluster Blog
Linux Network Bonding or Trunking on CentOS 5.x
Finally, I have tested and setup a Linux Network Trunking. You will be very surprise it is very straightforward. See my blog "Linux Network Bonding or Trunking on CentOS 5.x"
Network design consideration for NFS
Network Design are an important consideration for NFS. Often we miss the infrastructure design consideration of NFS. Read further at Network design consideration for NFS
Tuning LINUX Toolbox
Finding Bottlenecks
Vmstat
Columns Information
(r,b) - How many process can be run if a CPU is available and how many are blocked.
(swpd, free, buff, cache) - Show how memory space is used.
(si, so) - Page-In and Page-Out
(io, bi, bo) - Number of Blocks received and sent to block devices
(in, cs) - Number of Interupts and context switches
(us, sy, id, wa) - Indicate percentage of time the CPU(s) has spent in userspace applications, in the kernel, being idle....
# vmstat 5 10 (To run vmstat with ten updates, five seconds )
Disk Performance
Hdparm is a good tool to determine whether the disks are healthy and configured
# hdparm -tT /dev/sda
Article to consider:
Vmstat
Columns Information
(r,b) - How many process can be run if a CPU is available and how many are blocked.
(swpd, free, buff, cache) - Show how memory space is used.
(si, so) - Page-In and Page-Out
(io, bi, bo) - Number of Blocks received and sent to block devices
(in, cs) - Number of Interupts and context switches
(us, sy, id, wa) - Indicate percentage of time the CPU(s) has spent in userspace applications, in the kernel, being idle....
# vmstat 5 10 (To run vmstat with ten updates, five seconds )
Articles to consider:
- Monitoring Virtual Memory with vmstat by LINUX Journal
- Understanding the Linux Virtual Memory Manager by Mel Gorman
- Is swap space obsolete? by Martin Pool
Disk Performance
Hdparm is a good tool to determine whether the disks are healthy and configured
# hdparm -tT /dev/sda
Article to consider:
- Speeding up Linux Using hdparm by O'Reilly
# iostat -x sda 1
(%iowait) - High means CPU is idele and waiting for outstanding disk I/O requests.
(avgqu-sz) - Value should be less than 1
(%util) - pecentage of time the disk has requests.
To install on CentOS, Fedora, "yum install sysstat", iostat will be installed with the packages.
Articles to consider:
Registry Hack : Load Application Faster
We start with a simple hack .Type "regedit" in the Run Command Box from Start Menu.This hack will load applications faster.The Windows prefetcher aims to load applications faster bt "prefetching" the application and storing it in the pre-fetch cache.You can speed up the application loading by changing the default value of one the settings under the relevant key.
Navigate to :
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters
The default value of the EnablePrefetcher setting is 3.Change this value to 5 to make application load faster.You can also experiment with higher values upto 9 and can see whether you get any improvement or not.
Navigate to :
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters
The default value of the EnablePrefetcher setting is 3.Change this value to 5 to make application load faster.You can also experiment with higher values upto 9 and can see whether you get any improvement or not.
Subscribe to:
Posts (Atom)