#5. head/tail

Most of time, the sole purpose of logging in a server is to diagnose some issue and the common way to start this is to look at logs. Logs of different applications like Apache, MySQL, mail logs etc. What you use to look at logs? isn’t that tail? similarly we sometimes use ‘head’ to check few starting lines of any file.

Few examples:

* Continuously check Apache error log file:

$ tail -f /var/log/httpd/error_log

* View first 15 linues from MySQL log:

$ head -15 /var/log/mysqld.log

#4. vi/nano/emacs

Text editor basically needed frequently to create/update config files. I prefer vim, simply because I am very comfortable with it and remembers some of its useful commands for quick editing.

few example of working with vi. open a file with vi and without going in insert mode, here are useful character you can press:

=> jump to end of line
 $
=> start of line
 0
=> Delete rest of line
 D
=> Repeat the last command given: 
. (dot)
 
=> add 'maal' to the end of every line. 1 is line 1, $ is the last line
:1,$ s/$/maal/
 
=> put 'bingo' at the start of lines 5-10
:5,10 s/^/bingo/
 
=> change foo to bar for all occurrences in the rest of the file from where the cursor is
:s/foo/bar/g
 
=> Delete current line and got into insert mode.
C
 
=> Remove the ^M from files that came from windows:
:se ff=unix
 
=> Turn on/off display of line numbers: 
 :set nu
 :set nonu
 
=> if you want actual line numbers in your file:
:%!cat -n
 
=> find the word under cursor
 * (star)

#3. screen

screen is one of much underutilized command in nix world. take a scenario, when last time you issued a command in remote server and find out that the command will take hours to complete? or you are in need to login in 10 servers and check something.. copy files among them.. and voila.. your internet connection get reset and your ssh session get terminated. Here comes screen, once you start using it, you will get hooked to it. Screen is a terminal multiplexer that allows you to manage many processes (like ssh sessions) through one physical terminal. Each process gets its own virtual window, and you can bounce between virtual windows interacting with each process.

Let me give you more insight. Suppose you have many servers and ideally you should restrict ssh (port 22) access to selected IPs only. So, you login into one server which allows access from remote IPs. You can start screen there by typing ‘screen’ (all major Linux distributions have screen already installed). You can see a status bar. create new screen windows by pressing Ctrl+ac. switch between them by pressing Ctrl+an (next) and Ctrl+ap (previous). Basically, for b It offers very useful features like Remote terminal session management (detaching or sharing terminal sessions), unlimited windows (unlike the hardcoded number of Linux virtual consoles), copy/paste between windows, notification of either activity or inactivity in a window, split terminal (horizontally and vertically) into multiple regions, sharing terminals etc.

You can save your preferences in .screenrc, like here’s my .screenrc where I’ve redefining status bar look and feel and assigning key f5 (previous window) and f6 (next window):

$ cat ~/.screenrc 
# no annoying audible bell, please
vbell on
 
# detach on hangup
autodetach on
 
# don't display the copyright page
startup_message off
 
# emulate .logout message
pow_detach_msg "Screen session of \$LOGNAME \$:cr:\$:nl:ended."
 
# advertise hardstatus support to $TERMCAP
termcapinfo xterm* ti@:te@
 
# make the shell in every window a login shell
shell -$SHELL
 
defscrollback 10000
 
# Extend the vt100 desciption by some sequences.
 
termcap vt* AF=\E[3%dm:AB=\E[4%dm 
caption always
caption string '%{= wk}[ %{k}%H %{k}][%= %{= wk}%?%-Lw%?%{r}(%{r}%n*%f%t%?(%u)%?%{r})%{k}%?%+Lw%?%?%= %{k}][%{b} %d/%m %{k}%c %{k}]' 
 
 
# keybindings
 
bind -k F5 prev 
bind -k F6 next

#2. netstat/nmap

These are very useful commands to diagnose things about network. of course, ping/traceroute may be most commonly used ones but the usefulness wise, nmap and netstat are more useful than a basic ping. netstat stands for network status. nmap is a sort of security/port scanner or you can say a network exploration command.

few examples of netstat:
* Display total number of internet (port 80) connections:

$ netstat -an |grep :80 |wc -l

* Display all ports your machine listening on:

$ netstat -ant | grep LISTEN

* Scan a machine on your LAN with nmap and know which ports are open on it:

$ nmap ip

#1. find and grep
List of some routine tasks: How many files are there consuming most of disk space? Delete all temporary files older than 2 days, find out how many files have old server name written in them which is causing issue? rename all ‘.list’ to ‘.txt’. The commands find, grep are your best friend here.

Find command is used to search for files. you can specify many options with it like files created today or having size greater then you specified. Normally we also combine find with xargs or exec to issue commands on files returned by find.

examples of find command:
* find top 10 largest files in /var:

$ find /var -type f -ls | sort -k 7 -r -n | head -10

* find all files having size more than 5 GB in /var/log/:

$ find /var/log/ -type f -size +5120M -exec ls -lh {} \;

* find all today’s files and copy them to another directory:

$ find /home/me/files -ctime 0  -print -exec cp {} /mnt/backup/{} \;

* find all temp files older than a week and delete:

$ find /temp/ -mtime +7-type f | xargs /bin/rm -f

* find and rename all mp3 files by changing their uppercase names to lowercase:

$ find /home/me/music/ -type f -name *.mp3 -exec rename 'y/[A-Z]/[a-z]/' '{}' \;

some examples of grep command:

* Print Apache’s documentroot directory name:

$ grep -i documentroot  /etc/httpd/conf/httpd.conf

* View file contents without comments and empty lines:

$ grep -Ev “^$|^#” /etc/my.cnf

* print only IP address assigned to the interface:

$ ifconfig eth0 | grep 'inet addr:' | cut -d':' -f2 | awk '{ print $1}'

* How many email messages sent for a particular date:

$ cat /var/log/maillog | grep "status=sent" | grep "May 25" | wc -l

* Find out a running process/daemon from process list (thanks to staranneph for recalling this):

ps -ef | grep mysql

* You can also note cpu/mem usage by using above. like in below command output, you can see that Plesk’s statistics process is utilizing more than 18% cpu alone:

[root@myserver ~]# ps aux | grep statistics
root      8183 18.4  0.0  58384  2848 ?        D    04:05   3:00 /usr/local/psa/admin/sbin/statistics

I would like to know your thoughts, any command / tool you think should be included in top 5 here.

1. Download, compile and install MySQL sandbox:

 cd /usr/src
wget http://edge.launchpad.net/mysql-sandbox/mysql-sandbox-3/mysql-sandbox-3/+download/MySQL-Sandbox-3.0.05.tar.gz
tar xzf MySQL-Sandbox-3.0.05.tar.gz
cd MySQL-Sandbox-3.0.05
perl Makefile.PL
make
make test
make install

2. Get the latest binary tarball for MySQL server:

cd /usr/src
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.41-linux-x86_64-icc-glibc23.tar.gz/from/http://mirror.csclub.uwaterloo.ca/mysql/

Here, I’ve downloaded MySQL 5.1.41 server tarball. Please note that one MySQL was already there on server having different version (ver 5.0.77). For second installation, I’ve downloaded different version to test out things but for best compatibility, its recommended to download same version.

3. Create new sandbox for single MySQL server. Pls note that you should run following command from a non-root user. Though, in my case, I went ahead and wanted to run command by root user and for that, we must set an environment variable to > 0 value.

export SANDBOX_AS_ROOT=1     ## no need if your are executing below command from non-root user
make_sandbox  /usr/src/mysql-5.1.41-linux-x86_64-icc-glibc23.tar.gz

4. Sandbox should be created without any glitch. you can go inside the config directory and check that out:

cd /root/sandboxes/msb_5_1_41

Pls replace root here with your normal username in case you created sandbox with non-root user. Also note down that sandbox created new directory using 5_1_41 naming based on your MySQL server version. This is the default behaviour and we can change it via configuration of sandbox. Also note down that the new MySQL server will listen on same port number ie. 5141.

5. Start and use the service of newly installed MySQL server:

./start   ## make sure you are in /root/sandboxes/msb_5_1_41 directory

server should be started. now try to login in MySQL:

./use

You should be in MySQL prompt. You can update these scripts (use, start, stop etc.) as per your taste. The default configuration file for MySQL will be in this directory having name as my.sandbox.cnf. You can update this MySQL config file as per your requirements.

Also dont forget to update root password and if you ever encounter situation where you forgot the password, refer these instructions to quickly reset MySQL root password.

Try to connect this MySQl service using your script after making sure that you must specify port 5141 (or whatever you mentioned in configs) while connecting because this service is not listening to default MySQL port 3306.

If you want to use PhpMyAdmin to manage these multiple MySQL services, Here’s post that will help you configuring PhpMyAdmin.

Other useful posts on MySQL in this blog:

* Setup MySQL Cluster in Redhat based Linux.
* Optimize a large installation of MySQL.
* Quickly repair a huge corrupted or crashed table.
* How to ignore MySQL errors while executing bulk statements
* Fix MySQL running slow without any know reason.

You can use any preferred time server to sync. I’m using public time servers here:

$ ntpdate pool.ntp.org
5 Feb 11:17:00 ntpdate[3332]: step time server 220.225.252.202 offset -585.969659 sec

Network time update command (ntpdate) will contact the available time server from public pool and sync time with it. Here in above example, offset is denoting that the clock was running ahead of time in my server.

To remain updated, you should put a cron entry to sync with time server every midnight:

$ crontab -e
0 4 * * * /usr/sbin/ntpdate pool.ntp.org

This will update/correct your time in server at 04:00 am every morning. I’ve written a very quick trick to correct your time. If you are interested to study further, you can find detailed information here.

1. Check MySQL Configurations:

my.cnf: My Server machine has 2 GB of RAM, So, I’ve optimized my.cnf to make use of plenty of RAM. But that was not the issue because eventually no one was accessing this Server except some perl scripts residing on another Server in LAN. Its a good practice though, to replace your /etc/my.cnf with my-huge.cnf in case your machine have 2 or more GB of RAM and fast multi-core processor. You can find my-huge.cnf in /usr/share/doc/MySQL-server-community-5.0.51a/ directory if you have installed latest MySQL version (5.0.51a while writing this article).

DNS Resolution: I was accessing MySQL server from remote machine using its IP Address, so DNS should not be the issue. Check DNS stuff, if you are trying hostname instead of IP. For optimum response, always access db server with its IP. To do that, simply add “skip-name-resolve” keyword in mysqld section of your my.cnf.

Bind to IP: Its better to bind Server to a particular IP Address. you can use something like this bind-address=192.168.0.5 in your my.cnf

Turn on Logging: Turn on slow queries logging and find out queries running too slow. This was also not the issue at my side, but its better to have a look at it. You can use log-slow-queries = /var/log/mysql-slow-query.log statement in my.cnf under mysqld section to enable logging.

Note: You need to restart your MySQL Server everytime after updating config file. As we are investigating slow response issue, I assume you already have user’s and proper rights to access the Server from remote site.

2. System State: After checking the MySQL settings, you need to verify various parameters in the Server machine, some important points you should check:
Load: Check the current load by using top or cat /proc/loadavg commands.
Disk Space: Check disk space using df -h command and verify there’s some free space for proper working of Server. You can also check excessive I/O requests using iostate command.

3. Network: Time to check network settings. Check port 3306 (MySQL default port) is open. To check whether this port is open and accessible from remote, issue “telnet <mysql_server_IP> 3306″ command, it should get connected otherwise check your network/firewall settings.

So, you have checked everything and found that MySQL response is still slow even there’s no load, no disk space or network issue, accessing with IP Address so no DNS issue, etc. etc. and etc.

At the last resort, I checked hostname of my Server and amazed that by fixing hostname properly, My problem get resolved. How? check it:

$ hostname
testservre.example.com
$ cat /etc/hosts
127.0.0.1	testserver testserver.example.com localhost.localdomain localhost

You can see that there’s slight spelling mistake in hostname, displayed by hostname command and that written in hosts file. Just fix your hosts file and check it. You should note down that, even after using IP Address to access MySQL Server, its still important to set proper DNS Hostname for your Server as its was verified when authenticating remote clients.

You may also like to read:
* Quick and simple php based script to check replication status of multiple MySQL Servers
* Setup multiple MySQL database servers in a single Linux host
* Optimize MySQL on a large Database Server
* Recover or reset root password of MySQL and PostgreSQL Servers
* How to setup MySQL Cluster 7.0 in Redhat based Linux
* How to find out the clients connecting to your MySQL server
* Quickly repair huge corrupted or crashed table in MySQL
* Install and configure PhpMyAdmin to manage multiple MySQL Servers

# perl -MCPAN -e 'install Net::SSH::Perl'

It got stuck at:

t/01-compile.t…..ok
t/02-buffer.t……ok
t/03-packet.t……1/10

then I tried installation with force so that it can skip tests if any of them failed:

# perl -MCPAN -e 'force install Net::SSH::Perl'

The module seems installed ok but while using in a script, error occured:
Can’t locate Net/SSH/Perl/Buffer.pm in @INC …

that means, the module still not installed/configured properly. To get it done, I searched for rpms and found rpmforge repository http://dag.wieers.com/rpm/packages/rpmforge-release/ have all the sufficient rpms:

# rpm -ivh /usr/src/rpmforge-release-0.3.4-1.el4.rf.i386.rpm
# yum install perl-Net-SSH-Perl --enablerepo=rpmforge

and its done.