Transparent SQUID Proxy

Here I will show you how to configure SQUID as a transparent proxy server, which means that once the SQUID server is running, you don’t have to configure proxy server IP address on your individual browsers on your client systems. This is something that i forgot to mention in my previous two posts on SQUID – SQUID (Caching Proxy Server) Logs and SQUID ACLs

We all use more than one browser on our system nowadays and with the typical setup, you have to configure SQUID server IP on individual browsers so that all the request pass through the SQUID proxy server. A better way is to configure SQUID server to act as a transparent proxy server and update the client machine’s gateway to use SQUID server’s IP address. That way you don’t have to update proxy server’s IP address on all the browsers that you use on your system.

This is a 3 step process once SQUID is configured:

1. Edit http_port 3128 as below:

http_port 192.168.2.50:3128 transparent

2. Now update firewall rules so all the requests on port 80 gets redirected to port 3128 as follows:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128

3. Now, set the gateway on all your client machine to the IP address of your SQUID proxy server.

That’s it. Now, all requests on your client machines will be routed via SQUID server no matter how many different browsers you use.

Pound – A Reverse Proxy Load Balancer

Here is a simple way on how you can set up a muti-node Apache Web server cluster that provides load balancing. In front of the apache cluster I have created a load balancer that splits up the incoming HTTP/HTTPS requests between the multiple Apache nodes. The advantage of using a load balancer is that it takes care of the load on the web server nodes and tries to direct requests to the node with less load and it also takes care of the sessions.

For our setup we configure a program called Pound and behind this pound server, we have two apache webserver nodes.

Pound is a reverse-proxy server. This means that it passes requests from client browsers to one or more back-end servers. Also, Pound acts as a load balancer as it distributes the requests from the client browsers among serveral back-end servers, while keeping session information. Pound is aso an SSL wrapper that can decrypt HTTPS requests from client browsers and pass them as plain HTTP to the back-end servers. Pound can verify requests for correctness and accept only well formed ones. Pound can detect when the backend server fails or is recovered and take actions accordingly. So, it’s also a failover server. We can set priority for preferred back end server in case a particular server has a better hardware to accomodate the heavy load. If priority is not set, it takes the default priority as 5 and the priority ranges from 1 to 9; 9 being the highest priority. Remember that Pound is NOT a Web Server or Web Accelerator. So, no websites are cached.

You can download Pound for REDHAT from the below URL:

http://www.invoca.ch/pub/packages/pound/RPMS/ils-5/SRPMS/

Once installed, you can open the configuration file and update as required:

vi /etc/pound.cfg

User "pound"
Group "pound"
Control "/var/lib/pound/pound.cfg"

ListenHTTP
	Address 192.168.2.50
	Port 80
	Service
	   BackEnd
		Address 192.168.2.10
		Port 80
	   End
	   BackEnd
		Address 192.168.2.20
		Port 80
		Priority 6
	   End
	   Session
  	   	Type IP
		TTL 300
  	   End
	End
End

:wq!

#service pound start
#chkconfig pound on

This tells Pound to listen on the real IP address 192.168.2.50 @ port 80 and proxy all requests to the back-end servers listed. Failover is taken care of automatically and so there’s nothing to configure to accomplish this important feature. The second server has a priority of 6, which means that more requests will be routed to server with IP address 192.168.2.20

Also, Pound is a session-aware server, meaning it keeps track of sessions between a client browser and a back-end server. So, once a session is established, all subsequent requests from the same browser will be directed to the same back-end server. In the above configuration, you can see “Session” configured by client IP address. In this scheme Pound directs all requests from the same client IP address to the same back-end server within a five-minute interval. Though Pound is session-aware, accuracy is NOT guaranteed as HTTP is defined as a stateless protocol. There are five other ways to configure sessions using Pound, but none can guarantee accuracy. The above method of configuring session via IP address is the easiest one, though we have other methods like Basic Authentication, URL parameter, cookie value, HTTP parameter and header value.

Note:

SSL certificates can also be configured if you add the following to your pound.cfg configuration file.

ListenHTTPS
	Address xxx.xxx.x.xx
	Port 443
	Cert "/etc/pki/tls/certs/pound.pem"
End

That’s it.

Managing Storage with LVM

LVM or Logical Volume Manager is a key tool for system administrators managing storage on Linux servers. LVM introduces three layers of abstraction b/n the underlying storage devices on a Linux server and the file system which is accessed by the users.

This example has 3 hard drive devices – sdc1, sdd1 and sde. We will use LVM to aggregate these 3 hard drives to provide storage, extend 3 formatted file system and mount it under a directory called ‘/lvmdata’. This additional abstraction between the underlying storage devices and the user accessible file system enables us to manage storage more effectively and with greater flexibility.

Let’s discuss each LVM layer in sequence and take a look at some specific command used to configure each layer.

The 1st layer is know as LVM Physical Volumes (PV’s). A storage device must be formatted as a Physical Volume (PV) before it can participate in the LVM infrastructure that we will be building.

pvcreate /dev/sdc1 /dev/sdd1

You can check the newly created PV using the command ‘pvdisplay’. This command shows the 2 physcal volumes (sdc1 and sdd1) that we used right now.

Next we need to aggregate the PV into a single contiguous pool of storage known as Volume Group (VG).

vgcreate exampeVG /dev/sdc1 /dev/sdd1

You can check the newly created VG using ‘vgdisplay’ command. VGdisplay tells you the volume group name, how much size the volume has and so on.

Finally, we need to carve out a section of VG that we can then place our file system on. This carved out piece of storage is known as Logical Volume (LV).

lvcreate -n exampleLV -L 1G exampleVG

You can check the newly created LV using the command ‘lvdisplay’.

Now, create a file system, create a mount point (/lvmdata/) and then finally mount the new LVM file system back to the mount point.

mkfs.ext3 /dev/exampleVG/exampleLV

mkdir /lvmdata

mount /dev/exampleVG/exampleLV /lvmdata/

We can now expand our /lvmdata filesystem in real time while all the data remains online and fully accessible.

You can look the current storage utilization using df command:

df -h

You can see the new LVM (/lvmdata/) partition which is of 1 gig in size.

Now to extend the /lvmdata partition, you can use the command ‘lvextend’ as follows:

lvextend -L +500M /dev/exampleVG/exampleLV

Here we are adding additional 500 megs of space to the existing exampleLV logical volume. Note the output of df command now. You will see the additional 500 megs is not reflected on the /lvmdata although we have already added 500 megs to it. This is because we have extended the underlying logical volume, but not the file system itself. The command ‘resize2fs’ will do the trick.

resize2fs /dev/exampleVG/exampleLV

Now, check the ‘df -h’ command and see the difference. You will see the additional 500 megs added to the /lvmdata/ partition.

Remember, all this was done without unmounting the filesystem and all the data remained intact and accessible to users.

Migrating data to a SAN storage device:

Imagine you want to migrate all your data from your servers local hard drive to a SAN (Storage Area Network).

In our case the SAN device is /dev/sde

So, our job is to migrate all of our data from local storage (sdc1 and sdd1) to our SAN device (sde)

Using LVM you can easily accomplish this task. First we need to initialize our SAN storage using pvcreate.

pvcreate /dev/sde

Then we can extend our existing Volume Group (VG) on to our new SAN device using ‘vgextend’.

vgextend exampleVG /dev/sde

Now we need to migrate all the data from our local physical volume to our SAN PVs using the command ‘pvmove’

pvmove /dev/sdc1 /dev/sde
pvmove /dev/sdd1 /dev/sde

Finally, we can remove the local storage devices from our VG using the command ‘vgreduce’

vgreduce exampleVG /dev/sdc1 /dev/sdd1

Again, this was done in real time when the file system data remained online and fully accessible to the users.

mknod to your rescue to restore /dev/null

Recently one of my friend unknowingly deleted /dev/null file and it was throwing error when he tried to launch the terminal console. So, what exactly is the file /dev/null ?

/dev/null is a character special file, which means that it is a contentless file that is marked as being able to source and sink byte streams to it. What happens to those streams is determined by their device numbers. Character special files are like most other files in many respects, and can be deleted if you have the permissions. But, in order to re-create this file, you should know the major and minor number associated with this character file.

[root@ldapserver /]# ll /dev/null
crw-rw-rw-. 1 root root 1, 3 Dec 18 2011 /dev/null

In my server, I have major-minor numbers for /dev/null as 1,3. But, what if somebody has already deleted this file, and you want to know the major and minor numbers for /dev/null so you can recreate this file.

If you have installed linux from source, then you should have devices.txt file on the following path

/usr/src/linux-3.1.6/Documentation/devices.txt

Open this file ‘devices.txt’ in any text editor and search for /dev/null. You will reach the following section:

  1 char        Memory devices
                  1 = /dev/mem          Physical memory access
                  2 = /dev/kmem         Kernel virtual memory access
                  3 = /dev/null         Null device
                  4 = /dev/port         I/O port access

From the above, you can see the 1 and 3 highlighted in yellow. ’1′ is the major number and ’3′ is the minor number. The major number specifies the major device type (for example, the device driver number). The minor number specifies the device location, which is typically, but not always, the unit, drive, and/or line number.

Now that you know the major and minor numbers associated with /dev/null, you can re-create this file as follows:

mknod -m 666 /dev/null c 1 3

mknod is used to create the character and block devices that populates ‘/dev/’. So, the above command is used to create a character (c) device file with major and minor pair of 1,3 and world writable.

Enjoy creating character and block device files.

Create an iso image of a folder

We all know how to create an iso image of a DVD using ‘dd’ command. But, what if you want to create an iso image of a folder containing lot of files?

Here the command mkisofs comes to our help. The usage is as follows:

mkisofs -o /{dest_dir}/{dest_file.iso} /{absolute path to the actual folder that you want to create the image}

eg: mkisoft -o /mnt/mycontent.iso /opt/rhel/media/

In the above example, an image ‘mycontent.iso’ will be created under ‘/mnt’ partition for the folder ‘media’. The option ‘-o’ means the output file.

Inode and Inumber

An inode is a data structure that contains important information regarding file within a file system. Whenever file system is created in UNIX, a set amount of inode is created. Usually, about 1% of the total file system disk space is allocated to the inode table.

Inode and Inumber (or Inode Number):

Inode and Inumber seems to be same, but they are not. Inode refers to the data structure and the inumber refers to the identification number of the inode. So, inumber is also referred to as inode number. The inode table contains a listing of all inode numbers for the respective file system. When users search for or access a file, the UNIX system searches through the inode table for the correct inode number. When the inode number is found, the command in question can access the inode and make the appropriate changes if applicable.

Take, for example, editing a file with vi. When you type vi , the inode number is found in the inode table, allowing you to open the inode. Some attributes are changed during the edit session of vi, and when you have finished and typed :wq, the inode is closed and released. This way, if two users were to try to edit the same file, the inode would already have been assigned to another user ID (UID) in the edit session, and the second editor would have to wait for the inode to be released.

How to check the inode number for any file?

[soj@centos ~]$ ls -i firewall.txt
458778 firewall.txt

[soj@centos ~]$ stat firewall.txt
File: `firewall.txt’
Size: 426 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 458778 Links: 1
Access: (0644/-rw-r–r–) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-01-23 08:36:57.000000000 -0800
Modify: 2012-01-23 08:36:52.000000000 -0800
Change: 2012-01-23 08:36:52.000000000 -0800

The inode contains all the information about a file except it’s actual file name and the file contents.
The following definitions provide just some of the important information contained in the inode that UNIX users employ constantly:

Inode number
Mode information to discern file type and also for the stat C function
Number of links to the file
UID of the owner
Group ID (GID) of the owner
Size of the file
Actual number of blocks that the file uses
Time last modified
Time last accessed
Time last changed

The information listed above is important to files and is used heavily in UNIX. Without this information, a file would appear corrupt and unusable. In UNIX, directories are actually files that have a few additional settings in their inodes. A directory is basically a file containing other files. Also, the mode information has flags set to inform the system that the file is actually a directory.

Split huge files to small chunks

What if you have a huge file and want to store this file on a CD/DVD; say a file of size 5GB and you want to store this on a DVD or multiple DVD’s. Linux comes with a built in command ‘split’ which can be used to split any kind of huge file into small size chunks. This is achieved as follows:

I am creating a huge file using ‘seq’ command:

seq 1 10000000 myseq_file.txt

Now I am going to split this huge file into chunks:

split –byte=700m myseq_file.txt seq_small_

The underscore (_) at the end is mandatory as the file will be split as seq_small_aa, seq_small_ab etc. The output of the above command is as follows:

[soj@centos ~]$ ll seq*
-rw-r–r–. 1 soj root 734003200 Feb 5 16:24 seq_small_aa
-rw-r–r–. 1 soj root 734003200 Feb 5 16:25 seq_small_ab
-rw-r–r–. 1 soj root 734003200 Feb 5 16:25 seq_small_ac
-rw-r–r–. 1 soj root 734003200 Feb 5 16:26 seq_small_ad
-rw-r–r–. 1 soj root 734003200 Feb 5 16:27 seq_small_ae
-rw-r–r–. 1 soj root 653045760 Feb 5 16:27 seq_small_af

Now, you can use the command ‘cat’ to restore the chunks back to original size so you can use it.

[soj@centos ~]$ cat seq_small_a* > seq_restore.txt

Split command can be used to split video, mp3 or any kind of huge files.

Have fun!!!

Simple way to Password protect and Encrypt a file

Here is a simple way to password protect a file using gpg encryption. This is not the proper way to encrypt file using gpg, but this is the easiest method to do it and I use it often.

I have a file named dmesg.txt
-rw-rw-r–. 1 soj soj 63311 Dec 11 10:03 dmesg.txt

Now, I am going to encrypt with a one liner gpg command:

gpg -c dmesg.txt

The above command will prompt you for a new passphrase and once the password is provided, it will encrypt the file dmesg.txt. Now, you will have a new encrypted file named dmesg.txt.gpg. If you try to display this file using ‘cat’ command, you will see it has got garbage characters which means it’s encrypted. But, there is another better way to encrypt a file and displaying in ASCII character. It’s neat and tidy and this can be achieved with the same command, but with an extra option ‘-a’ (armor) as follows:

gpg -c -a dmesg.txt

Now, we will have an encrypted file ‘dmesg.txt.asc’ which is more tidier when displayed. You can now share this file with your friend or intended recipient and the friend can open it using the same password that you provided at the time of encryption. If you are on a GUI, you just need to double click the encrypted file which will prompt you for the password, but if you are on a shell, you just need to issue the following command which will prompt you for a password at the shell:

gpg dmesg.txt.asc

Using the above method, you can backup a folder and encrypt at the same time.

tar -cvzf dmsg.tar.gz dmsg
The above command will create a tar file dmsg.tar.gz. Now, we can encrypt using gpg/armor command as follows:
gpg -c -a dmsg.tar.gz
An encrypted armored file is created as dmsg.tar.gz.asc

If you double click the file, it will prompt for that password that we have provided when we created the gpg encrypted file.

Simple IPTABLES rules

List all iptables rules to display rules with line numbers

iptables -L --line-numbers

IPTABLES rule to allow all INCOMING SSH request on eth0 interface:

iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT

IPTABLES rule to allow all OUTGOING SSH request on eth0 interface:

iptables -A OUTPUT -o eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
{Here we allow NEW & ESTABLISHED state on the OUTPUT chain and only ESTABLISHED state on the INPUT chain}

Multiport usage to allow more than one ports via single rule:

iptables -A INPUT -i eth0 -p tcp -m multiport --dport 22,80,443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp -m multiport --sport 22,80,443 -m state --state ESTABLISHED -j ACCEPT

Allow PING from External Network to Internal Network:

iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT

Allow PING from Internal Network to External Network:

iptables -A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT

Allow Loopback Access:

iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

Allow Internal Network to External Network

eth0 => Internal Network
eth1 => External Network (Internet)

iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT

IPTABLES – Port Forwarding (Routes all traffic that comes to port 422 to port 22)

iptables -t nat -A PREROUTING -p tcp -d 192.168.10.14 --dport 422 -j DNAT --to 192.168.10.14:22
Now, you have to explicitly allow incoming connection on port 422 as follows:
iptables -A INPUT -i eth0 -p tcp --dport 422 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --sport 422 -m state --state ESTABLISHED -j ACCEPT

If condition – Shell script options

The following are the various options for checking the existence of a file via shell script

[ -a FILE ]	True if FILE exists.
[ -b FILE ]	True if FILE exists and is a block-special file.
[ -c FILE ]	True if FILE exists and is a character-special file.
[ -d FILE ]	True if FILE exists and is a directory.
[ -e FILE ]	True if FILE exists.
[ -f FILE ]	True if FILE exists and is a regular file.
[ -g FILE ]	True if FILE exists and its SGID bit is set.
[ -h FILE ]	True if FILE exists and is a symbolic link.
[ -k FILE ]	True if FILE exists and its sticky bit is set.
[ -p FILE ]	True if FILE exists and is a named pipe (FIFO).
[ -r FILE ]	True if FILE exists and is readable.
[ -s FILE ]	True if FILE exists and has a size greater than zero.
[ -t FD ]	True if file descriptor FD is open and refers to a terminal.
[ -u FILE ]	True if FILE exists and its SUID (set user ID) bit is set.
[ -w FILE ]	True if FILE exists and is writable.
[ -x FILE ]	True if FILE exists and is executable.
[ -O FILE ]	True if FILE exists and is owned by the effective user ID.
[ -G FILE ]	True if FILE exists and is owned by the effective group ID.
[ -L FILE ]	True if FILE exists and is a symbolic link.
[ -N FILE ]	True if FILE exists and has been modified since it was last read.
[ -S FILE ]	True if FILE exists and is a socket.
[ FILE1 -nt FILE2 ]	True if FILE1 has been changed more recently than FILE2, or if FILE1 exists and FILE2 does not.
[ FILE1 -ot FILE2 ]	True if FILE1 is older than FILE2, or is FILE2 exists and FILE1 does not.
[ FILE1 -ef FILE2 ]	True if FILE1 and FILE2 refer to the same device and inode numbers.
[ -o OPTIONNAME ]	True if shell option "OPTIONNAME" is enabled.
[ -z STRING ]	True of the length if "STRING" is zero.
[ -n STRING ] or [ STRING ]	True if the length of "STRING" is non-zero.
[ STRING1 == STRING2 ]	True if the strings are equal. "=" may be used instead of "==" for strict POSIX compliance.
[ STRING1 != STRING2 ]	True if the strings are not equal.
[ STRING1  STRING2 ]	True if "STRING1" sorts after "STRING2" lexicographically in the current locale.
[ ARG1 OP ARG2 ]	"OP" is one of -eq, -ne, -lt, -le, -gt or -ge. These arithmetic binary operators return true if "ARG1" is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to "ARG2", respectively. "ARG1" and "ARG2" are integers.
Expressions may be combined using the following operators, listed in decreasing order of precedence:

Combining expressions

Operation	Effect
[ ! EXPR ]	True if EXPR is false.
[ ( EXPR ) ]	Returns the value of EXPR. This may be used to override the normal precedence of operators.
[ EXPR1 -a EXPR2 ]	True if both EXPR1 and EXPR2 are true.
[ EXPR1 -o EXPR2 ]	True if either EXPR1 or EXPR2 is true.