Tuesday, December 16, 2008

All about Linux swap space

When your computer needs to run programs that are bigger than your available physical memory, most modern operating systems use a technique called swapping, in which chunks of memory are temporarily stored on the hard disk while other data is moved into physical memory space. Here are some techniques that may help you better manage swapping on Linux systems and get the best performance from the Linux swapping subsystem.

Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.

Swapping is necessary for two important reasons. First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately. Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

However, swapping does have a downside. Compared to memory, disks are very slow. Memory speeds can be measured in nanoseconds, while disks are measured in milliseconds, so accessing the disk can be tens of thousands times slower than accessing physical memory. The more swapping that occurs, the slower your system will be. Sometimes excessive swapping or thrashing occurs where a page is swapped out and then very soon swapped in and then swapped out again and so on. In such situations the system is struggling to find free memory and keep applications running at the same time. In this case only adding more RAM will help.

Linux has two forms of swap space: the swap partition and the swap file. The swap partition is an independent section of the hard disk used solely for swapping; no other files can reside there. The swap file is a special file in the filesystem that resides amongst your system and data files.

To see what swap space you have, use the command swapon -s. The output will look something like this:

Filename        Type            Size    Used    Priority
/dev/sda5 partition 859436 0 -1

Each line lists a separate swap space being used by the system. Here, the 'Type' field indicates that this swap space is a partition rather than a file, and from 'Filename' we see that it is on the disk sda5. The 'Size' is listed in kilobytes, and the 'Used' field tells us how many kilobytes of swap space has been used (in this case none). 'Priority' tells Linux which swap space to use first. One great thing about the Linux swapping subsystem is that if you mount two (or more) swap spaces (preferably on two different devices) with the same priority, Linux will interleave its swapping activity between them, which can greatly increase swapping performance.

To add an extra swap partition to your system, you first need to prepare it. Step one is to ensure that the partition is marked as a swap partition and step two is to make the swap filesystem. To check that the partition is marked for swap, run as root:

fdisk -l /dev/hdb

Replace /dev/hdb with the device of the hard disk on your system with the swap partition on it. You should see output that looks like this:

 Device Boot    Start   End     Blocks  Id      System
/dev/hdb1 2328 2434 859446 82 Linux swap / Solaris

If the partition isn't marked as swap you will need to alter it by running fdisk and using the 't' menu option. Be careful when working with partitions -- you don't want to delete important partitions by mistake or change the id of your system partition to swap by mistake. All data on a swap partition will be lost, so double-check every change you make. Also note that Solaris uses the same ID as Linux swap space for its partitions, so be careful not to kill your Solaris partitions by mistake.

Once a partition is marked as swap, you need to prepare it using the mkswap (make swap) command as root:

mkswap /dev/hdb1

If you see no errors, your swap space is ready to use. To activate it immediately, type:

swapon /dev/hdb1

You can verify that it is being used by running swapon -s. To mount the swap space automatically at boot time, you must add an entry to the /etc/fstab file, which contains a list of filesystems and swap spaces that need to be mounted at boot up. The format of each line is:

                           

Since swap space is a special type of filesystem, many of these parameters aren't applicable. For swap space, add:

/dev/hdb1       none    swap    sw      0       0

where /dev/hdb1 is the swap partition. It doesn't have a specific mount point, hence none. It is of type swap with options of sw, and the last two parameters aren't used so they are entered as 0.

To check that your swap space is being automatically mounted without having to reboot, you can run the swapoff -a command (which turns off all swap spaces) and then swapon -a (which mounts all swap spaces listed in the /etc/fstab file) and then check it with swapon -s.

Swap file

As well as the swap partition, Linux also supports a swap file that you can create, prepare, and mount in a fashion similar to that of a swap partition. The advantage of swap files is that you don't need to find an empty partition or repartition a disk to add additional swap space.

To create a swap file, use the dd command to create an empty file. To create a 1GB file, type:

dd if=/dev/zero of=/swapfile bs=1024 count=1048576

/swapfile is the name of the swap file, and the count of 1048576 is the size in kilobytes (i.e. 1GB).

Prepare the swap file using mkswap just as you would a partition, but this time use the name of the swap file:

mkswap /swapfile

And similarly, mount it using the swapon command: swapon /swapfile.

The /etc/fstab entry for a swap file would look like this:

/swapfile       none    swap    sw      0       0

How big should my swap space be?

It is possible to run a Linux system without a swap space, and the system will run well if you have a large amount of memory -- but if you run out of physical memory then the system will crash, as it has nothing else it can do, so it is advisable to have a swap space, especially since disk space is relatively cheap.

The key question is how much? Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.

The Linux 2.6 kernel added a new kernel parameter called swappiness to let administrators tweak the way Linux swaps. It is a number from 0 to 100. In essence, higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle. Kernel maintainer Andrew Morton has said that he runs his desktop machines with a swappiness of 100, stating that "My point is that decreasing the tendency of the kernel to swap stuff out is wrong. You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful."

One downside to Morton's idea is that if memory is swapped out too quickly then application response time drops, because when the application's window is clicked the system has to swap the application back into memory, which will make it feel slow.

The default value for swappiness is 60. You can alter it temporarily (until you next reboot) by typing as root:

echo 50 > /proc/sys/vm/swappiness

If you want to alter it permanently then you need to change the vm.swappiness parameter in the /etc/sysctl.conf file.

Conclusion

Managing swap space is an essential aspect of system administration. With good planning and proper use swapping can provide many benefits. Don't be afraid to experiment, and always monitor your system to ensure you are getting the results you need.

Swap space

Swap Memory is a space in the Hard Disk of your computer

that Operating Systems (Linux in our case) will use to put the info that is actually on the RAM to free it for another application.

This should be done when the system needs memory for a new process and there is none, so we can see that if our system has plenty of RAM it will maybe need no SWAP memory.

How much Swap Memory do I need?

As a rule of thumb if you have 512 MB RAM put 1 GByte Swap, but this stops being true when the limit at least in x386 PCs is achieved (2 GB or swap is the maximum I could ever allocate to a System).

How do I create Swap.

Usually when you install Linux you reserve a partition to be used as swap memory, and the rest of the disk for your files, but what happens if I need more swap memory?. How to create more swap memory?

If you disk is full you can try to shrink the partitions to make room for another swap partition but an easier way is to make a swap file, so now the question is.

How to create a swap file?

dd if=/dev/zero of=/swapfile bs=1024 count=100000
This will create file (swapfile) of size 100 MB (round)
mkswap /swapfile
add this file to your swap pool
swapon /swapfile

Incresing a swap space on Linux OS

This is simple method for Increase your available swap space with a swap file

All of your devices function, and everything is configured just the way you like it. At least you think so, until you start running out of memory when you have OpenOffice.org and lots of browser tabs open simultaneously. You realize you should have specified a larger swap partition during your install. this smiple method of installing gain .swap partition.

There’s more than one way to maintain your Linux system. Instead of creating a swap file, you could instead resize and or reshuffle your partitions with parted or its graphical front end QtParted.

To start , see how much swap space you already have. At a command line by typing ,
swapon -s (you might need to prepend /sbin/ if you’re not root)
the command should produce a message :

Filename Type Size Used Priority
/dev/hda2 partition 128044 92472 -1

The numbers under “Size” and “Used” are in kilobytes.

Let’s figure out where to put it. Running df -m from a command line should produce output something like this:

Filesystem 1M-blocks Used Available Use% Mounted on
/dev/hda1 11443 6191 5252 55% /

The -m switch we used provided us with output in megabytes. Under the “Available” column we have approximately 5GB of free space on our root partition. Let’s steal 512MB of that for our auxiliary swap file. You might want more or less, depending on your memory needs, how much swap space you already have available, and how much free disk space you have. The general rule of thumb for swap size is that your total available swap space should be around double your RAM size. If you have additional partitions, and one of those is a better candidate than the / partition, feel free to use it instead.

Please make Back up your important data before proceeding. If you carefully follow the steps below you should be fine,

In order to create our supplementary swap file, we’re going to use the dd (data dump) command. You’ll need to become root to perform the next few steps. su - and enter your root password. When you’re ready,

dd if=/dev/zero of=/extraswap bs=1M count=512

replacing 512 with the number of megabytes you want in your auxiliary swap file. if= and of= are short for infile and outfile. The /dev/zero device file will give us zeroes to be written to the output file. If you want this file on a different partition, say your /var partition, you would replace /extraswap with /var/extraswap.

Now we have a file the size we want on disk, and we can prepare it for use as a swap partition. We’ll use the mkswap command to make our file swap-consumable for the Linux kernel. Again as root,
#mkswap /extraswap

To turn on our swap file, we run swapon /extraswap. Now when we run swapon -s we should see our existing swap partition and our new swapfile. Also, the free command should show an increase in total swap space.

But we’re still not done yet. If we reboot our machine now, our new swapfile won’t be active, and we’ll have to run swapon /extraswap again. to make things more permanent, you need to edit our /etc/fstab file.

make a copy of the file. Something like this should do the trick:

cp /etc/fstab /etc/fstab.mybackup

open /etc/fstab in your favorite text editor and find a line about your swapfile that looks something like this:

/dev/hda2 none swap sw 0 0

You’ll need another line like that underneath it pointing to your new swap file. Replace the first column with the location of your new swap file. For our example, the new line should look like this:

/extraswap none swap sw 0 0

Save the file. Mistaken changes to /etc/fstab could render your system unbootable, so just to make sure you didn’t accidently change anything else in /etc/fstab, run diff /etc/fstab.mybackup /etc/fstab to check for differences. That should output only the single line you added, with a “>” sign in front of it. If you see anything else in diff’s output, edit /etc/fstab again, fix it, and run the above diff command again.

In practical terms, there’s a minimal performance hit from this extra step. By the time you’ve run out of RAM and are beginning to swap, you’re already suffering a massive performance hit. After your original swap partition is full and you’re spilling into your auxiliary swap file, your system should be suffering badly enough that the added performance hit will be completely imperceptible.

In order to avoid this sort of problem entirely with your next install, using Linux’s Logical Volume Manger is probably a good idea, and there are other Linux memory management techniques. Of course the ideal solution is to just install additional RAM.

Thursday, December 4, 2008

Linux creating CD-ROM ISO image

dd is a perfect tool for copy a file, converting and formatting according to the operands. It can create exact CD-ROM ISO image.

This is useful for making backup as well as for hard drive installations require a working the use of ISO images.

How do I use dd command to create an ISO image?

Put CD into CDROM

Do not mount CD. Verify if cd is mounted or not with mount command:

# mount

If cd was mouted automatically unmout it with umount command:

# umount /dev/cdrom

OR

# umount /mnt/cdrom

Create CD-ROM ISO image with dd command:

# dd if=/dev/cdrom of=/tmp/cdimg1.iso

Where,

  • if=/dev/cdrom: Read from /dev/cdrom (raw format)
  • of=/tmp/cdimg1.iso: write to FILE cdimg1.iso i.e. create an ISO image

Now you can use cdimg1.iso for hard disk installation or as a backup copy of cd. Please note that dd command is standard UNIX command and you should able to create backup/iso image under any UNIX like operating system.

Make an ISO Image

dd if=/dev/dvd of=dvd.iso # for dvd
dd if=/dev/cdrom of=cd.iso # for cdrom
dd if=/dev/scd0 of=cd.iso # if cdrom is scsi

To make an ISO from files on your hard drive, create a directory which holds the files you want. Then use the mkisofs command.

mkisofs -o /tmp/cd.iso /tmp/directory/

This results in a file called cd.iso in folder /tmp which contains all the files and directories in /tmp/directory/.

For more info, see the man pages for mkisofs, losetup, and dd, or see the CD-Writing-HOWTO at http://www.tldp.org.

If you want to create ISO images from a CD and you're using Windows, Cygwin has a dd command that will work. Since dd is not specific to CDs, it will also create disk images of floppies, hard drives, zip drives, etc.

For the Windows users, here are some other suggestions:

WinISO ~ http://www.winiso.com

VaporCD ~ http://vaporcd.sourceforge.net ~ "You can create ISOs from CD and mount them as 'virtual' CD drives. Works flawlessly with games and other CD based software. Unfortunately, it appears to be unmaintained now. Good thing it works so well." (P.B., 13 February 2002)