github linkedin
KVM storage caching modes perfomance comparison
In my everyday work i need perfomance rich KVM VMs on my workstation. I already use virtio drivers for network and storage. But what about tune it more?

First, I’ve moved my VMs from qcow images to raw LVM volumes.

Next, I’ve switced disk cache mode to ‘unsafe’ as most fastest. I don’t care about data integrity if smth goes wrong.

But decided to compare different modes. Detailed description about cache modes could be found here.

Cache modes differ mostly by used or not “page cache” of host operating sytem and used or not “write cache” of host hardware disk (See table 1).

Table 1

Disk Write Cache (Host)
Page Cache (Host)
writethrough
×
none
×
writeback
unsafe
Used, but ignores transfer operations
Used, but ignores transfer operations

I’ve done disk perfomance testing with different cache modes on raw LVM volume. Used bonnie++ benchmark. 5 tests for every cache mode. Full results here.

Table 2

    <td style="background-color: #b7b7b7; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        Write (sequental block), MBs
      </div>
    </td>

    <td style="background-color: #b7b7b7; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        Read (sequental block), MBs
      </div>
    </td>
  </tr>

  <tr style="height: 0px;">
    <td style="background-color: #b7b7b7; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        writethrough
      </div>
    </td>

    <td style="background-color: red; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        35
      </div>
    </td>

    <td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        2337 (page cache)
      </div>
    </td>
  </tr>

  <tr style="height: 0px;">
    <td style="background-color: #b7b7b7; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        none
      </div>
    </td>

    <td style="background-color: yellow; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        49
      </div>
    </td>

    <td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        103
      </div>
    </td>
  </tr>

  <tr style="height: 0px;">
    <td style="background-color: #b7b7b7; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        writeback
      </div>
    </td>

    <td style="background-color: yellow; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        45
      </div>
    </td>

    <td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        2381 (page cache)
      </div>
    </td>
  </tr>

  <tr style="height: 0px;">
    <td style="background-color: #b7b7b7; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        unsafe
      </div>
    </td>

    <td style="background-color: lime; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        54
      </div>
    </td>

    <td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;">
      <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
        2257 (page cache)
      </div>
    </td>
  </tr>
</table>

Most fastest mode is “unsafe”. But do not use “unsafe” mode if you need data integrity.
Optimal mode is “writeback”, it has good write perfomance and used page cache, that increases read perfomance a lot.
Safest mode is writethrough(by default in libvirt), but slowest.

Resume:
unsafe cache mode gave me +57% disk perfomance.

Specs:
Disk: Seagate Barracuda 500G (ST500DM002-1BC142)
MB: ASUS P8H67-M EVO
CPU: Intel® Core™ i5-2500K CPU @ 3.30GHz
OS: ARCH linux 3.11.6-1-ARCH x86_64 (libvirt 1.1.4-1, qemu 1.6.1-2)