Explanation of Steal Time

Updated: May 30th, 2012

A common misconception about steal time (due to the unfortunate naming) is that it is a metric for showing the amount of CPU cycles stolen by other virtual machines in the same virtual host. No doubt that cloud service providers tend to oversell but steal time should not be the basis for this assumption.

Steal time actually accounts for the cycles the local virtual machine is trying to go over its originally allocated resources. It should actually be named involuntary wait as mentioned in the Linux kernel documentation for /proc/stat.

For example, from this Netflix and Stolen Time article ...

So how does Netflix handle this problem when using the cloud? Adrian admits that they tracked this statistic so closely that when an instance crossed a stolen time threshold the standard operating procedure at Netflix was to kill the virtual machine and start it up on a different hypervisor. What Netflix realized over time was that once a virtual machine was performing poorly because another virtual machine was crashing the party, usually due to a poorly written or compute intensive application hogging the machine, it never really got any better and their best learned approach was to get off that machine.

The newly booted virtual machines are probably being provisioned on a non-busy virtual host allowing them to borrow SPARE cycles. The effect of rebooting a virtual machine along with its long-running processes can also fix problems especially if it hosts poorly written and compute intensive applications.

Good article about the topic:

http://www.axibase.com/cloud/2010/07/22/ec2-monitoring-the-case-of-stolen-cpu/

Straight to the point answer from AWS:

https://forums.aws.amazon.com/thread.jspa?threadID=79519

To clear doubts, we traced the 'steal' metric %st as shown by top(1) and vmstat(8) which is scraped from /proc/stat:

fs/proc/stat.c

   steal = cputime64_add(steal, kstat_cpu(i).cpustat.steal);

 

kernel/sched.c

   ...

   void account_steal_ticks(unsigned long ticks)

   {

           account_steal_time(jiffies_to_cputime(ticks));

   }

   ...

   void account_steal_time(cputime_t cputime)

   {

           struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;

           cputime64_t cputime64 = cputime_to_cputime64(cputime);

           cpustat->steal = cputime64_add(cpustat->steal, cputime64);

   }

   ...

 

arch/x86/xen/time.c

   ...

         runnable = state.time[RUNSTATE_runnable] -

snap->time[RUNSTATE_runnable];

         offline = state.time[RUNSTATE_offline] - snap->time[RUNSTATE_offline];

         *snap = state;

         /* Add the appropriate number of ticks of stolen time,

            including any left-overs from last time. */

         stolen = runnable + offline + __this_cpu_read(xen_residual_stolen);

         if (stolen < 0)

                 stolen = 0;

         ticks = iter_div_u64_rem(stolen, NS_PER_TICK, &stolen);

         __this_cpu_write(xen_residual_stolen, stolen);

         account_steal_ticks(ticks);

   ...

 

include/xen/interface/vcpu.h

   /* VCPU is currently running on a physical CPU. */

   #define RUNSTATE_running  0

   /* VCPU is runnable, but not currently scheduled on any physical CPU. */

   #define RUNSTATE_runnable 1

   /* VCPU is blocked (a.k.a. idle). It is therefore not runnable. */

   #define RUNSTATE_blocked  2

   /* VCPU is not runnable, but it is not blocked.

   * This is a 'catch all' state for things like hotplug and pauses by the

   * system administrator (or for critical sections in the hypervisor).

   * RUNSTATE_blocked dominates this state (it is the preferred state).

   */

   #define RUNSTATE_offline  3

  ...

The macros __this_cpu_read and __this_cpu_write references the DomU CPU. RUNSTATE_runnable and RUNSTATE_offline absolutely does not reflect stolen resources. If a virtual machine has enough resources for its applications the DomU CPU should not go in to the runnable state, it will be either running, blocked or sometimes offline. In Xen you can borrow but you can't steal. If the virtual machine's CPU is mostly in the runnable state or top/vmstat showing a high steal time it's trying hard to borrow cycles.

0

Please sign in to leave a comment.