Useful staps to track task movement across CPUs

Quite sometime back, I was faced with a situation where I needed to track instances when a particular task was being migrated away from a cpu. It was in the context of a real-time system, where a real-time task was facing huge context switch delays. Obvious suspect being the scheduler, I used systemtap to infer a few things, besides other debugging:

• To find if the task was being migrated away to some other cpu, used the following trivial stap script:

/* Filename: migrate.stp
* Author: Ankita Garg <ankita@in.ibm.com>
* Description: Captures information on the migration of threads
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* © Copyright IBM Corp. 2009. All Rights Reserved.
*
*/

probe kernel.function("__migrate_task")
{
if(($1 != 0 ) && (tid() == $1)) {
printf ("thread %d (%s) is migrating from %d to %d \n", $p->pid,
kernel_string($p->comm), $src_cpu, $dest_cpu);
}
}

• Below is a script that tracks all the cpus that a particular task ran on. Pl note it does not track the context switches.

/* Filename: chng_cpu.stp
* Author: Ankita Garg <ankita@in.ibm.com>
* Description: Captures information on the number of times java thread
* switches cpu
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* © Copyright IBM Corp. 2009. All Rights Reserved.
*
*/

global threads

probe kernel.function("finish_task_switch")
{
if ((threads[tid()] != cpu()) && (tid() != 0) && (execname() == @1)) {
printf("thread %d (%s) context switched on %d \n",
tid(), execname(), cpu());
printf("state: %d\n", task_state(task_current()))
print_stack(backtrace())
}
threads[tid()] = cpu();
}

These are a bit older techniques, as now there is a new tracepoints infrastructure which can do these things. But on older kernels, the above would be useful. Expect more posts on kernel RAS features in due time.