KSYMOOPS(8)                                                        KSYMOOPS(8)


       ksymoops - a utility to decode Linux kernel Oops


       [ -v vmlinux ] [ --vmlinux=vmlinux ] [ -V ] [ --no-vmlinux ]
       [ -k ksyms ] [ --ksyms=ksyms ] [ -K ] [ --no-ksyms ]
       [ -l lsmod ] [ --lsmod=lsmod ] [ -L ] [ --no-lsmod ]
       [ -o object ] [ --object=object ] [ -O ] [ --no-object ]
       [ -m system.map ]   [ --system-map=system.map ]  [ -M ]  [ --no-system-
       map ]
       [ -s save.map ] [ --save-map=save.map ]
       [ -S ] [ --short-lines ]
       [ -e ] [ --endian-swap ]
       [ -x ] [ --hex ]
       [ -1 ] [ --one-shot ]
       [ -i ] [ --ignore-insmod-path ]
       [ -I ] [ --ignore-insmod-all ]
       [ -T truncate ] [ --truncate=truncate ]
       [ -d ] [ --debug ]
       [ -h ] [ --help ]
       [ -t target ] [ --target=target ]
       [ -a architecture ] [ --architecture=architecture ]
       [ -A "address list" ] [ --addresses="address list" ]
       [ Oops.file ... ]


       ksymoops extracts kernel Oops  reports  from  the  Oops.file  and  uses
       various sources of symbol information to convert the addresses and code
       to meaningful text.  Reporting a kernel Oops is meaningless on its  own
       because  other people do not know what your kernel looks like, you need
       to feed the Oops text through ksymoops then send the ksymoops output as
       part of your bug report.

       The  ksymoops  executable  is meant to be run whenever you have Oops to
       report.  The original Oops text can come from anywhere.   Typically  it
       is  in a file created by your syslogd(8).  If syslogd is not available,
       the log might be available via dmesg(8).  If you are running  a  serial
       console   (see  linux/Documentation/serial-console.txt)  then  you  can
       capture the Oops text on another machine.  If all else fails, copy  the
       Oops by hand from the screen, reboot and enter it by hand.

       ksymoops can be run by anybody who has read access to the various input
       files.  It does not have to be run as root.


       Some of the options have default values that are set in  the  Makefile.
       The  text  below  describes the standard defaults but your distribution
       may have been  modified  to  use  different  defaults.   If  in  doubt,
       ksymoops -h will list the current defaults.

       The  first  10  options  (-v, -V, -k, -K, -l, -L, -o, -O, -m, -M or the
       corresponding long forms) are 5 pairs.  The lower case options  (vklom)
       take  a  value  and  turn the option on, the upper case options (VKLOM)
       take no value and turn the option off.  If you specify both  lower  and
       upper  case  versions  of the same option then the last one is used but
       you are warned that it may not be what you intended.

       ksymoops will run quite happily with no options.  However  there  is  a
       risk  that  the  default  values  for  the  symbol  sources  may not be
       suitable.  Therefore if none of -v vmlinux, -V, -k ksyms, -K, -l lsmod,
       -L, -o object, -O, -m system.map or -M are specified, ksymoops prints a
       warning message.

           You did not tell me where  to  find  symbol  information.   I  will
           assume that the log matches the kernel and modules that are running
           right now and  I'll  use  the  default  options  above  for  symbol
           resolution.   If the current kernel and/or modules do not match the
           log, you can get more accurate output  by  telling  me  the  kernel
           version  and  where  to  find map, modules, ksyms etc.  ksymoops -h
           explains the options.

       If  any  of  the   -v vmlinux,   -k ksyms,   -l lsmod,   -o object   or
       -m system.map  options  contain  the  string  *r  (*m, *n, *s) then the
       string is replaced at run time by the current value of `uname -r`  (-m,
       -n, -s).  This is mainly intended to let ksymoops automatically pick up
       version dependent files using its default parameters, however it  could
       be  used  by bug reporting scripts to automatically pick up files whose
       name or directory depends on the current kernel.

       -v vmlinux --vmlinux=vmlinux
              Name of the vmlinux file that corresponds to the failing kernel.
              Note: This  is  the  vmlinux  file, not zImage, bzImage, vmlinuz
              etc.  Typically this would be  /usr/src/linux/vmlinux.   If  you
              specify -v, you should only specify it once.

       -V --no-vmlinux
              Do not read any vmlinux file.

       Default is -V.

       -k ksyms --ksyms=ksyms
              Where  to  find  the  list  of kernel symbols at the time of the
              failure.  Unfortunately the kernel symbol list in /proc/ksyms is
              volatile,  it is updated as modules are loaded and removed.  Try
              to copy /proc/ksyms to a normal file as soon as  possible  after
              the Oops and point ksymoops at that copy using -k.  Modutils has
              support for automatically copying  ksyms  and  lsmod  data,  see
              insmod(8).   If  you had to reboot after the Oops and you do not
              have a copy of /proc/ksyms at the  time  of  the  Oops,  try  to
              reload  the  same  modules  in  the  same  order  before running
              ksymoops.  If you specify -k, you should only specify it once.

       -K --no-ksyms
              Do not read any kernel symbols.

       Default is -k /proc/ksyms.

       -l lsmod --lsmod=lsmod
              Where to find the list of loaded modules  at  the  time  of  the
              failure.   Unfortunately  the list in /proc/modules is volatile,
              it is updated as modules are loaded and removed.   Try  to  copy
              /proc/modules  to  a  normal  file as soon as possible after the
              Oops and point ksymoops at that copy  using  -l.   Modutils  has
              support  for  automatically  copying  ksyms  and lsmod data, see
              insmod(8).  If you had to reboot after the Oops and you  do  not
              have  a  copy  of  /proc/modules at the time of the Oops, try to
              reload the  same  modules  in  the  same  order  before  running
              ksymoops.  If you specify -l, you should only specify it once.

       -L --no-lsmod
              Do not read any list of loaded modules.

       Default is -l /proc/modules.

       -o object --object=object
              Where  to  find  the  objects  for  modules  used by the failing
              kernel.  This can be a directory name or an individual file.  If
              it is a directory then ksymoops does a recursive find(1) in that
              directory for all files matching '*.o'.   -o  can  be  specified
              more than once, the list is cumulative and can contain a mixture
              of directories and files.

              Note: When you specify a directory,  ksymoops  only  uses  files
              that  end  in  '.o'.   Any  modules  with non-standard names are
              ignored unless you specify those files explicitly.  For example,
              if  vmnet  and  vmmon  modules  do  not  end  in  '.o', you need
              something like this to pick up all the normal modules  plus  the
              non-standard names.

                -o /lib/modules/*r/ \
                -o /lib/modules/*r/misc/vmnet \
                -o /lib/modules/*r/misc/vmmon

              If  you  are using a version of insmod(8) that stores the module
              filename in /proc/ksyms, ksymoops can go directly to that  file,
              it  does  not  need  -o.   The -o option is only used when ksyms
              contains at least one module whose filename  is  not  explicitly
              listed in ksyms.

       -O --no-object
              Do  not  scan  for  any objects.  If /proc/ksyms is supplied and
              insmod added the  ksymoops  assistance  symbols  (starting  with
              __insmod)  then those symbols are used to access the objects, no
              directory scanning is done so neither -o nor -O have any effect.
              To  completely  disable  the  use  of  module objects when ksyms
              contains __insmod symbols, specify -O and one of -i or -I.

       Default is  -o /lib/modules/*r/.   For  example,  if  uname -r  reports
       2.2.7,  ksymoops  uses  -o /lib/modules/2.2.7/, but only if it does not
       already know where the objects are.

       -m system.map --system-map=system.map
              Where to  find  the  System.map  corresponding  to  the  failing

       -M --no-system-map
              Do not read any System.map.

       Default is -m /usr/src/linux/System.map.

       -s save.map --save-map=save.map
              After ksymoops reads all its sources of symbols, it generates an
              internal system map which contains  everything  from  System.map
              plus  a  best attempt to extract all symbols from all the loaded
              modules.  If you want to  see  that  consolidated  map,  specify
              -s save.map  to  write  it  out to save.map.  You do not need to
              save the map for normal bug reporting.

       Default is no saved map.

       -S --short-lines
              Some of the ksymoops output lines can be quite long,  especially
              in  the  code  disassembly,  but  if  you have a wide screen the
              ksymoops output is easier to read as long lines.  The -S  toggle
              switches  between short and long lines.  Note that lines printed
              by the kernel and extracted from the Oops.file are not  affected
              by -S, problem text is printed as is.

       Default is short lines.

       -e --endian-swap
              ksymoops  extracts code bytes from the reports and converts them
              to instructions.  All  kernels  print  code  bytes  in  hex  but
              unfortunately some systems print multiple bytes using the native
              machine endianess.  This only causes a problem if  the  code  is
              printed in anything other than 1 byte chunks.  For example, i386
              prints one byte at a  time  which  is  machine  portable,  alpha
              prints  4  bytes at a time in native endianess and the report is
              not machine portable.

              If you are doing cross system Oops  diagnosis  (say  for  a  new
              system or an embedded version of Linux), then the failing system
              and the reporting  system  can  have  different  endianess.   On
              systems  that support little and big endianess at the same time,
              ksymoops could be compiled with one  endianess  but  the  kernel
              dump could be using another.  If your code disassembly is wrong,
              specify -e.  The -e toggles between native and reverse endianess
              when  reading the bytes in each chunk of code.  In this context,
              a chunk of code is 4 or 8 hex digits (2 or 4 bytes of code),  -e
              has  no effect on code that is printed as 2 hex digits (one byte
              at a time).

              Note: Earlier versions of ksymoops used a -c code_bytes  option.
              That  is  now  obsolete,  use -e instead, but only when the code
              disassembly is incorrect.

       The default is to read code bytes using the endianess that ksymoops was
       compiled with.

       -x --hex
              Normally,  ksymoops  prints  offsets and lengths in hex.  If you
              want offsets and lengths to be printed in decimal,  use  the  -x

       Default is hex.

       -1 --one-shot
              Normally,  ksymoops reads its entire input file and extracts all
              Oops reports.  If the -1 toggle is set, it will run in one  shot
              mode  and  exit  after  the  first  Oops.   This  is  useful for
              automatically mailing reports as they happen, like this :-

                  # ksymoops1
                  while (true)
                      ksymoops -1 > $HOME/oops1
                      if [ $? -eq 3 ]
                         exit 0  # end of input, no Oops found
                      mail -s Oops admin < $HOME/oops1

                  tail -f /var/log/messages | ksymoops1

              Restarting the tail command after log rotation  is  left  as  an
              exercise for the reader.

              In  one  shot  mode,  reading  of  the various symbol sources is
              delayed until ksymoops sees  the  first  program  counter,  call
              trace  or  code  line.   This  ensures  that  the current module
              information is used.  The downside is that any parameter  errors
              are not detected until an Oops actually occurs.

       The  default  is  to read everything from the Oops.file, extracting and
       processing every Oops it finds.  Note that the default method reads the
       symbol  sources  once  and assumes that the environment does not change
       from one Oops to the next, not necessarily valid  when  you  are  using

       -i --ignore-insmod-path
              When you are using an initial ramdisk for modules, the path name
              to the  modules  is  typically  just  /lib.   That  pathname  is
              recorded  in  the  __insmod..._O  symbol  in  ksyms but ksymoops
              cannot find the files in the real /lib,  so  it  issues  warning
              messages.  If you specify -i then ksymoops ignores the path name
              in __insmod...O symbols, instead it searchs  the  -o  paths  (if
              any)  looking  for  the  object  with  the  correct basename and
              timestamp.  -i  is  recommended  when  loading  modules  from  a
              ramdisk.   This  assumes  that  the -o paths contain the modules
              used to build the ramdisk, with the same timestamp.

       Default is to use the path from __insmod...O symbols.

       -T --ignore-insmod-all
              Use this toggle if you want to completely ignore  all  insmod(8)
              assistance   information  (symbols  starting  with  __insmod  in
              ksyms).  This includes module paths, timestamps,  section  start
              and  length etc.  Then ksymoops will fall back on the old method
              of matching symbols to module objects, using the  -o  paths  (if
              any).   It is hard to think of a legitimate reason to use -I, -i
              is better if your only problem is a path name mismatch.

       Default is to use  the  path  from  __insmod...O  symbols  and  section
       information from __insmod...S symbols.

       -T truncate --truncate=truncate
              If  your binutils are configured for multiple targets, they tend
              to print addresses using the address size of the largest target.
              If  the  other inputs to ksymoops have shorter symbol sizes, the
              different representations cause symbols which  should  have  the
              same  address  to  appear  at  different  addresses.   This is a
              particular problem  when  building  for  mixed  32  and  64  bit
              targets.   To  remove the ambiguity, use --truncate=truncate.  A
              value  of  0  means  no  truncation,  a   value   greater   than
              8*sizeof(unsigned long) is silently converted to 0.

       Default is --truncate=0, no truncation.

       -d --debug
              Each  occurrence of -d increases the debugging level of ksymoops
              by one.

       Level 1
              Regular expression compile summaries.  Before and after text for
              *[mns]  expansion.   Option  processing,  but  only  for options
              appearing after -d.  Entry  to  the  main  processing  routines.
              KSYMOOPS_   environment   variables.    Object  files  extracted
              directly from ksyms.   Information  on  matches  between  loaded
              modules  and  module  objects.   Filename  of  the  Oops report.
              Version number for the oops.  Saving merged system map.

       Level 2
              Summary information on symbol table sizes.  Every version number
              found  in  the  oops.   Comparing symbol maps.  Appending symbol
              maps.  Full pathname of a program.   External  commands  issued.
              Progress  reports for -o object.  The names of '*.o' files found
              in a -o directory.   Offset  adjustments  for  module  sections.
              Every line output from running objdump on the code bytes.

       Level 3
              Every  input line from Oops.file.  Non-duplicate and low address
              symbols  dropped  from  the  merged  system  map.   Mapping   of
              addresses to symbols.

       Level 4
              Every  input line from all sources, this prints duplicate lines.
              The return code from every regexec call.  Ambiguous matches that
              are ignored.  Every symbol added to every table.  Copying symbol
              tables.  Increases in symbol table sizes.  Entry to  some  lower
              level routines.  Every symbol dropped.

       Level 5
              For matching regexecs, details on every substring.

       Default is no debugging.

       -h --help
              Prints the help text and the current defaults.

       -t target --target=target
              Normally  you  do  Oops diagnosis using the same hardware as the
              Oops itself.  But sometimes you need to  do  cross  system  Oops
              diagnosis,  taking  an  Oops  from  one  type  of  hardware  and
              processing it on an another.  For example, when you are  porting
              to  a  new system or you are building an embedded kernel.  To do
              cross system Oops processing, you must tell  ksymoops  what  the
              target  hardware  is,  using  -t target,  where  target is a bfd
              target name.  You  can  find  out  which  targets  your  machine
              supports by

                ksymoops -t '?'

       Default  is the same target as ksymoops itself, with one exception.  On
       sparc64, the kernel uses elf64-sparc but user programs are elf32-sparc.
       If   -t target   was  not  specified  and  ksymoops  was  compiled  for
       elf32-sparc  and  the  Oops  contains  a   TPC   line   then   ksymoops
       automatically switches to -t elf64-sparc.

       -a architecture --architecture=architecture
              To  do cross system Oops processing, you must tell ksymoops what
              the  target  architecture  is,  using   -a architecture,   where
              architecture is a bfd architecture name.  You can find out which
              architectures your machine supports by

                ksymoops -a '?'

       Default  is  the  same  architecture  as  ksymoops  itself,  with   one
       exception.  On sparc64, the kernel uses sparc:v9a but user programs are
       sparc.  If -a architecture was not specified and ksymoops was  compiled
       for  sparc and the Oops contains a TPC line then ksymoops automatically
       switches to -a sparcv:9a.

       -A "address list" --addresses="address list" If you
              have a few adhoc  addresses  to  convert  to  symbols,  you  can
              specify  them  explicitly using -A "address list".  Any words in
              the list that appear to be addresses are converted  to  symbols.
              Punctuation   characters  and  non-address  words  are  silently
              ignored, leading 0x on addresses is also  ignored,  so  you  can
              paste  text  including  words  and  only  the  addresses will be

       Oops.file ...
              ksymoops accepts zero or more input files and  reads  them  all.
              If  no  files are specified on the command line and no addresses
              are supplied via -A then ksymoops  reads  from  standard  input.
              You  can  even  type  the  Oops  text  directly at the terminal,
              although that is not recommended.


       ksymoops reads the input file(s), using regular expressions  to  select
       lines  that are to be printed and further analyzed.  You do not need to
       extract the Oops report by hand.

       All tabs are converted to spaces, assuming tabstop=8.  Where  the  text
       below  says  "at  least  one  space",  tabs  work  just as well but are
       converted to spaces before printing.  All nulls  and  carriage  returns
       are  silently  removed from input lines, both cause problems for string
       handling and printing.

       An input line can have a prefix which ksymoops will print  as  part  of
       the  line  but ignore during analysis.  A prefix can be from syslogd(8)
       (consisting  of  date,  time,  hostname,  'kernel:'),  from   syslog-ng
       (numbers and three other strings separated by '|'), it can be "start of
       line" means the first character after skipping all prefixes,  including
       all leading space.

       Every  kernel  architecture  team  uses  different  messages for kernel
       problems, see Oops_read in oops.c for the full, gory list.  If you  are
       entering  an Oops by hand, you need to follow the kernel format as much
       as possible, otherwise ksymoops may not recognize your input.  Input is
       not case sensitive.

       A  bracketed  address  is  optional  '[',  required '<', at least 4 hex
       digits, required '>',  optional  ']',  optional  spaces.   For  example
       [<01234567>] or <beaf>.

       An  unbracketed  address is at least 4 hex digits, followed by optional
       spaces.  For example 01234567 or abCDeF.

       The sparc PC line is 'PSR:' at start of line, space, hex digits, space,
       'PC:', space, unbracketed address.

       The  sparc64  TPC  line  is  'TSTATE:'  at start of line, space, 16 hex
       digits, space 'TPC:', space, unbracketed address.

       The ppc  NIP  line  has  several  formats.   'kernel pc'  'trap at PC:'
       'bad area pc'  or  'NIP:'.   Any  of those strings followed by a single
       space and an unbracketed address is the NIP value.

       The mips PC line is 'epc' at start of line, optional space, one or more
       ':', optional space, unbracketed address.

       The  ix86  EIP line is 'EIP:' at start of line, at least one space, any
       text, bracketed address.

       The x86_64 EIP line is 'RIP:' at start of line, at least one space, any
       text, bracketed address.

       The  m68k  PC  line  is  'PC'  at  start of line, optional spaces, '=',
       optional spaces, bracketed address.

       The arm PC line is  'pc'  at  start  of  line,  optional  spaces,  ':',
       optional spaces, bracketed address.

       The  IA64  IP  line  is  '  ip',  optional  space, ':', optional space,
       bracketed address.

       A mips ra line is 'ra', optional spaces,  one  or  more  '=',  optional
       spaces, unbracketed address.

       A  sparc  register  dump  line  is  ('i',  '0'  or  '4', ':', space) or
       ('Instruction DUMP:', space) or ('Caller[').

       The IA64  b0  line  is  'b0',  optional  space,  ':',  optional  space,
       unbracketed  address.  This can be repeated for other b registers, e.g.
       b6, b7.

       Register dumps have a plethora of formats.  Most are of the form 'name:
       value'  repeated  across  a line, some architectures use '=' instead of
       ':'.  See Oops_regs for the current list of recognised register  names.
       Besides  the  Oops_regs  list,  i370,  mips,  ppc and s390 have special
       register dump formats, typically one register name is printed  followed
       by  multiple  values.   ksymoops  extracts  all  register  contents and
       decodes those values above an arbitrary cutoff of 1024.

       A set of call trace lines starts  with  'Trace:'  or  'Call Trace:'  or
       'Call Backtrace:'  (ppc  only)  or  'Function entered at' (arm only) or
       'Caller[' (sparc64 only) followed by at least one space.

       For 'Trace:' and 'Call Trace:', the  rest  of  the  line  is  bracketed
       addresses,  they  can be continued onto extra lines.  Addresses can not
       be split across lines.

       For 'Call Backtrace:' (ppc only), the rest of the line  is  unbracketed
       addresses,  they  can be continued onto extra lines.  Addresses can not
       be split across lines.

       For 'Function entered at' (arm only), the  line  contains  exactly  two
       bracketed addresses and is not continued.

       For 'Caller[' (sparc64 only), the line contains exactly one unbracketed
       address and is not continued.

       Spin loop information is indicated by  a  line  starting  with  'bh: ',
       followed  by  lines  containing reverse bracketed trace back addresses.
       For some reason, these addresses are different from every other address
       and   look   like   this   '<[hex]> <[hex]>'   instead  of  the  normal
       '[<hex>] [<hex>]'.

       The Code line is identified by 'Instruction DUMP' or  ('Code'  followed
       by  optional spaces), ':', one or more spaces, followed by at least one
       hex value.  The line can contain multiple hex values, each separated by
       at least one space.  Each hex value must be 2 to 8 digits and must be a
       multiple of 2 digits.

       Any of the code values can be enclosed in <..> or (..), the  last  such
       value  is  assumed to be the failing instruction.  If no value has <..>
       or (..) then the first byte is assumed to be the failing instruction.

       Special cases where Code: can be  followed  by  text.   'Code:  general
       protection'  or  'Code:  <n>'.   Dump  the  data  anyway,  the code was

       Do you detect a slight note of inconsistency in the above?


       Addresses are converted to symbols based on  the  symbols  in  vmlinux,
       /proc/ksyms,  object  files  for  modules and System.map, or as many of
       those sources as ksymoops was told to  read.   ksymoops  uses  as  many
       symbol  sources  as  you  can  provide,  does  cross checks between the
       various sources to identify any discrepancies and builds a  merged  map
       containing all symbols, including loaded modules where possible.

       Symbols  which end in _R_xxxxxxxx (8 hex digits) or _R_smp_xxxxxxxx are
       symbol versioned, see genksyms(8).  ksymoops  strips  the  _R_...  when
       building its internal system map.

       Module  symbols  do  not  appear  in  vmlinux  nor  System.map and only
       exported  symbols  from  modules  appear  in  /proc/ksyms.    Therefore
       ksymoops  tries  to read module symbols from the object files specified
       by -o.  Without these module symbols, diagnosing a problem in a  module
       is almost impossible.

       There  are  many problems with module symbols, especially with versions
       of insmod(8) up to and including 2.1.121.  Some modules do  not  export
       any  symbols,  there  is  no  sign  of  them in /proc/ksyms so they are
       effectively  invisible.   Even  when  a  module  exports  symbols,   it
       typically only exports one or two, not the complete list that is really
       needed for Oops diagnosis.  ksymoops can build a complete symbol  table
       from the object module but it has to

       (a) Know that the module is loaded.

       (b) Find the correct object file for that module.

       (c) Convert  section  and  symbol  data  from  the  module  into kernel

       If a module exports no symbols then there is no  way  for  ksymoops  to
       obtain  any information about that module.  lsmod says it is loaded but
       without symbols, ksymoops cannot find the corresponding object file nor
       map offsets to addresses.  Sorry but that is the way it is, if you Oops
       in a module that displays no symbols in ksyms, forget it :(.

       When a module exports symbols, the next step is to find the object file
       for  that  module.  In most cases the loaded module and the object file
       has the same basename but that is not guaranteed.  For example,
         insmod uart401 -o xyz
       will load uart401.o from your module directories but store it  as  xyz.
       Both  ksyms and lsmod say module name 'xyz' with no indication that the
       original object file was uart401.  So  ksymoops  cannot  just  use  the
       module  name  from ksyms or lsmod, it has to do a lot more work to find
       the correct object.  It does this by looking for a unique match between
       exported symbols and symbols in the module objects.

       For  every  file  obtained from the -o option(s), ksymoops extracts all
       symbols (both static and external), using  nm(1).   It  then  runs  the
       exported module symbols in ksyms and, for every exported module symbol,
       it does a string compare of that symbol against every symbol  in  every
       object.   When ksymoops finds a module symbol that is exported in ksyms
       and appears exactly once amongst all the -o  objects  then  it  has  to
       assume that the object is the one used to load the module.  If ksymoops
       cannot find any match for any exported symbol in a module or finds more
       than  one  match  for  every exported symbol in a module then it cannot
       determine which object was actually loaded.

       After ksymoops has matched a loaded module against an  object  using  a
       unique symbol, it still has to calculate addresses for the symbols from
       the object.  To do this, ksymoops first needs the start address of  the
       text, data and read only data sections in the loaded module.  Given the
       start address of a section, ksymoops can calculate the  kernel  address
       of  every  symbol  in  that section and add the symbols to the combined
       system map, this includes symbols that are not exported.  Unfortunately
       the  start address of a section is only available if the module exports
       at least one symbol from that section.  For example, if a  module  only
       exports  text  symbols  (the  most  common case) then ksymoops can only
       calculate the start of the text section and has to discard symbols from
       the  data  and  read  only  data sections for that module, reducing the
       information available for diagnosis.

       When multiple symbol sources are available  and  those  symbol  sources
       contain  a  kernel  version  number,  ksymoops compares all the version
       numbers.  It flags a warning if there is any mismatch.  One of the more
       common  causes  of  problems  is force loading a module from one kernel
       into a different kernel.  Even if it was deliberate,  it  needs  to  be
       highlighted for diagnosis.

       When  both ksyms and lsmod are available, the list of modules extracted
       from ksyms is compared against the list of  modules  from  lsmod.   Any
       difference  is  flagged  as a warning, it typically indicates invisible
       modules.  However it can also be caused by a mismatch between ksyms and

       When  multiple symbol sources are available, ksymoops does cross checks
       between them.  Each check is only performed if both symbol sources  are
       present  and non-empty.  Every symbol in the first source should appear
       in the second source and should have the same address.  Where there  is
       any discrepancy, one of the sources takes precedence, the precedence is
       somewhat arbitrary.  Some discrepancies are  silently  ignored  because
       they are special cases but the vast majority of symbols are expected to

       * Exported module symbols in ksyms are compared against the symbols  in
         the corresponding object file.  ksyms takes precedence.

       * The  kernel  (non  module)  symbols  from  ksyms are compared against
         vmlinux.  vmlinux takes precedence.

       * The symbols from System.map are compared  against  vmlinux.   vmlinux
         takes precedence.

       * The  symbols  from  vmlinux are compared against System.map.  vmlinux
         takes precedence.  These two sources are compared in both directions,
         they should be identical.

       * The  kernel  (non  module)  symbols  from  ksyms are compared against
         System.map.  System.map takes precedence.

       After reading and cross checking  all  the  symbol  sources,  they  are
       merged into a single system map.  Duplicate symbols, registers (type a)
       and static 'gcc2_compiled.' symbols are dropped from  the  merged  map.
       Any symbols with an address below 4096 are discarded, these are symbols
       like Using_Versions which has an address of 0.

       Given all the above processing and deduction, it is  obvious  that  the
       merged  system map cannot be 100% reliable, which means that conversion
       of addresses to symbols cannot be reliable.  The  addresses  are  valid
       but the symbol conversion is only as good as the symbol sources you fed
       into ksymoops.

       /proc/ksyms and /proc/lsmod are volatile so unless  ksymoops  gets  the
       current  ksyms,  you always have to question the validity of the module
       information.  The only way I know to (almost) guarantee valid ksyms  is
       to  use ksymoops in one shot mode (see option -1).  Then ksymoops reads
       the log and decodes Oops in real time.


       Modutils 2.3.1 onwards has  support  to  make  oops  debugging  easier,
       especially  for  modules.   See  insmod(8)  for  details.   If you want
       automatic snapshots of ksyms and lsmod data as modules are  loaded  and
       unloaded,  create  /var/log/ksymoops,  it  should be owned by root with
       mode 644 or 600.  If you do not want automatic snapshots, do not create
       the   directory.   A  script  (insmod_ksymoops_clean)  is  provided  by
       modutils to delete old versions, this should be run by cron once a day.


       ksymoops  prints  all  lines  that  contain text which might indicate a
       kernel problem.  Due the complete lack of  standards  in  kernel  error
       messages,  I  cannot  guarantee that all problem lines are printed.  If
       you see a line in your logs which ksymoops should extract but does not,
       contact the maintainer.

       When  ksymoops  sees  EIP/PC/NIP/TPC  lines,  call  trace lines or code
       lines, it prints them and stores them for later processing.   When  the
       code line is detected, ksymoops converts the EIP/PC/NIP/TPC address and
       the call trace addresses to symbols.  These lines have  ';'  after  the
       header  instead  of  ':',  just  in case anybody wants to feed ksymoops
       output back into ksymoops, these generated lines are ignored.

       Formatted data for the program counter, trace and code is  only  output
       when  the  Code:  line  is seen.  If any data has been stored for later
       formatting and more than 5 lines other than Oops text or  end  of  file
       are encountered then ksymoops assumes that the Code: line is missing or
       garbled and dumps the formatted data anyway.  That should be fail  safe
       because  the Code: line (or its equivalent) signals the end of the Oops
       report.  Except for sparc64 on SMP which has a register dump after  the
       code.  ksymoops tries to cater for this exception.  Sigh.

       Addresses are converted to symbols wherever possible.  For example

         >>EIP; c0113f8c <sys_init_module+49c/4d0>
         Trace; c011d3f5 <sys_mremap+295/370>
         Trace; c011af5f <do_generic_file_read+5bf/5f0>
         Trace; c011afe9 <file_read_actor+59/60>
         Trace; c011d2bc <sys_mremap+15c/370>
         Trace; c010e80f <do_sigaltstack+ff/1a0>
         Trace; c0107c39 <overflow+9/c>
         Trace; c0107b30 <tracesys+1c/23>
         Trace; 00001000 Before first symbol

       Each  converted  address  is  followed by the nearest symbol below that
       address.  That symbol is followed by the offset of the address from the
       symbol.   The  value  after  '/'  is  the  "size"  of  the  symbol, the
       difference between the symbol and the next known symbol.  So
         >>EIP; c0113f8c  <sys_init_module+49c/4d0>  means  that  the  program
       counter  was  c0113f8c.   The  previous  symbol is sys_init_module, the
       address is 0x49c bytes from the start of the symbol, sys_init_module is
       0x4d0 bytes long.  If you prefer decimal offsets and lengths see option
       -x.   If  the  symbol  comes  from  a  module,  it   is   prefixed   by
       '[module_name]', several modules have the same procedure names.

       The use of 'EIP' for program counter above is for ix86.  ksymoops tries
       to use the correct acronym for the program counter (PC, NIP, TPC  etc.)
       but if it does not recognize the target hardware, it defaults to EIP.

       When  a  Code: line is read, ksymoops extracts the code bytes.  It uses
       the program counter line together with the code  bytes  to  generate  a
       small  object  file  in the target architecture.  ksymoops then invokes
       objdump(1)  to  disassemble  this  object  file.   The  human  readable
       instructions  are  extracted  from  the objdump output and printed with
       address to symbol conversion.  If the disassembled code does  not  look
       sensible, see the -e, -a and -t options.

       addresses are valid but the symbol conversion is only as  good  as  the
       input  you  gave  ksymoops.  See all the problems in "ADDRESS TO SYMBOL
       CONVERSION" above.  Also the stack trace is potentially ambiguous.  The
       kernel prints any addresses on the stack that might be valid addresses.
       The kernel has no way of telling which (if any) of these addresses  are
       real  and  which  are just lying on the stack from previous procedures.
       ksymoops just decodes what the kernel prints.


              Path for nm, defaults to /usr/bin/nm.

              Path for find, defaults to /usr/bin/find.

              Path for objdump, defaults to /usr/bin/objdump.


       To process an Oops from one system on another, you need access  to  all
       the  symbol  sources, including modules, System.map, ksyms etc.  If the
       two systems are different hardware, you also need versions  of  the  nm
       and  objdump  commands  that  run  on your system but handle the target
       system.  You also need versions of libbfd,  libopcodes,  and  libiberty
       that  handle the target system.  Consult the binutils documentation for
       instructions on how to build cross system versions of these  utilities.

       To  override  the  default versions of nm and find, use the environment
       variables above.  To use different versions of  libbfd  and  libiberty,
       use  the  --rpath  option  when linking ksymoops or the LD_LIBRARY_PATH
       environment variable when running ksymoops.  See the info pages for  ld
       and /usr/doc/glibc*/FAQ.


       0 - normal.

       1 - error(s) or warning(s) issued, results may not be reliable.

       2 - fatal error, no useful results.

       3 - One shot mode, end of input was reached without seeing an Oops.


       Because  of  the  plethora  of  possible  kernel  error and information
       strings, ksymoops's pattern matching sometimes prints  lines  that  are
       not errors at all.  For example, a line starting with 3c589 matches the
       pattern for a call trace line, both start with at least 4  hex  digits.
       Humans are smarter than programs, ignore spurious lines.


       Keith Owens <kaos@ocs.com.au> - maintainer.

       Patches  from Jakub Jelinek <jj@sunsite.mff.cuni.cz>, Richard Henderson


       The   original    ksymoops.cc    was    written    by    Greg    McGary
       <gkm@magilla.cichlid.com>     and    updated    by    Andreas    Schwab
       <schwab@issan.informatik.uni-dortmund.de>.  That version  required  C++
       and supported only ix86 and m68k.

       To  get  the equivalent of the old ksymoops.cc (no vmlinux, no modules,
       no  ksyms,  no  System.map)  use  ksymoops -VKLOM.   Or  to  just  read
       System.map, ksymoops -VKLO -m mapfile.


       find(1),     insmod(8),    nm(1),   objdump(1),   rmmod(8),   dmesg(8),
       genksyms(8), syslogd(8).  bfd info files.

4th Berkeley Distribution       March 18, 2002                     KSYMOOPS(8)

Man(1) output converted with man2html