Linux get symbols from so

Julia Evans

This week at work I spent all week trying to debug a segfault. I’d never done this before, and some of the basic things involved (get a core dump! find the line number that segfaulted!) took me a long time to figure out. So here’s a blog post explaining how to do those things!

At the end of this blog post, you should know how to go from “oh no my program is segfaulting and I have no idea what is happening” to “well I know what its stack / line number was when it segfaulted, at least!“.

what’s a segfault?

A “segmentation fault” is when your program tries to access memory that it’s not allowed to access, or tries to . This can be caused by:

• trying to dereference a null pointer (you’re not allowed to access the memory address 0 )
• trying to dereference some other pointer that isn’t in your memory
• a C++ vtable pointer that got corrupted and is pointing to the wrong place, which causes the program to try to execute some memory that isn’t executable
• some other things that I don’t understand, like I think misaligned memory accesses can also segfault

This “C++ vtable pointer” thing is what was happening to my segfaulting program. I might explain that in a future blog post because I didn’t know any C++ at the beginning of this week and this vtable lookup thing was a new way for a program to segfault that I didn’t know about.

But! This blog post isn’t about C++ bugs. Let’s talk about the basics, like, how do we even get a core dump?

step 1: run valgrind

I found the easiest way to figure out why my program is segfaulting was to use valgrind: I ran

and this gave me a stack trace of what happened. Neat!

But I also wanted to do a more in-depth investigation and find out more than just what valgrind was telling me! So I wanted to get a core dump and explore it.

How to get a core dump

A core dump is a copy of your program’s memory, and it’s useful when you’re trying to debug what went wrong with your problematic program.

When your program segfaults, the Linux kernel will sometimes write a core dump to disk. When I originally tried to get a core dump, I was pretty frustrated for a long time because – Linux wasn’t writing a core dump!! Where was my core dump.

Here’s what I ended up doing:

1. Run ulimit -c unlimited before starting my program
2. Run sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t

ulimit: set the max size of a core dump

ulimit -c sets the maximum size of a core dump. It’s often set to 0, which means that the kernel won’t write core dumps at all. It’s in kilobytes. ulimits are per process – you can see a process’s limits by running cat /proc/PID/limit

For example these are the limits for a random Firefox process on my system:

The kernel uses the soft limit (in this case, “max core file size = 0”) when deciding how big of a core file to write. You can increase the soft limit up to the hard limit using the ulimit shell builtin ( ulimit -c unlimited !)

kernel.core_pattern: where core dumps are written

kernel.core_pattern is a kernel parameter or a “sysctl setting” that controls where the Linux kernel writes core dumps to disk.

Kernel parameters are a way to set global settings on your system. You can get a list of every kernel parameter by running sysctl -a , or use sysctl kernel.core_pattern to look at the kernel.core_pattern setting specifically.

So sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t will write core dumps to /tmp/core-

If you want to know more about what these %e , %p parameters read, see man core.

It’s important to know that kernel.core_pattern is a global settings – it’s good to be a little careful about changing it because it’s possible that other systems depend on it being set a certain way.

kernel.core_pattern & Ubuntu

By default on Ubuntu systems, this is what kernel.core_pattern is set to

This caused me a lot of confusion (what is this apport thing and what is it doing with my core dumps??) so here’s what I learned about this:

• Ubuntu uses a system called “apport” to report crashes in apt packages
• Setting kernel.core_pattern=|/usr/share/apport/apport %p %s %c %d %P means that core dumps will be piped to apport
• apport has logs in /var/log/apport.log
• apport by default will ignore crashes from binaries that aren’t part of an Ubuntu packages

I ended up just overriding this Apport business and setting kernel.core_pattern to sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t because I was on a dev machine, I didn’t care whether Apport was working on not, and I didn’t feel like trying to convince Apport to give me my core dumps.

So you have a core dump. Now what?

Okay, now we know about ulimits and kernel.core_pattern and you have actually have a core dump file on disk in /tmp . Amazing! Now what. We still don’t know why the program segfaulted!

The next step is to open the core file with gdb and get a backtrace.

Getting a backtrace from gdb

You can open a core file with gdb like this:

Next, we want to know what the stack was when the program crashed. Running bt at the gdb prompt will give you a backtrace. In my case gdb hadn’t loaded symbols for the binary, so it was just like . . Luckily, loading symbols fixed it.

Here’s how to load debugging symbols.

This loads symbols from the binary and from any shared libraries the binary uses. Once I did that, gdb gave me a beautiful stack trace with line numbers when I ran bt .

If you want this to work, the binary should be compiled with debugging symbols. Having line numbers in your stack traces is extremely helpful when trying to figure out why a program crashed 🙂

look at the stack for every thread

Here’s how to get the stack for every thread in gdb!

gdb + core dumps = amazing

If you have a core dump & debugging symbols and gdb, you are in an amazing situation!! You can go up and down the call stack, print out variables, and poke around in memory to see what happened. It’s the best.

If you are still working on being a gdb wizard, you can also just print out the stack trace with bt and that’s okay 🙂

Another path to figuring out your segfault is to do one compile the program with AddressSanitizer (“ASAN”) ( $CC -fsanitize=address ) and run it. I’m not going to discuss that in this post because this is already pretty long and anyway in my case the segfault disappeared with ASAN turned on for some reason, possibly because the ASAN build used a different memory allocator (system malloc instead of tcmalloc).

I might write about ASAN more in the future if I ever get it to work 🙂

getting a stack trace from a core dump is pretty approachable!

This blog post sounds like a lot and I was pretty confused when I was doing it but really there aren’t all that many steps to getting a stack trace out of a segfaulting program:

if that doesn’t work, or if you want to have a core dump to investigate:

1. make sure the binary is compiled with debugging symbols
2. set ulimit and kernel.core_pattern correctly
3. run the program
4. open your core dump with gdb , load the symbols, and run bt
5. try to figure out what happened!!

I was able using gdb to figure out that there was a C++ vtable entry that is pointing to some corrupt memory, which was somewhat helpful and helped me feel like I understood C++ a bit better. Maybe we’ll talk more about how to use gdb to figure things out another day!

You might also like the Recurse Center, my very favorite programming community (my posts about it)

Источник

Nick Desaulniers

The enemy’s gate is down

Object Files and Symbols

Aug 13 th , 2016 | Comments

What was supposed to be one blog post about memory segmentation turned into what will be a series of posts. As the first in the series, we cover the extreme basics of object files and symbols. In follow up posts, I plan to talk about static libraries, dynamic libraries, dynamic linkage, memory segments, and finally memory usage accounting. I also cover command line tools for working with these notions, both in Linux and OSX.

A quick review of the compilation+execution pipeline (for terminology):

1. Lexing produces tokens
2. Parsing produces an abstract syntax tree
3. Analysis produces a code flow graph
4. Optimization produces a reduced code flow graph
5. Code gen produces object code
6. Linkage produces a complete executable
7. Loader instructs the OS how to start running the executable

This series will focus on part #6.

Let’s say you have some amazing C/C++ code, but for separations of concerns, you want to start moving it out into separate source files. Whereas previously in one file you had:

You now have two source files and maybe a header:

In the single source version, we would have compiled and linked that with clang main.c and had an executable file. In the multiple source version, we first compile our source files to object files, then link them altogether. That can be done separately:

We can also do the compilation and linkage in one step:

Nothing special thus far; C/C++ 101. In the first case of separate compilation and linkage steps, we were left with intermediate object files (.o). What exactly are these?

Object files are almost full executables. They contain machine code, but that code still requires a relocation step. It also contains metadata about the addresses of its variables and functions (called symbols) in an associative data structure called a symbol table. The addresses may not be the final address of the symbol in the final executable. They also contain some information for the loader and probably some other stuff.

Remember that if we fail to specify the helper object file, we’ll get an undefined symbol error.

The problem is main.o refers to some symbol called helper , but on it’s own doesn’t contain any more information about it. Let’s say we want to know what symbols an object file contains, or expects to find elsewhere. Let’s introduce our first tool, nm . nm will print the name list or symbol table for a given object or executable file. On OSX, these are prefixed with an underscore.

Let’s dissect what’s going on here. The output (as understood by man 1 nm ) is a space separated list of address, type, and symbol name. We can see that the addresses are placeholders in object files, and final in executables. The name should make sense; it’s the name of the function or variable. While I’d love to get in depth on the various symbol types and talk about sections, I don’t think I could do as great a job as Peter Van Der Linden in his book “Expert C Programming: Deep C Secrets.”

For our case, we just care about whether the symbol in a given object file is defined or not. The type U (undefined) means that this symbol is referenced or used in this object code/executable, but it’s value wasn’t defined here. When we compiled main.c alone and got the undefined symbol error, it should now make sense why we got the undefined symbol error for helper. main.o contains a symbol for main, and references helper. helper.o contains a symbol for helper, and references to puts. The final executable contains symbols for main and helper and references to puts.

You might be wondering where puts comes from then, and why didn’t we get an undefined symbol error for puts like we did earlier for helper. The answer is the C runtime. libc is implicitly dynamically linked to all executables created by the C compiler. We’ll cover dynamic linkage in a later post in this series.

When the linker performs relocation on the object files, combining them into a final executable, it goes through placeholders of addresses and fills them in. We did this manually in our post on JIT compilers.

While nm gave us a look into our symbol table, two other tools I use frequently are objdump on Linux and otool on OSX. Both of these provide disassembled assembly instructions and their addresses. Note how the symbols for functions get translated into labels of the disassembled functions, and that their address points to the first instruction in that label. Since I’ve shown objdump numerous times in previous posts, here’s otool .

readelf -s will give us a list of symbols on Linux. ELF is the file format used by the loader on Linux, while OSX uses Mach-O. Thus readelf and otool , respectively.

Also note that for static linkage, symbols need to be unique*, as they refer to memory locations to either read/write to in the case of variables or locations to jump to in the case of functions.

*: there’s a notion of weak symbols, and some special things for dynamic libraries we’ll see in a follow up post.

Languages like C++ that support function overloading (functions with the same name but different arguments, return types, namespaces, or class) must mangle their function names to make them unique.

Will produce symbols like:

Note: GNU nm on Linux distros will have a —demangle option:

On OSX, we can pipe nm into c++filt :

Finally, if you don’t have an object file, but instead a backtrace that needs demangling, you can either invoke c++filt manually or use demangler.com.

Rust also mangles its function names. For FFI or interface with C functions, other languages usually have to look for or expose symbols in a manner suited to C, the lowest common denominator. C++ has extern «C» blocks and Rust has extern blocks.

We can use strip to remove symbols from a binary. This can slim down a binary at the cost of making stack traces unreadable. If you’re following along at home, try comparing the output from your disassembler and nm before and after running strip on the executable. Luckily, you can’t strip the symbols out of object files, otherwise they’d be useless as you’d no longer be able to link them.

If we compile with the -g flag, we can create a different kind of symbol; debug symbols. Depending on your compiler+host OS, you’ll get another file you can run through nm to see an entry per symbol. You’ll get more info by using dwarfdump on this file. Debug symbols will retain source information such as filename and line number for all symbols.

This post should have been a simple refresher of some of the basics of working with C code. Finding symbols to be placed into a final executable and relocating addresses are the main job of the linker, and will be the main theme of the posts in this series. Keep your eyes out for more in this series on memory segmentation.

Источник

Linux get symbols from so

After many years programming solely on Windows I have recently started working on Linux – Ubuntu to be precise. This is the second post in a series of tutorials that will share what I have learned about handling Linux symbols. This second post will cover how to get symbols for gcc C and C++ library versions that are not installed on your machine, which can be important when investigating crashes from customers’ machines.

The previous post explained how to get symbols for the C and C++ libraries on your machine. That’s a good start, but now we’re going to go to the next level.

Future posts on this topic include:

Crash Analysis

The unfortunate reality of shipping a product to thousands or millions of people is that, despite all of your careful testing, it will crash on some users’ machines. Often this is due to some confluence of factors – different hardware, different software, different usage patterns – that was not anticipated. Sometimes it is due to bugs in the user’s hardware or other software, but no matter what the cause, if enough customers are hitting a crash you need to be able to investigate and fix it, even if the crashes are on a distribution which you don’t normally use.

If you are only interested in symbols for your install of Linux then this post has nothing to offer you – sorry.

At Valve we use google-breakpad to record crashes on Linux, Windows, and Mac OS. These crashes are uploaded to our servers where we can generate reports and then investigate the crashes, typically in order of frequency. In many cases we can fix these bugs without ever reproducing them on our machines – it’s amazing how much debugging can be done with just a call stack, with occasional reference to the raw stack memory and a list of loaded modules.

Not surprisingly this post-mortem crash analysis works best if we have symbols, both for our code and for every other shared object in our process. It is quite common for software to crash deep inside the C/C++ libraries and if we don’t have accurate function names then our investigation is unnecessarily complicated. In addition to function names, having full unwind data is also important – otherwise the stack walking may misbehave and we will be looking up incorrect addresses. This is particularly important on 64-bit platforms where there is typically no linked list of stack frames. Finally, it is best to have file name and line number information to allow zeroing in on the precise source line where a crash occurred.

Our development machines are mostly running fully patched Ubuntu 12.04 (Precise Pangolin). When we processed the libc6 symbols from our machines to breakpad format and pushed them to the crash dump analysis server this gave us C/C++ library symbols for most of our crash dumps. But not all.

Unpatched

We get a significant number of crashes from older versions of Ubuntu 12.04. The crash reports show that these customers are running an earlier kernel and a different version of the C/C++ libraries. The challenge then was how to get symbols for these older C/C++ library versions.

One obvious way to get older symbols would be to install the original version of Ubuntu 12.04 and then incrementally update it, publishing symbols along the way. This option did not make me happy. Luckily I found a better option.

Installing old symbol versions, a failed attempt

Normally when you install a package you get the latest version, but you can request a specific version. The option to do this can be seen if you run man apt-get and look in the help for install. It says:

A specific version of a package can be selected for installation by following the package name with an equals and the version of the package to select.

You can see the version of libc6 that you have installed by running dpkg -s libc6 or, alternately, running libc6 as a program, like this: /lib/i386-linux-gnu/libc.so.6

Either way I find that on my machine I am running version 2.15-0ubuntu10.3. I can verify this and the version-install syntax by running sudo apt-get install libc6=2.15-0ubuntu10.3, in which case I will be told that I already have that version. However if I make a guess as to the previous version and try installing its debug information I will hit problems:

$ sudo -E apt-get install libc6-dbg=2.15-0ubuntu10.2

…
The following packages have unmet dependencies:
libc6-dbg : Depends: libc6 (= 2.15-0ubuntu10.2) but 2.15-0ubuntu10.3 is to be installed
E: Unable to correct problems, you have held broken packages.

In short, I have version 10.3 of libc6 installed and apt-get, in order to save me from myself, refuses to install the incompatible 10.2 symbols. That’s a good thing in general, because incompatible packages can lead to serious problems, but in this case I just want to temporarily install some old symbols so the limitation is annoying.

Installation is optional

Thankfully we don’t really want to install these old versions of the symbols – we just want to extract the files so that we can make them available to breakpad or other crash-dump analysis tools. It turns out that extracting the contents of a package is far easier than installing it. These are the steps that I used:

$ cd $(mktemp -d)
$ apt-get download libc6-dbg=2.15-0ubuntu10.2
$ ar -x libc6-dbg_2.15-0ubuntu10.2_i386.deb
$ mkdir contents && cd contents
$ tar -xf ../data.tar.*z

These steps create and cd to a temporary directory, download the package, extract the archive contents, and then unpack the data.tar.*z file to the contents directory, giving us access to all of the symbol files.

Now we’re making progress. The next step is to find all the packages we might need.

Finding available versions

Probably the easiest way to find what packages are available is to find out what URL a particular debug package is downloaded from and then browse to the containing web page to see what other versions are there. The following command will get the URL for libc6-dbg:

$ apt-get download –print-uris libc6-dbg

If we browse to http://us.archive.ubuntu.com/ubuntu/pool/main/e/eglibc/ and search for libc6-dbg we will see symbols for about a dozen versions of libc6.

The same process can be repeated for libstdc++6.

$ apt-get download –print-uris libstdc++6-4.4-dbg

We can then navigate to http://us.archive.ubuntu.com/ubuntu/pool/main/g/gcc-4.4/ (and gcc-4.5 and gcc-4.6) and search on libstdc++6-4.4-dbg (and 6-4.5 and 6-4.6).

Finally, the libgcc1-dbg package (gcc support library) can be found in the same web directories as the libstdc++ symbols, with names like libgcc1-dbg_4.6.3-1ubuntu5_i386.deb.

With these four web directories and the names of the debug packages in hand we can construct a simple shell script that will download symbols for 32 different libc6, libstdc++6, and libgcc1 packages.

$ BASE=http://us.archive.ubuntu.com/ubuntu/pool/main
$ SUFFIX=i386.deb
$ wget -r -l1 -nd -nc -A “libc6*dbg*$SUFFIX” $BASE/e/eglibc
$ wget -r -l1 -nd -nc -A “libstdc*dbg*$SUFFIX” $BASE/g/gcc-4.4
$ wget -r -l1 -nd -nc -A “libstdc*dbg*$SUFFIX” $BASE/g/gcc-4.5
$ wget -r -l1 -nd -nc -A “libstdc*dbg*$SUFFIX” $BASE/g/gcc-4.6
$ wget -r -l1 -nd -nc -A “libgcc1-dbg*$SUFFIX” $BASE/g/gcc-4.4
$ wget -r -l1 -nd -nc -A “libgcc1-dbg*$SUFFIX” $BASE/g/gcc-4.5
$ wget -r -l1 -nd -nc -A “libgcc1-dbg*$SUFFIX” $BASE/g/gcc-4.6

Warning: I can’t get rid of the “smart quotes” in the script above so you’ll probably have to replace both quotes on each wget line with ‘real’ quotes. Sorry.

Depending on your needs this might be excessive or insufficient. You can download just a few packages as needed, or download every -dbg package you can find.

With slight modifications this script can download amd64.deb packages or others.

Other Ubuntu debug packages

The libc6 and libstdc++ debug packages are an anomaly in the land of Ubuntu because they can be found in the normal repositories. For the majority of debug packages we need to go to the ddebs repositories, but this is actually quite simple. Let’s say we want to get the symbols for libX11.so.6. First we execute these two commands to find out what package contains libX11.so.6, and what URL contains that package:

libx11-6: /usr/lib/i386-linux-gnu/libX11.so.6
$ apt-get download –print-uris libx11-6

Now we take that URL and replace “.*ubuntu.com/ubuntu” with http://ddebs.ubuntu.com – if you’re feeling lazy you can do this with sed:

Now navigate to the resulting web directory (ignore the file portion) and download whatever -dbgsym file seems promising. In my case I chose libx11-xcb1-dbgsym_1.4.99.1-0ubuntu2_i386.ddeb because I guessed it would match the customer’s machine.

In this case we were looking at a customer crash so we had a breakpad debug identifier of:

After fixing the byte ordering of the first 32-bit chunk and the two subsequent 16-bit chunks we are left with a partial build ID which looks like this:

I could then compare this to the result of readelf -n on the libX11.so.6 symbols that I had downloaded and I could see that they were a match:

As with all of the advice in this particular post this technique only makes sense when tracking down symbols for crashes happening on other machines. For development on your own machine you generally want to add the ddebs repository and then go:

sudo apt-get install libx11-6-dbgsym

Other Linux versions method one: guess and go

After I retrieved symbols for Ubuntu’s versions of libc6 and published them to our breakpad symbol server I was still seeing crashes with no symbols. It turned out these were from people running our code on other Linux distributions. It was time to find out how to get symbols from Linux versions that I didn’t have installed. A large chunk of crashes were on Arch Linux with libc-2.17.so on the stack so I tried that first.

A bit of web searching found a description of the official Arch Linux Repositories. From there I went to the mirrors page, which took me to the mirror status page, which gave me a list of repository mirrors. I chose a conveniently located mirror and navigated down until I found a glibc package. This wasn’t actually a debug package, but I downloaded it and hoped that it would be helpful.

This is similar to the process to unpack a .deb file except that the step to unpack the .tar.xz from the .deb file can be skipped.

Before proceeding I wanted to make sure that I had downloaded the correct libc version. Most ELF files contain a build ID which uniquely identifies the file:

Notes at offset 0x00000174 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID
Build ID: ca7f7b19456bdf0460ec93f7c3e92e19d439e96e

I compared that to breakpad’s module information for libc-2.17.so on the Arch Linux crashes:

At first glance it may seem like a mismatch, but breakpad treats build IDs as GUIDs and one consequence of this is that the first four bytes are endian-swapped, so that ca7f7b19 becomes 197B7FCA, so in fact this was the correct shared object.

This package doesn’t have full debug information – the filename and line number information is missing – but it does have symbols and unwind information. A bit more research showed that this embedded debug information was probably as good as I was going to get. It appears that, despite the efforts of some contributors, Arch Linux still doesn’t generally have debug packages. Which is a shame.

Other Linux versions method two: install ‘em

The web search method of finding symbols is unreliable, so if you really need symbols for a particular Linux version you may just have to install it. I’ve got a collection of virtual machines running various distributions of Linux. My usual pattern is to install, then use apt-get download –print-uris to find where the symbols for that version are, and then hope that a bit of exploring of the server will find other versions of the symbols. The actual downloading and unpacking can be done on my Linux development machine – in some cases I just run a couple of shell commands on the VM and then never use it again.

Fedora fun

When tracking down symbols the first thing you need to know is what version of Linux a crash is coming from. The breakpad reports include an OS field to help with this. The Ubuntu OS description is pretty clear:

Linux 0.0.0 Linux 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64

ARCH is similarly pretty obvious:

Linux 0.0.0 Linux 3.6.11-1-ARCH #1 SMP PREEMPT Tue Dec 18 08:57:15 CET 2012 x86_64

However this one had me confused for a while:

Linux 0.0.0 Linux 3.6.11-3.fc18.x86_64 #1 SMP Mon Dec 17 21:35:39 UTC 2012 x86_64

It took a bit of random searching before I realized that fc18 stood for Fedora Core version 18.

Within fc18 I knew from the crash reports that I wanted libc-2.16.so with breakpad debug identifier 99DD7BCFE74D7D4C4A0CFF6A0293D96B0. I then installed Fedora in order to try to find these symbols, but this brought some extra challenges. Fedora uses yum/rpm for package management instead of apt-get/dpkg, so all my tricks would have to be reworked. Here’s the path that I ended up taking:

I used ldd /bin/ls to find where libc-2.16.so is installed, and then rpm -qf /lib/libc-2.16.so to find what package installs libc-2.16.so – it’s glibc-2.16-24.fc18.1686.

Then I used cat /etc/yum.repos.d/* to get a list of respositories, including http://mirror.chpc.utah.edu/pub/fedora/linux/.

I wanted the latest version of libc-2.16.so so I drilled down into the updates directory, version 18, i386, and eventually found this: http://mirror.chpc.utah.edu/pub/fedora/linux/updates/18/i386/glibc-2.16-28.fc18.i686.rpm.

The .rpm packages used by Fedora are not standard archive files so I did some searching and found that rpm2cpio and cpio could be used to extract their contents, and those tools are available on Ubuntu.

At this point I could abandon my Fedora VM and resume working on my physical Ubuntu machine. So, on my Ubuntu machine I did the following dance:

$ sudo apt-get install rpm2cpio
$ wget http://mirror.chpc.utah.edu/pub/fedora/linux/updates/18/i386/glibc-2.16-28.fc18.i686.rpm
$ mkdir contents && cd contents
$ rpm2cpio ../glibc-2.16-28.fc18.i686.rpm | cpio -idmv
$ readelf -n lib/libc.so.6 | grep Build
Build ID: cf7bdd994de74c7d4a0cff6a0293d96b64681e06
$ readelf -s lib/libc.so.6 | wc -l
11005

The readelf -n step confirmed that the build ID matched the debug identifier I was looking, and the readelf -s step showed that there were enough symbols to be useful.

This system is still a bit ad-hoc as I haven’t yet found a way to directly find what URL a package comes from, but it worked relatively cleanly. It does appear that there should be debuginfo packages for glibc here but I could not find them, and running sudo yum install glibc-debuginfo-2.16-28.fc18.i686.rpm also got me nothing. I think the embedded symbols are sufficient so I am not looking further.

Using downloaded symbols

The question of what to do with all of these symbol files that have been downloaded is a topic for another post.

Summary

For reference purposes here are the new commands (not covered last time) covered in this post:

• sudo apt-get install libc6-dbg=2.15-0ubuntu10.2: installs a specific version of a package
• dpkg -s libc6: see what version of a package you have installed
• apt-get download libc6-dbg=2.15-0ubuntu10.2: downloads a package by name
• ar -x libc6-dbg_2.15-0ubuntu10.2_i386.deb: extract the contents of a package
• tar -xf data.tar.*z: extracts the contents of the data file from a package
• apt-get download –print-uris libc6-dbg: gets the download URL for a package
• cd $(mktemp -d): makes a temporary directory and sets it as the current directory
• wget $URL: download a package from a specific URL
• wget -r -l1 -nd -nc -A “libc6*dbg*$SUFFIX” $BASE/e/eglibc: download a set of files
• readelf -n usr/lib/libc-2.17.so: read the build ID from an ELF file
• readelf -s lib/libc.so.6 | wc -l: count how many symbol names are in a shared object

Here are the commands used for dealing with yum/rpm based package management tools:

Источник