Linux find python packages

Lee On Coding

My blog about coding and stuff.

How does python find packages?

I just ran into a situation where I compiled and installed Python 2.7.9 from source on Ubuntu, but Python could not find the packages I had previously installed. This naturally raises the question — how does Python know where to find packages when you call import ? This post applies specifically to Python 2.7.9, but I’m guessing Python 3x works very similarly.

In this post I first describe how Python finds packages, and then I’ll finish with the discovery I made regarding the default Python that ships with Ubuntu and how it differs from vanilla Python in how it finds packages.

sys.path

Python imports work by searching the directories listed in sys.path .

Using my default Ubuntu 14.04 Python:

So Python will find any packages that have been installed to those locations.

How sys.path gets populated

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

You can read more about sys.path in the Python docs.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

It’s a bit involved.

You can manipulate sys.path

You can manipulate sys.path during a Python session and this will change how Python finds modules. For example:

The module __file__ attribute

When you import a module, you usually can check the __file__ attribute of the module to see where the module is in your filesystem:

However, the Python docs state that:

The file attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.

So, for example this doesn’t work:

It makes sense that the sys module is statically linked to the interpreter — it is essentially part of the interpreter!

The imp module

Python exposes the entire import system through the imp module. That’s pretty cool that all of this stuff is exposed for us to abuse, if we wanted to.

imp.find_module can be used to find a module:

You can also import and arbitrary Python source as a module using imp.load_source . This is the same example before, except imports our module using imp instead of by manipulating sys.path :

Passing ‘hi’ to imp.load_source simply sets the __name__ attribute of the module.

Ubuntu Python

Now back to the issue of missing packages after installing a new version of Python compiled from source. By comparing the sys.path from both the Ubuntu Python, which resides at /usr/bin/python , and the newly installed Python, which resides at /usr/local/bin/python , I could sort things out:

Ubuntu Python ( /usr/bin/python ):

Python compiled from source ( /usr/local/bin/python )

Turns out what mattered for me was dist-packages vs. site-packages . Using Ubuntu’s Python, my packages were installed to /usr/local/lib/python2.7/dist-packages , whereas the new Python I installed expects packages to be installed to /usr/local/lib/python2.7/site-packages . I just had to manipulate the PYTHONPATH environment variable to point to dist-packages in order to gain access to the previously installed packaged with the newly installed version of Python.

How did Ubuntu manipulate the sys.path ?

So how does the Ubuntu distribution of Python know to use /usr/local/lib/python2.7/dist-packages in sys.path ? It’s hardcoded into their site module! First, find where the site module code lives:

Here is an excerpt from Ubuntu Python’s site.py , which I peeked by opening /usr/lib/python2.7/site.py in a text editor. First, a comment at the top:

For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python /dist-packages, Debian addons install into /usr//python /dist-packages. /usr/lib/python /site-packages is not used.

OK so there you have it. They explain how the Debian distribution of Python is different.

And now, for the code that implementes this change:

It’s all there, if you are crazy enough to dig this deep.

© Lee Mendelowitz – Built with Pure Theme for Pelican

Источник

Installing Packages¶

This section covers the basics of how to install Python packages .

It’s important to note that the term “package” in this context is being used to describe a bundle of software to be installed (i.e. as a synonym for a distribution ). It does not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.

Requirements for Installing Packages¶

This section describes the steps to follow before installing other Python packages.

Ensure you can run Python from the command line¶

Before you go any further, make sure you have Python and that the expected version is available from your command line. You can check this by running:

You should get some output like Python 3.6.3 . If you do not have Python, please install the latest 3.x version from python.org or refer to the Installing Python section of the Hitchhiker’s Guide to Python.

If you’re a newcomer and you get an error like this:

It’s because this command and other suggested commands in this tutorial are intended to be run in a shell (also called a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating system’s shell and interacting with Python.

If you’re using an enhanced shell like IPython or the Jupyter notebook, you can run system commands like those in this tutorial by prefacing them with a ! character:

It’s recommended to write rather than plain python in order to ensure that commands are run in the Python installation matching the currently running notebook (which may not be the same Python installation that the python command refers to).

Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the system Python without creating a virtual environment first should replace the python command in this tutorial with python3 and the python -m pip command with python3 -m pip —user . Do not run any of the commands in this tutorial with sudo : if you get a permissions error, come back to the section on creating virtual environments, set one up, and then continue with the tutorial as written.

Ensure you can run pip from the command line¶

Additionally, you’ll need to make sure you have pip available. You can check this by running:

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you’re on Linux and installed using your OS package manager, you may have to install pip separately, see Installing pip/setuptools/wheel with Linux Package Managers .

If pip isn’t already installed, then first try to bootstrap it from the standard library:

If that still doesn’t allow you to run python -m pip :

Run python get-pip.py . 2 This will install or upgrade pip. Additionally, it will install setuptools and wheel if they’re not installed already.

Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. You can use python get-pip.py —prefix=/usr/local/ to install in /usr/local which is designed for locally-installed software.

Ensure pip, setuptools, and wheel are up to date¶

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives:

Optionally, create a virtual environment¶

See section below for details, but here’s the basic venv 3 command to use on a typical Linux system:

This will create a new virtual environment in the tutorial_env subdirectory, and configure the current shell to use it as the default python environment.

Creating Virtual Environments¶

Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally. If you are looking to safely install global command line tools, see Installing stand alone command line tools .

Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python3.6/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.

In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.

Currently, there are two common tools for creating Python virtual environments:

venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.

virtualenv needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and pip , setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).

Источник

How to List Installed Python Packages

The Pip, Pipenv, Anaconda Navigator, and Conda Package Managers can all be used to list installed Python packages.

You can also use the ActiveState Platform’s command line interface (CLI), the State Tool to list all installed packages using a simple “state packages” command. For a complete list of all packages and dependencies (including OS-level and transitive dependencies, as well as shared libraries), you can use the Web GUI, which provides a full Bill of Materials view. Give it a try by signing up for a free ActiveState Platform account .

Before getting a list of installed packages, it’s always a good practice to ensure that up-to-date versions of Python, Pip, Anaconda Navigator and Conda are in place.

List Installed Packages with Pip

Both pip list and pip freeze will generate a list of installed packages, just with differently formatted results. Keep in mind that pip list will list ALL installed packages (regardless of how they were installed). while pip freeze will list only everything installed by Pip.

List Packages in a Console with Pip

To list all installed packages from a Python console using pip, you can utilize the following script:

List Modules in a Console without Pip

To list all installed modules from a python console without pip, you can use the following command:

Note that there are some drawbacks to this approach, including:

• If there are a lot of installed packages, this method can take a long time to import each module before it can search that module’s path for sub-modules.
• Modules that have code outside of an if __name__ == “__main__”: code block, and if user input is expected, may cause the code to enter an infinite loop or hang.

List Installed Packages with Pipenv

The pipenv lock -r command can be used to generate output from a pipfile.lock file in a pipenv environment. All packages, including dependencies will be listed in the output. For example:

List Installed Packages with Anaconda Navigator

To list installed packages in an Anaconda environment using Anaconda Navigator, do the following:

• Start the Anaconda Navigator application.
• Select Environments in the left column.
• A dropdown box at the center-top of the GUI should list installed packages. If not, then select Installed in the dropdown menu to list all packages.

List Installed Packages with Conda

The conda list command can be used to list all packages in a conda environment:

Globally vs Locally Installed Packages

For information about generating a list of installed packages globally vs locally, refer to:

List Installed Packages with the ActiveState Platform

To view a list of installed Python packages in your currently active project using the ActiveState Platform, run the following command on the command line:

The output is a full list of installed packages in your current project:

You can also obtain a complete software bill of materials view of all packages, dependencies, transitives dependencies (ie., dependencies of dependencies), OS-level dependencies and shared libraries (ie., OpenSSL) using the ActiveState Platform’s Web GUI:

The ActiveState Platform automatically builds all Python packages including linked C libraries from source code, and packages them for Windows, Linux and macOS. Because it does it all server-side, there’s no need to maintain local build environments.

Источник

Where are the python modules stored?

I have recently started learning Python and I have 2 questions relating to modules.

1. Is there a way to obtain a list of Python modules available (i.e. installed) on a machine?
2. I am using Ubuntu Karmic and Synaptic for package management. I have just installed a python module.Where is the module code actually stored on my machine? (is there a default [recommended] location that modules are stored)?

8 Answers 8

1. Is there a way to obtain a list of Python modules available (i.e. installed) on a machine?

This works for me:

1. Where is the module code actually stored on my machine?

Usually in /lib/site-packages in your Python folder. (At least, on Windows.)

You can use sys.path to find out what directories are searched for modules.

On python command line, first import that module for which you need location.

For example to find out «pygal» location:

On Windows machine python modules are located at (system drive and python version may vary):

You can find module code by first listing the modules:

This spits out a list of modules Python can import. At the bottom of this list is a phrase:

Enter any module name to get more help. Or, type «modules spam» to search for modules whose name or summary contain the string «spam».

To find module location:

A lot of information here. Scroll to the bottom to find its location

Copy link. To see code, after exiting Python REPL:

1. You can iterate through directories listed in sys.path to find all modules (except builtin ones).
2. It’ll probably be somewhere around /usr/lib/pythonX.X/site-packages (again, see sys.path ). And consider using native Python package management (via pip or easy_install , plus yolk ) instead, packages in Linux distros-maintained repositories tend to be outdated.

1) Using the help function

Get into the python prompt and type the following command:

This will list all the modules installed in the system. You don’t need to install any additional packages to list them, but you need to manually search or filter the required module from the list.

2) Using pip freeze

Even though you need to install additional packages to use this, this method allows you to easily search or filter the result with grep command. e.g. pip freeze | grep feed .

You can use whichever method is convenient for you.

Источник