Python lib directory linux

Lee On Coding

My blog about coding and stuff.

How does python find packages?

I just ran into a situation where I compiled and installed Python 2.7.9 from source on Ubuntu, but Python could not find the packages I had previously installed. This naturally raises the question — how does Python know where to find packages when you call import ? This post applies specifically to Python 2.7.9, but I’m guessing Python 3x works very similarly.

In this post I first describe how Python finds packages, and then I’ll finish with the discovery I made regarding the default Python that ships with Ubuntu and how it differs from vanilla Python in how it finds packages.

sys.path

Python imports work by searching the directories listed in sys.path .

Using my default Ubuntu 14.04 Python:

So Python will find any packages that have been installed to those locations.

How sys.path gets populated

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

You can read more about sys.path in the Python docs.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

It’s a bit involved.

You can manipulate sys.path

You can manipulate sys.path during a Python session and this will change how Python finds modules. For example:

The module __file__ attribute

When you import a module, you usually can check the __file__ attribute of the module to see where the module is in your filesystem:

However, the Python docs state that:

The file attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.

So, for example this doesn’t work:

It makes sense that the sys module is statically linked to the interpreter — it is essentially part of the interpreter!

The imp module

Python exposes the entire import system through the imp module. That’s pretty cool that all of this stuff is exposed for us to abuse, if we wanted to.

imp.find_module can be used to find a module:

You can also import and arbitrary Python source as a module using imp.load_source . This is the same example before, except imports our module using imp instead of by manipulating sys.path :

Passing ‘hi’ to imp.load_source simply sets the __name__ attribute of the module.

Ubuntu Python

Now back to the issue of missing packages after installing a new version of Python compiled from source. By comparing the sys.path from both the Ubuntu Python, which resides at /usr/bin/python , and the newly installed Python, which resides at /usr/local/bin/python , I could sort things out:

Ubuntu Python ( /usr/bin/python ):

Python compiled from source ( /usr/local/bin/python )

Turns out what mattered for me was dist-packages vs. site-packages . Using Ubuntu’s Python, my packages were installed to /usr/local/lib/python2.7/dist-packages , whereas the new Python I installed expects packages to be installed to /usr/local/lib/python2.7/site-packages . I just had to manipulate the PYTHONPATH environment variable to point to dist-packages in order to gain access to the previously installed packaged with the newly installed version of Python.

How did Ubuntu manipulate the sys.path ?

So how does the Ubuntu distribution of Python know to use /usr/local/lib/python2.7/dist-packages in sys.path ? It’s hardcoded into their site module! First, find where the site module code lives:

Here is an excerpt from Ubuntu Python’s site.py , which I peeked by opening /usr/lib/python2.7/site.py in a text editor. First, a comment at the top:

For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python /dist-packages, Debian addons install into /usr//python /dist-packages. /usr/lib/python /site-packages is not used.

OK so there you have it. They explain how the Debian distribution of Python is different.

And now, for the code that implementes this change:

It’s all there, if you are crazy enough to dig this deep.

© Lee Mendelowitz – Built with Pure Theme for Pelican

Источник

2. Using Python on Unix platforms¶

2.1. Getting and installing the latest version of Python¶

2.1.1. On Linux¶

Python comes preinstalled on most Linux distributions, and is available as a package on all others. However there are certain features you might want to use that are not available on your distro’s package. You can easily compile the latest version of Python from source.

In the event that Python doesn’t come preinstalled and isn’t in the repositories as well, you can easily make packages for your own distro. Have a look at the following links:

for Debian users

for OpenSuse users

for Fedora users

for Slackware users

2.1.2. On FreeBSD and OpenBSD¶

FreeBSD users, to add the package use:

OpenBSD users, to add the package use:

For example i386 users get the 2.5.1 version of Python using:

2.1.3. On OpenSolaris¶

You can get Python from OpenCSW. Various versions of Python are available and can be installed with e.g. pkgutil -i python27 .

2.2. Building Python¶

If you want to compile CPython yourself, first thing you should do is get the source. You can download either the latest release’s source or just grab a fresh clone. (If you want to contribute patches, you will need a clone.)

The build process consists of the usual commands:

Configuration options and caveats for specific Unix platforms are extensively documented in the README.rst file in the root of the Python source tree.

make install can overwrite or masquerade the python3 binary. make altinstall is therefore recommended instead of make install since it only installs exec_prefix /bin/python version .

2.3. Python-related paths and files¶

These are subject to difference depending on local installation conventions; prefix ( $ ) and exec_prefix ( $ ) are installation-dependent and should be interpreted as for GNU software; they may be the same.

For example, on most Linux systems, the default for both is /usr .

Recommended location of the interpreter.

prefix /lib/python version , exec_prefix /lib/python version

Recommended locations of the directories containing the standard modules.

prefix /include/python version , exec_prefix /include/python version

Recommended locations of the directories containing the include files needed for developing Python extensions and embedding the interpreter.

2.4. Miscellaneous¶

To easily use Python scripts on Unix, you need to make them executable, e.g. with

and put an appropriate Shebang line at the top of the script. A good choice is usually

which searches for the Python interpreter in the whole PATH . However, some Unices may not have the env command, so you may need to hardcode /usr/bin/python3 as the interpreter path.

To use shell commands in your Python scripts, look at the subprocess module.

2.5. Custom OpenSSL¶

To use your vendor’s OpenSSL configuration and system trust store, locate the directory with openssl.cnf file or symlink in /etc . On most distribution the file is either in /etc/ssl or /etc/pki/tls . The directory should also contain a cert.pem file and/or a certs directory.

Download, build, and install OpenSSL. Make sure you use install_sw and not install . The install_sw target does not override openssl.cnf .

Build Python with custom OpenSSL (see the configure –with-openssl and –with-openssl-rpath options)

Patch releases of OpenSSL have a backwards compatible ABI. You don’t need to recompile Python to update OpenSSL. It’s sufficient to replace the custom OpenSSL installation with a newer version.

Источник

Installing Packages¶

This section covers the basics of how to install Python packages .

It’s important to note that the term “package” in this context is being used to describe a bundle of software to be installed (i.e. as a synonym for a distribution ). It does not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.

Requirements for Installing Packages¶

This section describes the steps to follow before installing other Python packages.

Ensure you can run Python from the command line¶

Before you go any further, make sure you have Python and that the expected version is available from your command line. You can check this by running:

You should get some output like Python 3.6.3 . If you do not have Python, please install the latest 3.x version from python.org or refer to the Installing Python section of the Hitchhiker’s Guide to Python.

If you’re a newcomer and you get an error like this:

It’s because this command and other suggested commands in this tutorial are intended to be run in a shell (also called a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating system’s shell and interacting with Python.

If you’re using an enhanced shell like IPython or the Jupyter notebook, you can run system commands like those in this tutorial by prefacing them with a ! character:

It’s recommended to write rather than plain python in order to ensure that commands are run in the Python installation matching the currently running notebook (which may not be the same Python installation that the python command refers to).

Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the system Python without creating a virtual environment first should replace the python command in this tutorial with python3 and the python -m pip command with python3 -m pip —user . Do not run any of the commands in this tutorial with sudo : if you get a permissions error, come back to the section on creating virtual environments, set one up, and then continue with the tutorial as written.

Ensure you can run pip from the command line¶

Additionally, you’ll need to make sure you have pip available. You can check this by running:

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you’re on Linux and installed using your OS package manager, you may have to install pip separately, see Installing pip/setuptools/wheel with Linux Package Managers .

If pip isn’t already installed, then first try to bootstrap it from the standard library:

If that still doesn’t allow you to run python -m pip :

Run python get-pip.py . 2 This will install or upgrade pip. Additionally, it will install setuptools and wheel if they’re not installed already.

Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. You can use python get-pip.py —prefix=/usr/local/ to install in /usr/local which is designed for locally-installed software.

Ensure pip, setuptools, and wheel are up to date¶

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives:

Optionally, create a virtual environment¶

See section below for details, but here’s the basic venv 3 command to use on a typical Linux system:

This will create a new virtual environment in the tutorial_env subdirectory, and configure the current shell to use it as the default python environment.

Creating Virtual Environments¶

Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally. If you are looking to safely install global command line tools, see Installing stand alone command line tools .

Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python3.6/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.

In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.

Currently, there are two common tools for creating Python virtual environments:

venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.

virtualenv needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and pip , setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).

Источник

Где хранятся модули Python?

Я недавно начал изучать Python, и у меня есть 2 вопроса, касающихся модулей.

1. Есть ли способ получить список модулей Python, доступных (т.е. установленных) на компьютере?
2. Я использую Ubuntu Karmic и Synaptic для управления пакетами. Я только что установил модуль python. Где на самом деле хранится код модуля на моей машине? (есть ли [рекомендуемое] место по умолчанию для хранения модулей)?

7 ответов

1) Есть ли способ получить список доступных модулей Python (т.е. установленных) на mchine?

Это работает для меня:

2) Где на самом деле хранится код модуля на моей машине?

Обычно в /lib/site-packages в вашей папке Python. (По крайней мере, на Windows.)

Вы можете использовать sys.path , чтобы узнать, в каких каталогах ищутся модули.

1. Вы можете перебирать каталоги, перечисленные в sys.path , чтобы найти все модули (кроме встроенных).
2. Вероятно, это будет где-то около /usr/lib/pythonX.X/site-packages (опять же, смотрите sys.path ). И подумайте об использовании собственного управления пакетами Python (через pip или easy_install , плюс yolk ) вместо этого пакеты в репозиториях, поддерживаемых дистрибутивами Linux, как правило, устарели.

Если вы используете conda или pip для установки модулей, вы можете использовать

Отобразить все модули. Это отобразит все модули в самом терминале и будет намного быстрее, чем

Вы можете найти код модуля, сначала перечислив модули:

Это выдает список модулей, которые Python может импортировать. Внизу этого списка есть фраза:

Введите любое имя модуля, чтобы получить дополнительную помощь. Или введите «модули спам» для поиска модулей, чье имя или сводка содержат строку «спам».

Чтобы найти местоположение модуля:

Здесь много информации. Прокрутите вниз, чтобы найти его местоположение

Копировать ссылку. Чтобы увидеть код после выхода из Python REPL:

В командной строке python сначала импортируйте тот модуль, для которого вам нужно местоположение.

Например, чтобы узнать местоположение «pygal»:

На машине Windows модули Python расположены по адресу (системный диск и версия Python могут отличаться):

1) Использование функции справки

Войдите в приглашение Python и введите следующую команду:

Это перечислит все модули, установленные в системе. Вам не нужно устанавливать никаких дополнительных пакетов для их перечисления, но вам нужно вручную искать или фильтровать нужный модуль из списка.

2) Использование заморозки в пипсах

Даже если вам нужно установить дополнительные пакеты, чтобы использовать это, этот метод позволяет вам легко искать или фильтровать результат с помощью команды grep . например < < Х1 >> .

Вы можете использовать любой удобный для вас метод.

Источник