Linux python dist packages

Lee On Coding

My blog about coding and stuff.

How does python find packages?

I just ran into a situation where I compiled and installed Python 2.7.9 from source on Ubuntu, but Python could not find the packages I had previously installed. This naturally raises the question — how does Python know where to find packages when you call import ? This post applies specifically to Python 2.7.9, but I’m guessing Python 3x works very similarly.

In this post I first describe how Python finds packages, and then I’ll finish with the discovery I made regarding the default Python that ships with Ubuntu and how it differs from vanilla Python in how it finds packages.

sys.path

Python imports work by searching the directories listed in sys.path .

Using my default Ubuntu 14.04 Python:

So Python will find any packages that have been installed to those locations.

How sys.path gets populated

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

You can read more about sys.path in the Python docs.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

It’s a bit involved.

You can manipulate sys.path

You can manipulate sys.path during a Python session and this will change how Python finds modules. For example:

The module __file__ attribute

When you import a module, you usually can check the __file__ attribute of the module to see where the module is in your filesystem:

However, the Python docs state that:

The file attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.

So, for example this doesn’t work:

It makes sense that the sys module is statically linked to the interpreter — it is essentially part of the interpreter!

The imp module

Python exposes the entire import system through the imp module. That’s pretty cool that all of this stuff is exposed for us to abuse, if we wanted to.

imp.find_module can be used to find a module:

You can also import and arbitrary Python source as a module using imp.load_source . This is the same example before, except imports our module using imp instead of by manipulating sys.path :

Passing ‘hi’ to imp.load_source simply sets the __name__ attribute of the module.

Ubuntu Python

Now back to the issue of missing packages after installing a new version of Python compiled from source. By comparing the sys.path from both the Ubuntu Python, which resides at /usr/bin/python , and the newly installed Python, which resides at /usr/local/bin/python , I could sort things out:

Ubuntu Python ( /usr/bin/python ):

Python compiled from source ( /usr/local/bin/python )

Turns out what mattered for me was dist-packages vs. site-packages . Using Ubuntu’s Python, my packages were installed to /usr/local/lib/python2.7/dist-packages , whereas the new Python I installed expects packages to be installed to /usr/local/lib/python2.7/site-packages . I just had to manipulate the PYTHONPATH environment variable to point to dist-packages in order to gain access to the previously installed packaged with the newly installed version of Python.

How did Ubuntu manipulate the sys.path ?

So how does the Ubuntu distribution of Python know to use /usr/local/lib/python2.7/dist-packages in sys.path ? It’s hardcoded into their site module! First, find where the site module code lives:

Here is an excerpt from Ubuntu Python’s site.py , which I peeked by opening /usr/lib/python2.7/site.py in a text editor. First, a comment at the top:

For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python /dist-packages, Debian addons install into /usr//python /dist-packages. /usr/lib/python /site-packages is not used.

OK so there you have it. They explain how the Debian distribution of Python is different.

And now, for the code that implementes this change:

It’s all there, if you are crazy enough to dig this deep.

© Lee Mendelowitz – Built with Pure Theme for Pelican

Источник

Installing Packages¶

This section covers the basics of how to install Python packages .

It’s important to note that the term “package” in this context is being used to describe a bundle of software to be installed (i.e. as a synonym for a distribution ). It does not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.

Requirements for Installing Packages¶

This section describes the steps to follow before installing other Python packages.

Ensure you can run Python from the command line¶

Before you go any further, make sure you have Python and that the expected version is available from your command line. You can check this by running:

You should get some output like Python 3.6.3 . If you do not have Python, please install the latest 3.x version from python.org or refer to the Installing Python section of the Hitchhiker’s Guide to Python.

If you’re a newcomer and you get an error like this:

It’s because this command and other suggested commands in this tutorial are intended to be run in a shell (also called a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating system’s shell and interacting with Python.

If you’re using an enhanced shell like IPython or the Jupyter notebook, you can run system commands like those in this tutorial by prefacing them with a ! character:

It’s recommended to write rather than plain python in order to ensure that commands are run in the Python installation matching the currently running notebook (which may not be the same Python installation that the python command refers to).

Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the system Python without creating a virtual environment first should replace the python command in this tutorial with python3 and the python -m pip command with python3 -m pip —user . Do not run any of the commands in this tutorial with sudo : if you get a permissions error, come back to the section on creating virtual environments, set one up, and then continue with the tutorial as written.

Ensure you can run pip from the command line¶

Additionally, you’ll need to make sure you have pip available. You can check this by running:

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you’re on Linux and installed using your OS package manager, you may have to install pip separately, see Installing pip/setuptools/wheel with Linux Package Managers .

If pip isn’t already installed, then first try to bootstrap it from the standard library:

If that still doesn’t allow you to run python -m pip :

Run python get-pip.py . 2 This will install or upgrade pip. Additionally, it will install setuptools and wheel if they’re not installed already.

Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. You can use python get-pip.py —prefix=/usr/local/ to install in /usr/local which is designed for locally-installed software.

Ensure pip, setuptools, and wheel are up to date¶

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives:

Optionally, create a virtual environment¶

See section below for details, but here’s the basic venv 3 command to use on a typical Linux system:

This will create a new virtual environment in the tutorial_env subdirectory, and configure the current shell to use it as the default python environment.

Creating Virtual Environments¶

Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally. If you are looking to safely install global command line tools, see Installing stand alone command line tools .

Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python3.6/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.

In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.

Currently, there are two common tools for creating Python virtual environments:

venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.

virtualenv needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and pip , setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).

Источник

Packaging and distributing projects¶

This section covers the basics of how to configure, package and distribute your own Python projects. It assumes that you are already familiar with the contents of the Installing Packages page.

The section does not aim to cover best practices for Python project development as a whole. For example, it does not provide guidance or tool recommendations for version control, documentation, or testing.

For more reference material, see Building and Distributing Packages in the setuptools docs, but note that some advisory content there may be outdated. In the event of conflicts, prefer the advice in the Python Packaging User Guide.

Requirements for packaging and distributing¶

First, make sure you have already fulfilled the requirements for installing packages .

You’ll need this to upload your project distributions to PyPI (see below ).

Configuring your project¶

Initial files¶

setup.py¶

The most important file is setup.py which exists at the root of your project directory. For an example, see the setup.py in the PyPA sample project.

setup.py serves two primary functions:

It’s the file where various aspects of your project are configured. The primary feature of setup.py is that it contains a global setup() function. The keyword arguments to this function are how specific details of your project are defined. The most relevant arguments are explained in the section below .

It’s the command line interface for running various commands that relate to packaging tasks. To get a listing of available commands, run python setup.py —help-commands .

setup.cfg¶

setup.cfg is an ini file that contains option defaults for setup.py commands. For an example, see the setup.cfg in the PyPA sample project.

README.rst / README.md¶

All projects should contain a readme file that covers the goal of the project. The most common format is reStructuredText with an “rst” extension, although this is not a requirement; multiple variants of Markdown are supported as well (look at setup() ’s long_description_content_type argument).

Projects using setuptools 0.6.27+ have standard readme files ( README.rst , README.txt , or README ) included in source distributions by default. The built-in distutils library adopts this behavior beginning in Python 3.7. Additionally, setuptools 36.4.0+ will include a README.md if found. If you are using setuptools, you don’t need to list your readme file in MANIFEST.in . Otherwise, include it to be explicit.

MANIFEST.in¶

A MANIFEST.in is needed when you need to package additional files that are not automatically included in a source distribution. For details on writing a MANIFEST.in file, including a list of what’s included by default, see “ Including files in source distributions with MANIFEST.in ”.

For an example, see the MANIFEST.in from the PyPA sample project.

MANIFEST.in does not affect binary distributions such as wheels.

LICENSE.txt¶

Every package should include a license file detailing the terms of distribution. In many jurisdictions, packages without an explicit license can not be legally used or distributed by anyone other than the copyright holder. If you’re unsure which license to choose, you can use resources such as GitHub’s Choose a License or consult a lawyer.

For an example, see the LICENSE.txt from the PyPA sample project.

Although it’s not required, the most common practice is to include your Python modules and packages under a single top-level package that has the same name as your project, or something very close.

For an example, see the sample package that’s included in the PyPA sample project.

setup() args¶

As mentioned above, the primary feature of setup.py is that it contains a global setup() function. The keyword arguments to this function are how specific details of your project are defined.

The most relevant arguments are explained below. Most of the snippets given are taken from the setup.py contained in the PyPA sample project.

This is the name of your project, determining how your project is listed on PyPI . Per PEP 508, valid project names must:

Consist only of ASCII letters, digits, underscores ( _ ), hyphens ( — ), and/or periods ( . ), and

Start & end with an ASCII letter or digit.

Comparison of project names is case insensitive and treats arbitrarily-long runs of underscores, hyphens, and/or periods as equal. For example, if you register a project named cool-stuff , users will be able to download it or declare a dependency on it using any of the following spellings:

version ¶

This is the current version of your project, allowing your users to determine whether or not they have the latest version, and to indicate which specific versions they’ve tested their own software against.

Versions are displayed on PyPI for each release if you publish your project.

See Choosing a versioning scheme for more information on ways to use versions to convey compatibility information to your users.

If the project code itself needs run-time access to the version, the simplest way is to keep the version in both setup.py and your code. If you’d rather not duplicate the value, there are a few ways to manage this. See the “ Single-sourcing the package version ” Advanced Topics section.

description ¶

Give a short and long description for your project.

These values will be displayed on PyPI if you publish your project. On pypi.org , the user interface displays description in the grey banner and long_description in the section named “Project Description”.

description is also displayed in lists of projects. For example, it’s visible in the search results pages such as https://pypi.org/search/?q=jupyter, the front-page lists of trending projects and new releases, and the list of projects you maintain within your account profile (such as https://pypi.org/user/jaraco/).

A content type can be specified with the long_description_content_type argument, which can be one of text/plain , text/x-rst , or text/markdown , corresponding to no formatting, reStructuredText (reST), and the Github-flavored Markdown dialect of Markdown respectively.

Give a homepage URL for your project.

author ¶

Provide details about the author.

license ¶

The license argument doesn’t have to indicate the license under which your package is being released, although you may optionally do so if you want. If you’re using a standard, well-known license, then your main indication can and should be via the classifiers argument. Classifiers exist for all major open-source licenses.

The license argument is more typically used to indicate differences from well-known licenses, or to include your own, unique license. As a general rule, it’s a good idea to use a standard, well-known license, both to avoid confusion and because some organizations avoid software whose license is unapproved.

classifiers ¶

Provide a list of classifiers that categorize your project. For a full listing, see https://pypi.org/classifiers/.

Although the list of classifiers is often used to declare what Python versions a project supports, this information is only used for searching & browsing projects on PyPI, not for installing projects. To actually restrict what Python versions a project can be installed on, use the python_requires argument.

keywords ¶

List keywords that describe your project.

project_urls ¶

List additional relevant URLs about your project. This is the place to link to bug trackers, source repositories, or where to support package development. The string of the key is the exact text that will be displayed on PyPI.

packages ¶

Set packages to a list of all packages in your project, including their subpackages, sub-subpackages, etc. Although the packages can be listed manually, setuptools.find_packages() finds them automatically. Use the include keyword argument to find only the given packages. Use the exclude keyword argument to omit packages that are not intended to be released and installed.

py_modules ¶

If your project contains any single-file Python modules that aren’t part of a package, set py_modules to a list of the names of the modules (minus the .py extension) in order to make setuptools aware of them.

install_requires ¶

“install_requires” should be used to specify what dependencies a project minimally needs to run. When the project is installed by pip , this is the specification that is used to install its dependencies.

python_requires ¶

If your project only runs on certain Python versions, setting the python_requires argument to the appropriate PEP 440 version specifier string will prevent pip from installing the project on other Python versions. For example, if your package is for Python 3+ only, write:

If your package is for Python 2.6, 2.7, and all versions of Python 3 starting with 3.3, write:

Support for this feature is relatively recent. Your project’s source distributions and wheels (see Packaging your project ) must be built using at least version 24.2.0 of setuptools in order for the python_requires argument to be recognized and the appropriate metadata generated.

In addition, only versions 9.0.0 and higher of pip recognize the python_requires metadata. Users with earlier versions of pip will be able to download & install projects on any Python version regardless of the projects’ python_requires values.

package_data ¶

Often, additional files need to be installed into a package . These files are often data that’s closely related to the package’s implementation, or text files containing documentation that might be of interest to programmers using the package. These files are called “package data”.

The value must be a mapping from package name to a list of relative path names that should be copied into the package. The paths are interpreted as relative to the directory containing the package.

data_files ¶

Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages . The data_files directive allows you to do that. It is mostly useful if you need to install files which are used by other programs, which may be unaware of Python packages.

Each (directory, files) pair in the sequence specifies the installation directory and the files to install there. The directory must be a relative path (although this may change in the future, see wheel Issue #92), and it is interpreted relative to the installation prefix (Python’s sys.prefix for a default installation; site.USER_BASE for a user installation). Each file name in files is interpreted relative to the setup.py script at the top of the project source distribution.

For more information see the distutils section on Installing Additional Files .

When installing packages as egg, data_files is not supported. So, if your project uses setuptools , you must use pip to install it. Alternatively, if you must use python setup.py , then you need to pass the —old-and-unmanageable option.

scripts ¶

Although setup() supports a scripts keyword for pointing to pre-made scripts to install, the recommended approach to achieve cross-platform compatibility is to use console_scripts entry points (see below).

entry_points ¶

Use this keyword to specify any plugins that your project provides for any named entry points that may be defined by your project or others that you depend on.

For more information, see the section on Advertising Behavior from the setuptools docs.

The most commonly used entry point is “console_scripts” (see below).

console_scripts ¶

Use console_script entry points to register your script interfaces. You can then let the toolchain handle the work of turning these interfaces into actual scripts 2. The scripts will be generated during the install of your distribution .

Choosing a versioning scheme¶

Standards compliance for interoperability¶

Different Python projects may use different versioning schemes based on the needs of that particular project, but all of them are required to comply with the flexible public version scheme specified in PEP 440 in order to be supported in tools and libraries like pip and setuptools .

Here are some examples of compliant version numbers:

To further accommodate historical variations in approaches to version numbering, PEP 440 also defines a comprehensive technique for version normalisation that maps variant spellings of different version numbers to a standardised canonical form.

Scheme choices¶

Semantic versioning (preferred)¶

For new projects, the recommended versioning scheme is based on Semantic Versioning, but adopts a different approach to handling pre-releases and build metadata.

The essence of semantic versioning is a 3-part MAJOR.MINOR.MAINTENANCE numbering scheme, where the project author increments:

MAJOR version when they make incompatible API changes,

MINOR version when they add functionality in a backwards-compatible manner, and

MAINTENANCE version when they make backwards-compatible bug fixes.

Adopting this approach as a project author allows users to make use of “compatible release” specifiers, where name

= X.Y requires at least release X.Y, but also allows any later release with a matching MAJOR version.

Python projects adopting semantic versioning should abide by clauses 1-8 of the Semantic Versioning 2.0.0 specification.

Date based versioning¶

Semantic versioning is not a suitable choice for all projects, such as those with a regular time based release cadence and a deprecation process that provides warnings for a number of releases prior to removal of a feature.

A key advantage of date based versioning is that it is straightforward to tell how old the base feature set of a particular release is given just the version number.

Version numbers for date based projects typically take the form of YEAR.MONTH (for example, 12.04 , 15.10 ).

Serial versioning¶

This is the simplest possible versioning scheme, and consists of a single number which is incremented every release.

While serial versioning is very easy to manage as a developer, it is the hardest to track as an end user, as serial version numbers convey little or no information regarding API backwards compatibility.

Hybrid schemes¶

Combinations of the above schemes are possible. For example, a project may combine date based versioning with serial versioning to create a YEAR.SERIAL numbering scheme that readily conveys the approximate age of a release, but doesn’t otherwise commit to a particular release cadence within the year.

Pre-release versioning¶

Regardless of the base versioning scheme, pre-releases for a given final release may be published as:

zero or more dev releases (denoted with a “.devN” suffix)

zero or more alpha releases (denoted with a “.aN” suffix)

zero or more beta releases (denoted with a “.bN” suffix)

zero or more release candidates (denoted with a “.rcN” suffix)

pip and other modern Python package installers ignore pre-releases by default when deciding which versions of dependencies to install.

Local version identifiers¶

Public version identifiers are designed to support distribution via PyPI . Python’s software distribution tools also support the notion of a local version identifier, which can be used to identify local development builds not intended for publication, or modified variants of a release maintained by a redistributor.

A local version identifier takes the form

version identifier>+ version label> . For example:

Working in “development mode”¶

You can install a project in “editable” or “develop” mode while you’re working on it. When installed as editable, a project can be edited in-place without reinstallation: changes to Python source files in projects installed as editable will be reflected the next time an interpreter process is started.

To install a Python package in “editable”/”development” mode Change directory to the root of the project directory and run:

The pip command-line flag -e is short for —editable , and . refers to the current working directory, so together, it means to install the current directory (i.e. your project) in editable mode. This will also install any dependencies declared with install_requires and any scripts declared with console_scripts . Dependencies will be installed in the usual, non-editable mode.

You may want to install some of your dependencies in editable mode as well. For example, supposing your project requires “foo” and “bar”, but you want “bar” installed from VCS in editable mode, then you could construct a requirements file like so:

The first line says to install your project and any dependencies. The second line overrides the “bar” dependency, such that it’s fulfilled from VCS, not PyPI.

If, however, you want “bar” installed from a local directory in editable mode, the requirements file should look like this, with the local paths at the top of the file:

Otherwise, the dependency will be fulfilled from PyPI, due to the installation order of the requirements file. For more on requirements files, see the Requirements File section in the pip docs. For more on VCS installs, see the VCS Support section of the pip docs.

Lastly, if you don’t want to install any dependencies at all, you can run:

For more information, see the Development Mode section of the setuptools docs .

Packaging your project¶

To have your project installable from a Package Index like PyPI , you’ll need to create a Distribution (aka “ Package ”) for your project.

Before you can build wheels and sdists for your project, you’ll need to install the build package:

Источник