3 major Python shortcomings — and their solutions

While Python has been around 30 years, only in the last few years has its popularity surged to surpass all programming languages but Java and C. On the whole, immense popularity has been great for Python: The language has become a staple for teaching and learning programming, a good place to start with software development, and a valuable part of any technology stack.

Unfortunately, popularity also tends to magnify one’s flaws. Like Python’s virtues, Python’s problems—most notably performance, packaging and application delivery, and project management—are now well-known. They’re not fatal flaws, but they are obstacles to Python adoption that will only become more pronounced with time, as competition rises from other languages like Julia, Nim, Rust, and Go.

Following are the three main challenges faced by Python programmers, and some of the ways the developers behind Python and third-party Python tools and libraries are working to solve them.

Python multithreading and speed

The problem: Python’s slow overall performance, and its limited threading and multiprocessing capabilities, remain major roadblocks to its future development.

Python has long valued ease of programming over runtime speed. This tradeoff is not the worst thing in the world when so many performance-intensive tasks can be accomplished in Python by tapping high-speed, external libraries written in C or C++, such as NumPy or Numba. Still, Python’s out-of-the-box performance is an order of magnitude behind other languages with equally straightforward syntaxes, like Nim or Julia, that compile to machine code.

One of Python’s longest-lived performance shortcomings is its poor use of multiple cores or processors. While Python does have threading facilities, they’re effectively limited to a single core. And while Python also supports multiprocessing by way of launching sub-instances of its runtime, dispatching and synchronizing the results from those subprocesses isn’t always efficient.

The solution(s): Right now, there is no single, top-down, holistic solution to Python’s performance issues. Rather, there is a patchwork of initiatives to speed up Python, each contributing improvements in a specific area. Some examples:

• Improving CPython’s internal behaviors. CPython improvements provide modest but universal speed-ups. For instance, Python 3.8’s Vectorcall protocol provides a faster calling convention for Python objects. While these improvements aren’t dramatic, they’re measurable and predictable, they don’t break backward compatibility, and they benefit existing Python apps without rewriting.
• Improving CPython’s subinterpreter functionality. A new programmatic interface to instances of Python’s interpreter may in time allow more elegant sharing of data between interpreters for multi-core processing. Right now the proposal is set to be adopted in Python 3.9, so the relevant bits won’t even show up for at least one more Python version.
• Improving object sharing among multiple processes. Multiprocessing in Python starts a new interpreter instance for each core for maximum performance, but many of those performance gains are lost when the interpreters try to operate on the same memory object. New functionality like the SharedMemory class and the new pickle protocol can reduce the amount of copying or serialization needed to pass data between interpreters, a source of performance issues.

Projects outside of Python itself also offer ways to improve performance, but again only in specific ways:

• PyPy. An alternative Python interpreter, PyPy just-in-time-compiles Python to native machine code. It’s long worked best with pure-Python projects, but it now works reasonably well with popular binary-dependent libraries like NumPy. Best for long-running services rather than one-off applications, however.
• Cython. Cython lets you incrementally translate Python code into C. The project was originally devised for scientific and numerical computing, but it can be used in most any scenario. Biggest downside is the syntax, which is unique to Cython and makes the conversion something of a one-way process. Best for “hot spot” portions of code that benefit best from optimizing rather than whole programs.
• Numba. Numba just-in-time-compiles Python to machine code for selected functions. Like Cython, Numba is mainly for scientific computing. It works best for code that is run in-place rather than redistributed.
• Mypyc. Mypyc is a work-in-progress that generates C from Python code decorated with mypy type annotations. Mypyc is promising, since it uses Python’s native typing, but it is still some distance from being useful in production.
• Optimized Python distributions. Some third-party versions of Python, like Intel’s Python distribution, have specially compiled versions of math and stats libraries that take advantage of Intel processor extensions (e.g., AVX512). Note that while they can speed up specific math functions dramatically, they don’t provide across-the-board speed boosts.

Longtime Python programmers also talk about getting rid of the Global Interpreter Lock (GIL), which serializes access to objects and ensures threads don’t stomp on each others’ workloads. In theory, ditching the GIL could improve performance. But a GIL-less Python is, by and large, a backward-incompatible Python (especially with Python C extensions). Thus far, all attempts to remove the GIL have either reached dead ends or slowed down Python instead of speeding it up.

One ongoing Python initiative that should unlock many speed improvements is the refactoring of the implementation of Python’s internal C APIs. A less messy API set is intended to make it easier to add many kinds of performance improvements in the long run: a reworked or eliminated GIL, hooks for powerful just-in-time compilations, better ways to federate data between interpreter instances, and so on.

Python packaging and standalone executables

The problem: Even after 30 years, Python still has no good way to take a program or script, turn it into a self-contained executable, and deploy it across multiple platforms. There are ways to do that, but they’re mainly third-party tools that aren’t a native part of Python, and they can be difficult to work with.

The best-known and best-supported of these tools, PyInstaller, is still quite useful. PyInstaller can package up apps that use many of the most popular third-party extensions, like NumPy. But PyInstaller must be kept in sync manually with those third-party extensions, a tough job in an ecosystem as sprawling as Python’s. Also, PyInstaller generates outsized app packages, as it bundles everything asked for in the program’s import statements and doesn’t determine which components are actually used at runtime. Nor can PyInstaller package programs cross-platform; you have to create the package on the platform you’re deploying for.

The solution: With all due respect to PyInstaller, what we need is a solution that is native to Python—either in the standard library or as a built-in—and that allows us to package and deliver Python apps efficiently as standalone binaries for the most common platforms. Ideally, this built-in packager would use runtime code-coverage information to bundle only the libraries needed (instead of everything asked for) and would be kept in sync with the rest of Python automatically.

Right now, nothing like that exists. But hints about how to build such a tool keep popping up. The PyOxidizer project uses the Rust language to generate binaries that embed Python, as one way to create standalone Python apps. It’s a long way from being a full-blown solution for app delivery, but it suggests how influences from outside the Python ecosystem could lead to a solution.

Python installation, package management, and project management

The problem: You know what’s far too complicated? Setting up a workspace for a pro-level Python project, with a directory structure and scaffolding; and managing the environment, packages, and dependencies associated with that project; and redistributing the source for the project in a reproducible way; and not ending up eating your keyboard while trying to do all of the above over and over.

One thing the Rust and Go languages set out to do from the beginning was provide a single, canonical way to set up a project and manage it throughout its lifecycle. What Rust and Go developers lost in flexibility, they won back in consistency, predictability, and manageability. 

Python’s tools and methods for managing installs, packages, and projects accrued over time, a legacy of its 30-year lifespan. It’s a crazy quilt. There’s pip for package management, venv/virtualenv for creating virtual environments, virtualenvwrapper and Pipenv to meta-manage both of those things, pip-tools for generating project dependencies, distutils and setuptools for creating Python code distributions, and so on. Then there’s the welter of other parts needed to define a project: setup.py, requirements.txt, setup.cfg, MANIFEST.in, Pipfile … the list goes on.

The solution: Again, what’s needed is one tool and process to supersede this hodgepodge, offered as a canonical solution by the core Python development team, and capable of elegantly migrating projects that use any of the existing methods. It’s a tough bill to fill, to be sure, but the more popular Python becomes, the more crucial it is to lower barriers of entry and maintenance, and to provide consistent and approachable tooling.

A few steps have already been taken in this direction. Build dependencies for Python are being consolidated into the pyproject.toml file format as per PEP 518. And third-party tools like poetry inch closer to an all-in-one management offering, even if they merely wrap existing tools. In time, one of those solutions may catch on with the community at large, and become a de facto standard or even a candidate for an actual accepted standard.

—

Read more about Python:

• The best new features in Python 3.8
• How to get started with Python
• 24 Python libraries for every Python developer
• 13 Python web frameworks compared
• 5 essential Python tools for data science—now improved
• Virtualenv and venv: Python virtual environments explained
• Python virtualenv and venv do’s and don’ts
• What is Cython? Python at the speed of C
• What is PyPy? Faster Python without pain
• Why you should use Python for machine learning