Upgrading a Database Project to Python 3.12

Upgrading to major versions of Python (e.g., 3.12) can be non-trivial; here's a good article for reference. I recently upgraded API Logic Server, and offer this information in hopes it can make things a bit easier for you.

Aside: API Logic Server is open source. It creates executable API/Admin App projects from a database with 1 command. Customize with rules and Python in your IDE.

There were 2 areas that required attention:

1. Packaging: Preparing a project for pip installaccess
   • This issue was unique to Python 3.12: the old setup procedures have been removed.
2. Dependent libraries: This is a consideration for any new release. In general, I found this page helpful.

My project is database-oriented (using SQLAlchemy), so key risk areas usually involve database access. MySQL and Oracle are generally straightforward, but I always need to address Postgres (psycopg) and SQL/Server (pyodbc). These affect requirements.txt and product packaging. 

Let's consider packaging first.

Project Packaging

My project requires packaging for PyPi. This has changed in Python 3.12.

Let's go over some quick background. To make a package available for pip install, you must upload it to PyPi. Here's an uploaded example. This is 2 step process as follows:

1. Build local install files: This gathers your dependent libraries, CLI entry points, and so forth.
2. Upload to PyPi: This is unchanged: python3 -m twine upload --skip-existing dist/*. 

The first step has changed in two ways:

1. How you run the setup process
2. How you specify your dependent libraries

Run Setup (Dependencies)

This process prepares for python3 -m twine upload..., by creating local files that identify the libraries you require, CLI entry points, and so forth.

In the past, you ran python3 setup.py sdist bdist_wheel. That is no longer supported. It has been replaced by:

python3 -m build

pyproject.toml (Not setup.py)

In the past, your setup.py file identified the libraries you require, CLI entry points, and so forth. setup.py is no longer supported in Python 3.12. Instead, you must provide a pyproject.toml file, as described in this guide. The python3 -m build uses this file.

For me, this set off a mild panic: I was unable to find a setup-to-toml migration utility, except for those looking to replace the entire pip install workflow.

As it turned out, migrating setup.py was not so painful by hand; mainly a series of copy/paste procedures as shown below. Here's a working pyproject.toml shown in the diagram below.

psycopg2-binary: Postgres

This is used by SQLAlchemy for Postgres access. In addition to pyproject.toml, I had to change requirements.txt, as shown here. I changed psycopg2-binary==2.9.5 to:

psycopg2-binary==2.9.9

My project is large, so I found it convenient to create a small venv, and test the install. It took a few tries to straighten out the -binary bit. 

odbc: SQL/Server

Microsoft SQL/Server requires 3 packages (this on a Mac): 

unixodbc

Install unixobdbc. You might get:

==> Running `brew cleanup unixodbc`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /opt/homebrew/Cellar/unixodbc/2.3.11... (48 files, 2.3MB)
Warning: The following dependents of upgraded formulae are outdated but will not
be upgraded because they are not bottled:
  msodbcsql18
(venv) val@Vals-MPB-14 Desktop % 

It seemed to work.

odbc driver

I required the Microsoft odbc driver.

pyodbc

This is used by SQLAlchemy. In requirements.txt and pyproject.toml, I had to change pyodbc==4.0.34 --> pyodbc==5.0.0.

Minor Issues: Escape Characters

As noted in Python docs, mistakes in strings (e.g., \but I forgot the n) were previously not flagged; now they are.

I mention this because unexpected messages show up when you start your program under the debugger.