Operations#
This document explains the processes to keep our shipping process up and running.
Versioning#
Important
Only repository administrators can execute this process.
To create a new release version:
pkgmt version
This will ask us to confirm the version number and then proceed to tag the commit and push it.
pkgmt version
will also run a few checks before creating a tagged commit:
Check there are no pending deprecations (e.g., if we said we’d remove function
do_something
in version1.0
, such function should not appear in a release tagged as1.0
)Check that
.. versionchanged::
and.. versionadded::
are correct (they point to either the current release or previous releases)If making a new minor release, check that there are no
[API Change]
changes in the CHANGELOG
Releasing#
Important
Only repository administrators can execute this process.
Note
In some projects, we’ve automated the release process, to learn more, see Continuous Delivery.
To upload a package to PyPI, execute:
pkgmt release VERSION
Where VERSION
is the version to release. By default, this command uploads to PyPI’s test server, to upload to the production server:
pkgmt release VERSION --production
Continuous Delivery#
Important
Ensure you’re running the latest pkgmt
version before proceeding: pip install pkgmt -U
Only repository administrators can execute this process.
We’ve auomated uploading to PyPI in some projects (pkgmt
, ploomber-core
, jupysql
, and ploomber-engine
).
To make a new release, move to the master
(or main
) branch in a local copy of the git repository and execute:
pkgmt version
The command will ask for confirmation. Once confirmed, it’ll push a new commit and a git tag; then, if all tests pass, the version will be uploaded to PyPI.
You can see a sample GitHub Actions workflow here.
GitHub Actions workflows#
We have a few workflows to automate tasks, you can find the templates in the in the sample-github-workflows/
directory in this repository.
chatops.yml
: triggers certain actions when posting specific comments in a PRchangelog.yml
: comments on a PR if theCHANGELOG.md
file hasn’t been updated
Conda releases (conda-forge
)#
Some of our packages are available in conda (via conda-forge). The recipes are located here:
When uploading a new version to PyPI, the conda-forge bot automatically opens a PR to the feedstocks; upon approval, the new versions are available to install via conda install ploomber --channel conda-forge
.
Note that conda-forge implements a CI pipeline that checks that the recipe works. Thus, under most circumstances, the PR will pass. One exception is when adding new dependencies to setup.py
; in such a case, we must manually edit the recipe (meta.yml
) and open a PR to the feedstock. See the next section for details.
Note that it takes some time for packages to be available for download.
To check if packages are available: conda search ploomber --channel cf-staging
. Pending packages will appear in channel cf-staging
while available packages in conda-forge
. It usually takes less than one hour for packages to move from one to the other.
Manually updating the conda recipe#
If conda-forge
’s bot PR fails (usually because a new dependency was added), we must submit a PR ourselves:
Clone it:
git clone https://github.com/{your-user}/ploomber-feedstock
(changeyour-user
)Create a new branch:
git checkout -b branch-name
Update recipe (
meta.yaml
):Update the version in the
{% set version = "version" %}
lineUpdate
source.sha256
, you can get that fromhttps://pypi.org/project/{package-name}/{version}/#files
, just change the{package-name}
,{version}
, and copy the SHA256 hash from the.tar.gz
fileIf there are new dependencies (or new constraints), add them to
requirements.run
You may need to run
conda smithy rerender -c auto
(click here for details)
If you already forked the repository, you can sync with the original repository like this:
git remote add upstream https://github.com/conda-forge/ploomber-feedstock
git fetch upstream
git checkout main
git merge upstream/main
Repository quality checklist#
This is a checklist of things we need to routinely verify:
Unit tests in the CI should run with the
pytest
command (no arguments), if arguments are needed you can add them to thepyproject.toml
file (The exception are repositories where we have unit and integration tests separated, for example, in JupySQL’s case, where we have different configurations)Broken links should be checked using
pkgmt check-links
Unit testing from Python 3.8 until 3.11 on Linux, macOS, and Windows
Lint with
pkgmt lint
(this runsblack
andnbqa
)There must be a Pull Request template
Read the docs must be configured to build docs on each Pull Request
pkgmt setup
must install development dependencies (alternatively, there must be atasks.py
file with asetup
command)pkgmt doc
must build docs (alternatively, there must be atasks.py
file with asetup
command)Ensure we have tag protections for all tags
If anything is missing, open a GitHub issue in the corresponding repository.