Python / Guild Packaging

[vv111y]

This is a thread to document the use of python packaging in Guild and for ML projects generally.

TODOs

• [ ] How to create packages for a bare model (not trained, empty) and several trained versions
  • naming conflicts?
  • automated scripts to re-package from bare?
• [ ] proper
• [ ] package untrained model
• [ ] package trained model
• [ ] test Guild packaging
• [ ] test Guild remote execution

Refs

• Main Docs
  • Python Packaging Authority — PyPA documentation (via Guild meet [2020-10-15 Thu])
  • Python Packaging User Guide — Python Packaging User Guide
    • An Overview of Packaging for Python — Python Packaging User Guide
    • Packaging Python Projects — Python Packaging User Guide
    • Packaging and distributing projects — Python Packaging User Guide
• Guides
  • Minimal Structure — Python Packaging Tutorial
  • activescott/python-package-example: A simple example of creating and consuming a distributable Python package.
  • Knowledge Bits — Common Python Packaging Mistakes [2020-08-31 Mon] (via C. Martin)
  • Publishing (Perfect) Python Packages on PyPi - YouTube [2020-10-14 Wed 21:55]
• package server
  • devpi: PyPI server and packaging/testing/release tool documentation
  • muccg/docker-devpi: Dockerfile for devpi pypi server
• Relevant packages (also can use as guides for project setup) (via Guild meet [2020-10-15 Thu])
  • pypa/pip: The Python package installer
  • pypa/setuptools: Official project repository for the Setuptools build system
  • pypa/virtualenv: Virtual Python Environment builder
  • pypa/twine: Utilities for interacting with PyPI
• Fora, Q&A:
  • What is the best project structure for a Python application? - Stack Overflow
  • Guild AI and cookiecutter structure - General - Guild AI
• Tools (incomplete)
  • mgedmin/check-manifest: Tool to check the completeness of MANIFEST.in for Python packages [2020-10-14 Wed 22:12]
  • Coverage.py — Coverage.py 5.3 documentation [2020-10-14 Wed 22:12]
  • unittest.mock — getting started — Python 3.9.0 documentation [2020-10-14 Wed 22:12]
• Readme file
  • Making a PyPI-friendly README — Python Packaging User Guide
  • A template to make good README.md
  • template-python/README.md at master · jacebrowning/template-python
• Licenses
  • Choose an open source license | Choose a License
  • Licenses | Choose a License
  • Over 10% of Python Packages on PyPI are Distributed Without Any License | Snyk
  • TLDRLegal - Software Licenses Explained in Plain English

[vv111y]

Today I tried out Devpi (see package server above). I really like it.
It provides a local py package repository, but it is geared to development.
I used the docker implementation above with slight modifications for versioning.
You can upload a package with a simple devpi upload in the package root directory.
I also used a bare coockiecutter template

cookiecutter https://github.com/audreyr/cookiecutter-pypackage

just for a test upload and pip install

[garrett]

Nice!

I wrote a how-to article on integrating Guild models/ops with distributions created for such projects.

Guild packages with setup.py

The package command is used to generate an installable package that includes Guild models and operations. For more information, see Packages. This command simplifies the process of packaging. However, it cannot be used in conjunction with a project’s setup.py script. To support Guild packages using setup.py you must make a few changes to that script. The steps below assume a standard Python distribution created with setup.py. You can use Cookiecutter to generate a Python project that support…