Hi, Jose here!
I recently began developing a private git package to be used by many services from our organization. While the basic setup was relatively straightforward, I quickly realized how scaling it encompassed many concepts. Factors like integrating with various repositories and adapting your CI/CD pipelines can significantly raise the bar.
In this blog post, I will walk you through the process of installing a private git dependency and demonstrate how to use Poetry effectively to manage packages from multiple code repositories.
Requirements
- Python 3.10
- Poetry version 1.8.4
Working with public git dependencies
Installing a dependency in Poetry is simple enough. Just run poetry add package_name. This will add the respective package to the pyproject.toml file.
For git dependencies, we must specify the location of the repository with the git key. Let’s install the requests library from Github.
poetry add git+https://github.com/psf/requests.git@v2.32.2
Now your pyproject.toml will look as follows
requests = {git = "https://github.com/psf/requests.git", rev = "v2.32.2"}
If you don’ specify the rev property, Poetry will take up the latest commit of the main branch. Check the official docs for more information.
Installing a private git dependency
Poetry needs to authenticate to your git provider to install private dependencies. In the case of Github, we create a Personal Access Token (PAT). A Personal Access Token provides a secure way to authenticate to GitHub without the need of a password. Generate a PAT, set up authentication and install the package.
$ poetry config repositories.git-org-project https://github.com/org/private_lib.git $ poetry config http-basic.git-org-project username $PAT_TOKEN $ poetry add git+https://github.com/org/private_lib.git
Notice that the pyproject.toml looks almost the same as the public repo. Poetry ensures that your private credentials aren’t reflected.
[tool.poetry.dependencies] python = ">=3.10, <3.13" requests = {git = "https://github.com/psf/requests.git", rev = "v2.32.2"} pydantic = "^2.10.3" private_lib = {git = "https://github.com/org/private_lib.git"}
Managing multiple private repositories.
Now imagine your project needs your sales team’ internal libraries hosted in a private GitHub repository, but your research team maintains their libraries in an AWS CodeArtifact repository. How would you seamlessly integrate both?
Enter sources. Poetry uses sources to discover and install packages in your project. The default one is PyPI. Sources enable seamless integration of internal, third-party libraries without disrupting the main dependency flow.
In the example above, we would run
$ poetry source add --priority=supplemental research https://domain.d.codeartifact.ap-northeast-1.amazonaws.com/pypi/research/ $ poetry source add --priority=supplemental sales https://github.com/org/sales.git
Poetry will add the following to your pyproject.toml .
[[tool.poetry.source]] name = "research" url = "https://domain.d.codeartifact.ap-northeast-1.amazonaws.com/pypi/research/" priority = "supplemental" [[tool.poetry.source]] name = "sales" url = "https://github.com/org/sales.git" priority = "supplemental"
We use priority supplemental to tell Poetry that PyPI should still be the main code repository.
You can tweak the priorities to your needs, for instance, you can disable PyPI completely.
Remember that we still need to setup authentication for each source.
$ poetry config repositories.research https://domain.d.codeartifact.ap-northeast-1.amazonaws.com/pypi/research/ $ poetry config http-basic.research username ca_token $ poetry config repositories.sales https://github.com/org/sales.git $ poetry config http-basic.sales username token
Finally we install our libraries
$ poetry add sales-lib --source sales $ poetry add research-lib --source research
Conclusion
In this blog post we reviewed the steps to add private git repositories into Poetry. We also looked at how to manage multiple code repositories with Poetry. In a production environment, we would containerize the application and integrate it to a CI/CD pipeline. Those steps, although similar in nature, require extra care specially when using secret tokens.
References
https://python-poetry.org/docs/dependency-specification/#git-dependencies
https://python-poetry.org/docs/repositories/#package-sources
