mirror of
https://github.com/vale981/poetry2nix
synced 2025-03-04 08:41:42 -05:00
Document explaining known issues with python packages
Also corresponding changes in issue template to reference new document.
This commit is contained in:
Andrey "MOU" Larionov
parent
b37a389df4
commit
2b413427c1
2 changed files with 155 additions and 3 deletions
8
.github/ISSUE_TEMPLATE/issue-template.md
vendored
8
.github/ISSUE_TEMPLATE/issue-template.md
vendored
|
|
@ -9,9 +9,11 @@ Is this an issue in the style of "package "X fails to build"?
|
|||
Most such issues aren't issues in poetry2nix as such, but rather an incomplete dependency specification.
|
||||
We don't have the capacity to act on most such issues, they often require domain knowledge and they generate noise in the issue tracker.
|
||||
|
||||
The best way forward for these types of issues is _not_ to create an issue but to try to create an override yourself and submit a pull request.
|
||||
|
||||
If you are struggling to create a pull request, come to the Matrix chat and we will happily help you through the process.
|
||||
Please take a look into known edge cases (https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md),
|
||||
maybe it's already solved or you find an inspiration for your own solution.
|
||||
If you did not found your issue in list of known, then it will be more productive, faster and creates less burden on maintainer
|
||||
if you first try to discuss your case in Matrix chat (you can find information about it in README).
|
||||
Community will happily help to find soulution and if needed create issue or pull request
|
||||
-->
|
||||
|
||||
### Describe the issue
|
||||
|
|
|
|||
150
docs/edgecases.md
Normal file
150
docs/edgecases.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
# Edge cases
|
||||
## Preamble
|
||||
The python ecosystem has a very long history. Over the years, an uncountable number of modules have been created, but more importantly, the assembly technology of these modules has changed many times. `ez_install`, `setuptools`, `distutils`, eggs, `pip`, wheels, PEP-518, PEP-517, `flit`, `poetry`. All this diversity creates great difficulties. Poetry, on the one hand, being a build system, can abandon the setuptools and distutils formats, but at the same time, being a dependency installation and management system, it must support all current distribution technologies. And it can do this without revealing the internal structure and the machinery behind curtains. In other words, when you use Poetry, both `setuptools`, `pip` and other build tools can be used to install some modules, and this happens without formally declaring dependence on these tools.
|
||||
|
||||
All these factors can cause the module not to be assembled or installed in a certain environment. Nix is a much stricter ecosystem, it has more formal requirements, but at the same time more guarantees. And since `poetry2nix` relies on Nix and its strict rules in its work, there are many rough edges and inaccuracies in the metadata on pypi.org leads to errors.
|
||||
This is very sad, and we will inevitably face the fact that some modules can not be installed, can not be built, or just cause errors, have unnecessary or missing direct and transitive dependencies. However, Nix is a very flexible system that allows you to modify some aspects of the module with pinpoint accuracy, without changing the source code of the module (and sometimes changing it). This is what this section is dedicated to. We have tried to collect here and describe typical errors and ways to eliminate them.
|
||||
|
||||
## Cases
|
||||
|
||||
#### ModuleNotFoundError: No module named 'poetry'
|
||||
**Conditions:** you have declared a dependency on git repository with the python package. And this package uses poetry as a build tool. So since `poetry2nix` cannot obtain this dependency in form of a wheel, it needs to build it from the source by calling pip. But pip requires poetry to build this package. And that’s when an error occurs.
|
||||
|
||||
**Solution:** in order to make dependency build we need to override it build dependencies by adding poetry package to it.
|
||||
|
||||
**Example:** let's consider situation when our package has declared `git` dependency in `pyproject.toml` on `extralib`, which uses poetry to be built. And we have this declaration in our Nix definition:
|
||||
```
|
||||
poetry2nix.mkPoetryApplication {
|
||||
projectDir = ./.;
|
||||
}
|
||||
```
|
||||
And trying to build it get us next error log
|
||||
<details>
|
||||
<summary>error log (click to expand)</summary>
|
||||
|
||||
```
|
||||
Sourcing python-catch-conflicts-hook.sh
|
||||
Sourcing python-remove-bin-bytecode-hook.sh
|
||||
Sourcing pip-build-hook
|
||||
Using pipBuildPhase
|
||||
Using pipShellHook
|
||||
Sourcing pip-install-hook
|
||||
Using pipInstallPhase
|
||||
Sourcing python-imports-check-hook.sh
|
||||
Using pythonImportsCheckPhase
|
||||
Sourcing python-namespaces-hook
|
||||
@nix { "action": "setPhase", "phase": "unpackPhase" }
|
||||
unpacking sources
|
||||
unpacking source archive /nix/store/f2mb5sy6vxm81sy5apzvbxmnvj8f62la-source
|
||||
source root is source
|
||||
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/extralib/utils.py
|
||||
@nix { "action": "setPhase", "phase": "patchPhase" }
|
||||
patching sources
|
||||
Removing path dependencies
|
||||
Finished removing path dependencies
|
||||
Removing git dependencies
|
||||
Finished removing git dependencies
|
||||
@nix { "action": "setPhase", "phase": "configurePhase" }
|
||||
configuring
|
||||
no configure script, doing nothing
|
||||
@nix { "action": "setPhase", "phase": "buildPhase" }
|
||||
building
|
||||
Executing pipBuildPhase
|
||||
Creating a wheel...
|
||||
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
|
||||
Ignoring indexes: https://pypi.org/simple
|
||||
Created temporary directory: /build/pip-ephem-wheel-cache-oluqv9z9
|
||||
Created temporary directory: /build/pip-req-tracker-6nbzvq94
|
||||
Initialized build tracking at /build/pip-req-tracker-6nbzvq94
|
||||
Created build tracker: /build/pip-req-tracker-6nbzvq94
|
||||
Entered build tracker: /build/pip-req-tracker-6nbzvq94
|
||||
Created temporary directory: /build/pip-wheel-ecarqqhj
|
||||
Processing /build/source
|
||||
Created temporary directory: /build/pip-req-build-thqopu_1
|
||||
DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
|
||||
pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
|
||||
Added file:///build/source to build tracker '/build/pip-req-tracker-6nbzvq94'
|
||||
Created temporary directory: /build/pip-modern-metadata-5otx3b_0
|
||||
Running command /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin/python3.9 /nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmp8cfof26u
|
||||
Preparing wheel metadata ... done
|
||||
ERROR: Exception:
|
||||
Traceback (most recent call last):
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
|
||||
status = self.run(options, args)
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
|
||||
return func(self, options, args)
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 142, in run
|
||||
requirement_set = resolver.resolve(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
|
||||
r = self.factory.make_requirement_from_install_req(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 429, in make_requirement_from_install_req
|
||||
cand = self._make_candidate_from_link(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 200, in _make_candidate_from_link
|
||||
self._link_candidate_cache[link] = LinkCandidate(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 306, in __init__
|
||||
super().__init__(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 151, in __init__
|
||||
self.dist = self._prepare()
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 234, in _prepare
|
||||
dist = self._prepare_distribution()
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 317, in _prepare_distribution
|
||||
return self._factory.preparer.prepare_linked_requirement(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
|
||||
return self._prepare_linked_requirement(req, parallel_builds)
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 570, in _prepare_linked_requirement
|
||||
dist = _get_prepared_distribution(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 60, in _get_prepared_distribution
|
||||
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 36, in prepare_distribution_metadata
|
||||
self.req.prepare_metadata()
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 549, in prepare_metadata
|
||||
self.metadata_directory = self._generate_metadata()
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 534, in _generate_metadata
|
||||
return generate_metadata(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/build/metadata.py", line 31, in generate_metadata
|
||||
distinfo_dir = backend.prepare_metadata_for_build_wheel(
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 184, in prepare_metadata_for_build_wheel
|
||||
return self._call_hook('prepare_metadata_for_build_wheel', {
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 275, in _call_hook
|
||||
raise BackendUnavailable(data.get('traceback', ''))
|
||||
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
|
||||
File "/nix/store/11wvwr8f2dp4x8xjnrgqn3inmh418apn-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 86, in _build_backend
|
||||
obj = import_module(mod_path)
|
||||
File "/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
|
||||
return _bootstrap._gcd_import(name[level:], package, level)
|
||||
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
|
||||
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
|
||||
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
|
||||
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
|
||||
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
|
||||
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
|
||||
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
|
||||
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
|
||||
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
|
||||
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
|
||||
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
|
||||
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
|
||||
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
|
||||
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
|
||||
File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
|
||||
ModuleNotFoundError: No module named 'poetry'
|
||||
|
||||
Removed file:///build/source from build tracker '/build/pip-req-tracker-6nbzvq94'
|
||||
Removed build tracker: '/build/pip-req-tracker-6nbzvq94'
|
||||
```
|
||||
</details>
|
||||
|
||||
In order to be able to build `extralib` we should modify our nix definition as follows:
|
||||
```
|
||||
poetry2nix.mkPoetryApplication {
|
||||
projectDir = ./.;
|
||||
overrides = poetry2nix.defaultPoetryOverrides.extend ( self: super: {
|
||||
extralib = super.extralib.overridePythonAttrs (
|
||||
old: {
|
||||
buildInputs = (old.buildInputs or [ ]) ++ [ self.poetry ];
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
```
|
||||
This override will instruct underlying build logic to includ additional build dependency into inputs of `extralib`.
|
||||
Loading…
Add table
Reference in a new issue