Google Magenta Portfiles Help Request for new py-note-seq port

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
I’m preparing a PR for the Google Magenta project <https://magenta.tensorflow.org>, and am running into this issue for the “note-seq” dependent package with this setup.py:


raise DistutilsError('the `allow-hosts` option is not supported '
distutils.errors.DistutilsError: the `allow-hosts` option is not supported when using pip to install requirements.

I (believe I) have all the correct dependencies installed, and see that this is an issue with setuptools elsewhere, .e.g., https://github.com/pypa/setuptools/issues/1916#issuecomment-562788843

Is there a straightforward workaround to this issue?


Full log error:
:info:build Executing:  cd "/opt/local/var/macports/build/_opt_local_ports_python_py-note-seq/py37-note-seq/work/note-seq-0.0.0" && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j12 
:debug:build system:  cd "/opt/local/var/macports/build/_opt_local_ports_python_py-note-seq/py37-note-seq/work/note-seq-0.0.0" && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j12 
:info:build Traceback (most recent call last):
:info:build   File "setup.py", line 67, in <module>
:info:build     'pylint >= 2.4.2',
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 160, in setup
:info:build     _install_setup_requires(attrs)
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 155, in _install_setup_requires
:info:build     dist.fetch_build_eggs(dist.setup_requires)
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/dist.py", line 698, in fetch_build_eggs
:info:build     replace_conflicting=True,
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
:info:build     replace_conflicting=replace_conflicting
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
:info:build     return self.obtain(req, installer)
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
:info:build     return installer(requirement)
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/dist.py", line 754, in fetch_build_egg
:info:build     return fetch_build_egg(self, req)
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/installer.py", line 83, in fetch_build_egg
:info:build     raise DistutilsError('the `allow-hosts` option is not supported '
:info:build distutils.errors.DistutilsError: the `allow-hosts` option is not supported when using pip to install requirements.
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_ports_python_py-note-seq/py37-note-seq/work/note-seq-0.0.0" && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j12 
:info:build Exit code: 1
:error:build Failed to build py37-note-seq: command execution failed
:debug:build Error code: CHILDSTATUS 10565 1
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_ports_python_py-note-seq/py37-note-seq/main.log for details.

Draft Portfile:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          1.0
PortGroup           github 1.0
PortGroup           python 1.0

github.setup        magenta note-seq a18f4ba
revision            20200622
name                py-${github.project}
categories-append   audio

version             0.0.0
platforms           darwin
license             Apache-2
maintainers         nomaintainer

description         Use machine learning to create art and music.
long_description    ${description}. This is\
                    the home for our serializable NoteSequence\
                    representation along with utilities for: creating\
                    them from various formats (e.g., MIDI, abc,\
                    MusicXML)\; manipulating them (e.g., slicing,\
                    quantizing) extracting components (e.g., melodies,\
                    drum tracks, chord)\; exporting them (e.g., to MIDI\
                    or audio)\; or converting them to representations\
                    useful for model training (e.g., one-hot tensors).

homepage            https://magenta.tensorflow.org/
distname            ${github.project}-${version}

checksums           rmd160  96aa00282fa38a8ce63d563c2c667d6f241e1850 \
                    sha256  5a96d9d4a179cc6f03975d61f379689afc282df524af0613107591dd2aed9223 \
                    size    952994

python.versions     37 38

if {${name} ne ${subport}} {
    depends_build-append \
                    port:py${python.version}-setuptools

    depends_run-append \
                    port:py${python.version}-absl \
                    port:py${python.version}-attrs \
                    port:py${python.version}-bokeh \
                    port:py${python.version}-intervaltree \
                    port:py${python.version}-ipython \
                    port:py${python.version}-librosa \
                    port:py${python.version}-numba \
                    port:py${python.version}-numpy \
                    port:py${python.version}-pandas \
                    port:py${python.version}-pretty-midi \
                    port:py${python.version}-protobuf3 \
                    port:py${python.version}-scikit-image \
                    port:py${python.version}-scipy

    depends_test-append \
                    port:py${python.version}-pylint \
                    port:py${python.version}-pytest \
                    port:py${python.version}-pytest-xdist

    test.run        yes
    test.cmd        py.test-${python.branch}
    test.target
    test.env-append \
                    "PATH=$env(PATH):${workpath}/bin" \
                    PYTHONPATH=${worksrcpath}/build/lib

    post-destroot {
        set docdir ${prefix}/share/doc/${subport}
        xinstall -d ${destroot}${docdir}
        xinstall -m 0644 -W ${worksrcpath} LICENSE README.md \
            ${destroot}${docdir}
    }

    livecheck.type      none
}





smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
Running the basic build command by hand yields a successful build:

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j12

This suggests that one of the environment variables is causing the issue.

How does one remove all environment variables from the build phase?

Simply invoking

build.env


doesn’t change anything and the default build environment is visible in the log:

:debug:build Environment:
:debug:build CC='/usr/bin/clang'
:debug:build CC_PRINT_OPTIONS='YES'
:debug:build CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_ports_python_py-note-seq/py37-note-seq/work/.CC_PRINT_OPTIONS'
:debug:build CFLAGS='-arch x86_64 -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk'
:debug:build CPATH='/opt/local/include'
:debug:build CXX='/usr/bin/clang++'
:debug:build CXXFLAGS='-arch x86_64 -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk'
:debug:build DEVELOPER_DIR='/Applications/Xcode.app/Contents/Developer'
:debug:build F90FLAGS='-m64'
:debug:build FCFLAGS='-m64'
:debug:build FFLAGS='-m64'
:debug:build LDFLAGS='-arch x86_64'
:debug:build LIBRARY_PATH='/opt/local/lib'
:debug:build MACOSX_DEPLOYMENT_TARGET='10.15'
:debug:build OBJC='/usr/bin/clang'
:debug:build OBJCFLAGS='-arch x86_64 -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk'
:debug:build SDKROOT='/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk'

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Joshua Root-8
On 2020-7-4 10:40 , Steven Smith wrote:
> Running the basic build command by hand yields a successful build:
>
>> /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
>> setup.py --no-user-cfg build -j12
>
> This suggests that one of the environment variables is causing the issue.

I would say that suggests rather that an unsatisfied requirement is
being satisfied by automatically downloading a module from pypi, which
is what the allow-hosts setting is there to prevent.

I would check the requirements again, paying attention to versions as
well as module names.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

ryandesign2
Administrator
In reply to this post by Steven Smith


On Jul 3, 2020, at 14:23, Steven Smith wrote:

>> revision            20200622

revision should be 0.


>> version             0.0.0

version should probably be 20200622, unless they have a real version number.


>>     test.env-append \
>>                     "PATH=$env(PATH):${workpath}/bin" \

Don't you want your ${workpath}/bin to precede what's already in $env(PATH)?


On Jul 3, 2020, at 19:40, Steven Smith wrote:

> Running the basic build command by hand yields a successful build:
>
>> /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j12
>
> This suggests that one of the environment variables is causing the issue.

No, it suggests that when you run it by hand, you're not benefitting from the features of the python portgroup, one of which is to ensure that you've specified your python module dependencies correctly by preventing them from being automatically downloaded.


> How does one remove all environment variables from the build phase?

You don't do that.


> Simply invoking
>
> build.env
>
> doesn’t change anything

Correct.

Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
Re: versioning, here’s the situation:

pypi has a version of 0.0.0, but the github repo has no official version number or even tags, and it’s not clear which commit the pypi version has used.

I prefer to build from a github repo because it contains tests and other useful things that the pypi distribution doesn’t have.

In this situation, I specified the commit hash, set the version to be consistent with pypi’s, and set the revision to be the github commit date.

Is there another preferred way in this situation?

Magenta just migrated to tf2, still depends on deprecated packages like tensor2tensor, and I expect will be pretty dynamic for a bit. Yet it has working capabilities now that are useful to get under install/version control.


> On Jul 4, 2020, at 10:03, Ryan Schmidt <[hidden email]> wrote:
>
> 
>
> On Jul 3, 2020, at 14:23, Steven Smith wrote:
>
>>> revision            20200622
>
> revision should be 0.
>
>
>>> version             0.0.0
>
> version should probably be 20200622, unless they have a real version number.
>
>
>>>    test.env-append \
>>>                    "PATH=$env(PATH):${workpath}/bin" \
>
> Don't you want your ${workpath}/bin to precede what's already in $env(PATH)?
>
>
> On Jul 3, 2020, at 19:40, Steven Smith wrote:
>
>> Running the basic build command by hand yields a successful build:
>>
>>> /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j12
>>
>> This suggests that one of the environment variables is causing the issue.
>
> No, it suggests that when you run it by hand, you're not benefitting from the features of the python portgroup, one of which is to ensure that you've specified your python module dependencies correctly by preventing them from being automatically downloaded.
>
>
>> How does one remove all environment variables from the build phase?
>
> You don't do that.
>
>
>> Simply invoking
>>
>> build.env
>>
>> doesn’t change anything
>
> Correct.
>

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
In reply to this post by Joshua Root-8
Thanks. I’ll look again, but am scratching my head about why this fails when run from a port command, but works when run by hand. And I don’t see any download attempts in the log file.

FWIW, I hacked out a path that comments out all the *_depends lines in setup.py and this creates a working install.

Works if these setup.py lines are all deleted:


On Jul 4, 2020, at 02:11, Joshua Root <[hidden email]> wrote:

On 2020-7-4 10:40 , Steven Smith wrote:
Running the basic build command by hand yields a successful build:

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
setup.py --no-user-cfg build -j12

This suggests that one of the environment variables is causing the issue.

I would say that suggests rather that an unsatisfied requirement is
being satisfied by automatically downloading a module from pypi, which
is what the allow-hosts setting is there to prevent.

I would check the requirements again, paying attention to versions as
well as module names.

- Josh

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Joshua Root-8
On 2020-7-5 01:25 , Steven Smith wrote:
> Thanks. I’ll look again, but am scratching my head about why this fails
> when run from a port command, but works when run by hand. And I don’t
> see any download attempts in the log file.
>
> FWIW, I hacked out a path that comments out all the *_depends lines in
> setup.py and this creates a working install.
>
> Works if these setup.py lines are all deleted:
> https://github.com/magenta/note-seq/blob/a18f4ba1b47ffb69b81ec058efd00c48fa27b911/setup.py#L62

REQUIRED_PACKAGES contains: 'numba == 0.48.0'

That's not the version installed by the py-numba port.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Joshua Root-8
In reply to this post by Steven Smith
On 2020-7-5 01:20 , Steven Smith wrote:
> Re: versioning, here’s the situation:
>
> pypi has a version of 0.0.0, but the github repo has no official version number or even tags, and it’s not clear which commit the pypi version has used.
>
> I prefer to build from a github repo because it contains tests and other useful things that the pypi distribution doesn’t have.
>
> In this situation, I specified the commit hash, set the version to be consistent with pypi’s, and set the revision to be the github commit date.
>
> Is there another preferred way in this situation?

If upstream doesn't have a useful version number, you need to make one
up. It should monotonically increase over time. It should change
whenever the upstream source code you are installing from changes.

Revision is a property of the port, not the upstream sources. It should
start at 0 with each new upstream version, and increment whenever you
make changes to the port that change the installed files, while still
using the same upstream version.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
In reply to this post by Joshua Root-8
> REQUIRED_PACKAGES contains: 'numba == 0.48.0'
>
> That's not the version installed by the py-numba port.

What’s the recommended Portfile approach in this situation?

Just edit out the version pinning in setup.py and hope for the best with the up-to-date version used by MacPorts?

I’m not aware of another solution than that.
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Jason Liu
In reply to this post by Joshua Root-8
> pypi has a version of 0.0.0, but the github repo has no official version number or even tags, and it’s not clear which commit the pypi version has used.

If upstream doesn't have a useful version number, you need to make one up. It should monotonically increase over time. It should change
whenever the upstream source code you are installing from changes.

For ports I've submitted which don't have any version numbering in the project's GitHub repo, I've been using the date of the commit as the version number in the portfile... but don't try to use the GitHub commit hash, since that is not a monotonically increasing number. So, for example, if I submitted a port for a project's commit that occurred today, the version number would be 20200704. Some of the MacPorts devs have indicated to me that this date-based version number shouldn't be separated using dashes or dots; it should just look like a single integer. Also, don't include the commit's time in the version number string, only the date.

-- 
Jason Liu


On Sat, Jul 4, 2020 at 1:49 PM Joshua Root <[hidden email]> wrote:
On 2020-7-5 01:20 , Steven Smith wrote:
> Re: versioning, here’s the situation:
>
> pypi has a version of 0.0.0, but the github repo has no official version number or even tags, and it’s not clear which commit the pypi version has used.
>
> I prefer to build from a github repo because it contains tests and other useful things that the pypi distribution doesn’t have.
>
> In this situation, I specified the commit hash, set the version to be consistent with pypi’s, and set the revision to be the github commit date.
>
> Is there another preferred way in this situation?

If upstream doesn't have a useful version number, you need to make one
up. It should monotonically increase over time. It should change
whenever the upstream source code you are installing from changes.

Revision is a property of the port, not the upstream sources. It should
start at 0 with each new upstream version, and increment whenever you
make changes to the port that change the installed files, while still
using the same upstream version.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
In reply to this post by Joshua Root-8
REQUIRED_PACKAGES contains: 'numba == 0.48.0'

That's not the version installed by the py-numba port.

I don’t believe that this is causing the problem because replacing this line with:

    'numba >= 0.48.0',  # temporary fix for librosa import

still causes the issue. I have this port installed:

  py37-numba @0.50.0_0 (active)

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Joshua Root-8
In reply to this post by Jason Liu
On 2020-7-5 06:35 , Jason Liu wrote:

>     If upstream doesn't have a useful version number, you need to make
>     one up. It should monotonically increase over time. It should change
>     whenever the upstream source code you are installing from changes.
>
>
> For ports I've submitted which don't have any version numbering in the
> project's GitHub repo, I've been using the date of the commit as the
> version number in the portfile... but don't try to use the GitHub commit
> hash, since that is not a monotonically increasing number. So, for
> example, if I submitted a port for a project's commit that occurred
> today, the version number would be 20200704. Some of the MacPorts devs
> have indicated to me that this date-based version number shouldn't be
> separated using dashes or dots; it should just look like a single
> integer. Also, don't include the commit's time in the version number
> string, only the date.

Those last couple sentences aren't official rules. I would actually
prefer the human-readable ISO 8601 format (separated with dashes).
Appending the time is likely overkill since that would only be useful if
you're updating multiple times per day.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Joshua Root-8
In reply to this post by Steven Smith
On 2020-7-5 07:16 , Steven Smith wrote:

>> REQUIRED_PACKAGES contains: 'numba == 0.48.0'
>>
>> That's not the version installed by the py-numba port.
>
> I don’t believe that this is causing the problem because replacing this
> line with:
>
>>     'numba >= 0.48.0',  # temporary fix for librosa import
>
> still causes the issue. I have this port installed:
>
>>   py37-numba @0.50.0_0 (active)

It's not the *only* problem. 'pytest-xdist < 1.30.0' is also in
tests_require.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

ryandesign2
Administrator
In reply to this post by Joshua Root-8
On Jul 5, 2020, at 03:01, Joshua Root wrote:

> On 2020-7-5 06:35 , Jason Liu wrote:
>>    If upstream doesn't have a useful version number, you need to make
>>    one up. It should monotonically increase over time. It should change
>>    whenever the upstream source code you are installing from changes.
>>
>>
>> For ports I've submitted which don't have any version numbering in the
>> project's GitHub repo, I've been using the date of the commit as the
>> version number in the portfile... but don't try to use the GitHub commit
>> hash, since that is not a monotonically increasing number. So, for
>> example, if I submitted a port for a project's commit that occurred
>> today, the version number would be 20200704. Some of the MacPorts devs
>> have indicated to me that this date-based version number shouldn't be
>> separated using dashes or dots; it should just look like a single
>> integer. Also, don't include the commit's time in the version number
>> string, only the date.
>
> Those last couple sentences aren't official rules. I would actually
> prefer the human-readable ISO 8601 format (separated with dashes).

We can move toward that if you like, but it has been our de-facto standard not to use dashes. I count 288 ports using a YYYYMMDD version and only 65 ports using a YYYY-MM-DD version.

Moving from a YYYY-MM-DD version to a YYYYMMDD version is easy; you just do it. Going the other way requires increasing the epoch. :(

Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Joshua Root-8
On 2020-7-5 19:54 , Ryan Schmidt wrote:

> On Jul 5, 2020, at 03:01, Joshua Root wrote:
>
>> On 2020-7-5 06:35 , Jason Liu wrote:
>>>    If upstream doesn't have a useful version number, you need to make
>>>    one up. It should monotonically increase over time. It should change
>>>    whenever the upstream source code you are installing from changes.
>>>
>>>
>>> For ports I've submitted which don't have any version numbering in the
>>> project's GitHub repo, I've been using the date of the commit as the
>>> version number in the portfile... but don't try to use the GitHub commit
>>> hash, since that is not a monotonically increasing number. So, for
>>> example, if I submitted a port for a project's commit that occurred
>>> today, the version number would be 20200704. Some of the MacPorts devs
>>> have indicated to me that this date-based version number shouldn't be
>>> separated using dashes or dots; it should just look like a single
>>> integer. Also, don't include the commit's time in the version number
>>> string, only the date.
>>
>> Those last couple sentences aren't official rules. I would actually
>> prefer the human-readable ISO 8601 format (separated with dashes).
>
> We can move toward that if you like, but it has been our de-facto standard not to use dashes. I count 288 ports using a YYYYMMDD version and only 65 ports using a YYYY-MM-DD version.
>
> Moving from a YYYY-MM-DD version to a YYYYMMDD version is easy; you just do it. Going the other way requires increasing the epoch. :(

I don't think we need to change any of them.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Google Magenta Portfiles Help Request for new py-note-seq port

Steven Smith
In reply to this post by Joshua Root-8
That’s the issue—thank you!

Now upstream needs to fix their number.decorators issue…

On Jul 5, 2020, at 4:06 AM, Joshua Root <[hidden email]> wrote:

It's not the *only* problem. 'pytest-xdist < 1.30.0' is also in
tests_require.


smime.p7s (5K) Download Attachment