request for port create command, to build a portfile from a URL

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

request for port create command, to build a portfile from a URL

Ken Cunningham
It would be nice to have a command like

port create URL

that would download the URL, calc the checksums for it, and build a basic Portfile for it.

It could ask for a category to use for the Port.

In a more advance version, it could extract the URL and do some quick analysis on it.

if there's a CMakeLists.txt file in the root, do the right things to make it a cmake based port.

If there's a configure file in the root, do the right things for that.

Might make things faster and easier for people to get started and up and running.

Certainly would make it easier for people who want to use MacPorts infrastructure to build their own files, as many don't know how to properly set up the include and lib directories, etc.

Best,

Ken


Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Michael Dickens-4
Hi Ken - I think that's a great idea. In "GNU Radio" land, we have such a tool for creating out-of-tree GR modules, called "gr_modtool"; it's great for setting up the skeleton for such OOT GR scripts. The end-user still has to fill in many "blanks", but this tool provides the starting point. I could see something similar for MacPorts, much as you describe. Maybe this could be added to the GSoC wiki page? It shouldn't be too difficult to implement something like what you propose, whether directly in 'port' or as a tool separate from 'port'. Cheers! - MLD

On Tue, Mar 6, 2018, at 9:58 AM, Ken Cunningham wrote:

> It would be nice to have a command like
>
> port create URL
>
> that would download the URL, calc the checksums for it, and build a
> basic Portfile for it.
>
> It could ask for a category to use for the Port.
>
> In a more advance version, it could extract the URL and do some quick
> analysis on it.
>
> if there's a CMakeLists.txt file in the root, do the right things to
> make it a cmake based port.
>
> If there's a configure file in the root, do the right things for that.
>
> Might make things faster and easier for people to get started and up and
> running.
>
> Certainly would make it easier for people who want to use MacPorts
> infrastructure to build their own files, as many don't know how to
> properly set up the include and lib directories, etc.
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Joshua Root-8
In reply to this post by Ken Cunningham
On 2018-3-7 01:58 , Ken Cunningham wrote:

> It would be nice to have a command like
>
> port create URL
>
> that would download the URL, calc the checksums for it, and build a basic Portfile for it.
>
> It could ask for a category to use for the Port.
>
> In a more advance version, it could extract the URL and do some quick analysis on it.
>
> if there's a CMakeLists.txt file in the root, do the right things to make it a cmake based port.
>
> If there's a configure file in the root, do the right things for that.
>
> Might make things faster and easier for people to get started and up and running.
>
> Certainly would make it easier for people who want to use MacPorts infrastructure to build their own files, as many don't know how to properly set up the include and lib directories, etc.

There's a fairly basic tool called portfile-gen in contrib. Improvements
are certainly welcome.

I'd advise not just blindly setting whatever checksums the downloaded
file happens to have though; the maintainer should be getting those from
a secure source (e.g. https web site) and verifying that they match, or
verifying the distfile against a gpg signature. If that can be automated
somehow then great, but let's not encourage skipping it.

- Josh
G A
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

G A
In reply to this post by Ken Cunningham
I like the auto calculate hashes

For grabbing something from url sounds like having a lot of switches,

for archive type and depedencies

what if link is a redirect to cdn, or something like Dropbox or symbolic

> On Mar 6, 2018, at 06:58, Ken Cunningham <[hidden email]> wrote:
>
> Best,
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Mojca Miklavec-2
In reply to this post by Ken Cunningham
Hi,

On 6 March 2018 at 15:58, Ken Cunningham wrote:

> It would be nice to have a command like
>
> port create URL
>
> that would download the URL, calc the checksums for it, and build a basic Portfile for it.
>
> It could ask for a category to use for the Port.
>
> In a more advance version, it could extract the URL and do some quick analysis on it.
>
> if there's a CMakeLists.txt file in the root, do the right things to make it a cmake based port.
>
> If there's a configure file in the root, do the right things for that.
>
> Might make things faster and easier for people to get started and up and running.
>
> Certainly would make it easier for people who want to use MacPorts infrastructure to build their own files, as many don't know how to properly set up the include and lib directories, etc.

I would make a GSOC project proposal for this :)

This would certainly be nice functionality, it's just that someone
needs to write it.

Mojca
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Ken Cunningham
In reply to this post by G A

On 2018-03-06, at 7:22 AM, G Alexander wrote:

I like the auto calculate hashes

For grabbing something from url sounds like having a lot of switches,

for archive type and depedencies


I was kinda hoping that the archive type might be figured out from the tail of the URL. All the use_bzip2 yes business if plenty confusing for people. Even the whole idea of master_sites and distnames takes getting used to.

dependencies would have to be manually specified. A basic template could be laid down for them, however, in the generated Portfile.


what if link is a redirect to cdn, or something like Dropbox or symbolic


some things would have to be manually done, unless AI can sort this out in ways I don't know how to do...

K

db
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

db
In reply to this post by Joshua Root-8
On 6 Mar 2018, at 16:19, Joshua Root <[hidden email]> wrote:
> On 2018-3-7 01:58 , Ken Cunningham wrote:
>>
>> port create URL
>> Might make things faster and easier for people to get started and up and running.
> There's a fairly basic tool called portfile-gen in contrib.

I took a bit to find it, so here's the link https://github.com/macports/macports-contrib/tree/master/portfile-gen.

I learned to write a portfile following some article in the wild, then I went through the whole guide, landed eventually in the mailing list.

While portfile-gen might help (homebrew has a similar feature), an *overview* of how to write a portfile is much needed.
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Ken Cunningham
In reply to this post by Joshua Root-8

On 2018-03-06, at 7:19 AM, Joshua Root wrote:

>
> There's a fairly basic tool called portfile-gen in contrib. Improvements
> are certainly welcome.

That is indeed a step towards what I was thinking of.

I'll look at this.

Ken

Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Rainer Müller-4
In reply to this post by db
On 2018-03-06 23:00, db wrote:
> [...] an *overview* of how to write a portfile is much needed.

Isn't this what this chapter in the guide is supposed to provide?

https://guide.macports.org/#development

Rainer
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Enrico Maria Crisostomo
In reply to this post by Ken Cunningham
Hi Ken,

I think it's a great idea. I'm maintaining a lot of software both on my personal machines and at work using MacPorts and I've setup some local Makefile targets to do some of this work. Perhaps I could contribute some work on this area, but first I'd like to know what is it that we'd like to have.

What is currently working for me is a very simple, template-based approach. Since 90% of the time I'm dealing with ports using the github portgroup, I'm basically calculating the relevant variables (port name, version, checksums, etc.) and fill in a template. The template is a file where shell expansion is performed (i.e.: variables in the form ${VAR}) are expanded.) I think this approach could be both very useful and simple to maintain if we had a list of pre-defined templates. It doesn't scale well with the number of different templates to support and their complexity. If we needed templates to have logic in it (flow control, expressions, etc.),then my guts say we should move to a suitable templating engine (or a different kind of solution altogether).

I've had a look at portfile-gen and it's a nice source of inspiration. Yet, I'm not convinced that outputting text directly from the program flow is either easy to read or easily maintainable, especially if complexity goes up.

portfile-gen, if I understand correctly, supports the following groups:

  * perl5
  * php
  * python
  * ruby
  * github

If we extracted a template for each one of those groups (or more templates in case a group may have multiple templates), then the effort required to come up with a working prototype would be implementing the functions required to populate the environment (i.e.: downloading a file if needed, calculating checksums, examining it). Perhaps we would have to split a Portfile in multiple segments, and generate each segment from a specific template (e.g.: if a port requires GNU Autotools).

What are your gut feelings about a template-based solution? Do you think it would be able to manage the level of Portfile complexity we desire for this tool?

Cheers,
--
Enrico


> On 7 Mar 2018, at 01:27, Ken Cunningham <[hidden email]> wrote:
>
>
> On 2018-03-06, at 7:19 AM, Joshua Root wrote:
>
>>
>> There's a fairly basic tool called portfile-gen in contrib. Improvements
>> are certainly welcome.
>
> That is indeed a step towards what I was thinking of.
>
> I'll look at this.
>
> Ken
>

db
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

db
In reply to this post by Rainer Müller-4
On 7 Mar 2018, at 01:53, Rainer Müller <[hidden email]> wrote:
> On 2018-03-06 23:00, db wrote:
>> [...] an *overview* of how to write a portfile is much needed.
> Isn't this what this chapter in the guide is supposed to provide?
> https://guide.macports.org/#development

Yes, supposed. When you're in it's difficult to say, but AFAIR I was probably trying to write a portfile for something hosted on GitHub without knowing about the relative portgroup and its documentation being buried somewhere under the prefix in the tcl file itself.
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Enrico Maria Crisostomo
This is a little bit offtopic IMHO, anyway: I found surprising too that the github portgroup was not documented, given the high percentage of software that I build out of github repositories, and that's why I recently contributed the documentation of the github portgroup:


It has been merged some days ago. Yet, I don't see it online yet.

On 7 Mar 2018, at 11:29, db <[hidden email]> wrote:

On 7 Mar 2018, at 01:53, Rainer Müller <[hidden email]> wrote:
On 2018-03-06 23:00, db wrote:
[...] an *overview* of how to write a portfile is much needed.
Isn't this what this chapter in the guide is supposed to provide?
https://guide.macports.org/#development

Yes, supposed. When you're in it's difficult to say, but AFAIR I was probably trying to write a portfile for something hosted on GitHub without knowing about the relative portgroup and its documentation being buried somewhere under the prefix in the tcl file itself.

G A
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

G A
I play in a different playground and primarily use either master, dev or develop branches as I run with latest codebase available.  ffmpeg, imagemagick, opencv, clang6, openblas, etc.  These get a lot of PRs everyday. 

Having to update the Portfiles manually with latest hash and date/version is not optimal.  

Why can't a script find the repo, verify hash and date automatically then archive or if you have a GitHub account verify your SSH, and just grab the repo without creating an archive.

On Mar 7, 2018, at 02:32, Enrico Maria Crisostomo <[hidden email]> wrote:

This is a little bit offtopic IMHO, anyway: I found surprising too that the github portgroup was not documented, given the high percentage of software that I build out of github repositories, and that's why I recently contributed the documentation of the github portgroup:


It has been merged some days ago. Yet, I don't see it online yet.

On 7 Mar 2018, at 11:29, db <[hidden email]> wrote:

On 7 Mar 2018, at 01:53, Rainer Müller <[hidden email]> wrote:
On 2018-03-06 23:00, db wrote:
[...] an *overview* of how to write a portfile is much needed.
Isn't this what this chapter in the guide is supposed to provide?
https://guide.macports.org/#development

Yes, supposed. When you're in it's difficult to say, but AFAIR I was probably trying to write a portfile for something hosted on GitHub without knowing about the relative portgroup and its documentation being buried somewhere under the prefix in the tcl file itself.

Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Rainer Müller-4
In reply to this post by Enrico Maria Crisostomo
On 2018-03-07 11:32, Enrico Maria Crisostomo wrote:
> This is a little bit offtopic IMHO, anyway: I found surprising too that
> the github portgroup was not documented, given the high percentage of
> software that I build out of github repositories, and that's why I
> recently contributed the documentation of the github portgroup:
>
> https://github.com/macports/macports-guide/pull/12
>
> It has been merged some days ago. Yet, I don't see it online yet.

That is because we still have not any automatic deployment on
build.macports.org for any of our websites. I am irregularly running
these jobs from my local buildbot setup on my personal MacBook every few
weeks, which is the only reason they are not completely outdated.

I triggered another run of these jobs, so these changes should be
available shortly online.

Rainer
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Ryan Schmidt-24

On Mar 7, 2018, at 07:32, Rainer Müller wrote:

> On 2018-03-07 11:32, Enrico Maria Crisostomo wrote:
>> This is a little bit offtopic IMHO, anyway: I found surprising too that
>> the github portgroup was not documented, given the high percentage of
>> software that I build out of github repositories, and that's why I
>> recently contributed the documentation of the github portgroup:
>>
>> https://github.com/macports/macports-guide/pull/12
>>
>> It has been merged some days ago. Yet, I don't see it online yet.
>
> That is because we still have not any automatic deployment on
> build.macports.org for any of our websites. I am irregularly running
> these jobs from my local buildbot setup on my personal MacBook every few
> weeks, which is the only reason they are not completely outdated.
>
> I triggered another run of these jobs, so these changes should be
> available shortly online.

We now have one of the new "jobs" running on the buildbot, so hopefully adding the others will be easy.

Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Enrico Maria Crisostomo
Thanks Ryan for the explanation.

> On 7 Mar 2018, at 19:51, Ryan Schmidt <[hidden email]> wrote:
>
>
> On Mar 7, 2018, at 07:32, Rainer Müller wrote:
>
>> On 2018-03-07 11:32, Enrico Maria Crisostomo wrote:
>>> This is a little bit offtopic IMHO, anyway: I found surprising too that
>>> the github portgroup was not documented, given the high percentage of
>>> software that I build out of github repositories, and that's why I
>>> recently contributed the documentation of the github portgroup:
>>>
>>> https://github.com/macports/macports-guide/pull/12
>>>
>>> It has been merged some days ago. Yet, I don't see it online yet.
>>
>> That is because we still have not any automatic deployment on
>> build.macports.org for any of our websites. I am irregularly running
>> these jobs from my local buildbot setup on my personal MacBook every few
>> weeks, which is the only reason they are not completely outdated.
>>
>> I triggered another run of these jobs, so these changes should be
>> available shortly online.
>
> We now have one of the new "jobs" running on the buildbot, so hopefully adding the others will be easy.
>

Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Enrico Maria Crisostomo
In reply to this post by Enrico Maria Crisostomo
Ken,

I pushed to GitHub a skeleton of the idea:

    https://github.com/emcrisostomo/macports-utils

I've moved what I'm using to a new script to see what the end result looked like. If you want to try it, just grab the release tarball here (if you don't have the Autotools installed):

    ./configure && make install

otherwise just clone the repo, bootstrap it and use it:

    ./autogen.sh && ./configure && make install

An example:

    $ port-gen --url https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz

outputs:

    # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
   
    PortSystem          1.0
    PortGroup           github 1.0

    github.setup        emcrisostomo semver-utils 1.1.3
    github.tarball_from releases

    categories          changeme
    platforms           darwin
    license             GPL-3

    maintainers         obfuscated-maintainer-mail \
                        openmaintainer
    description         Add a short port description here.
    long_description    Add a long port description here.

    homepage            https://github.com/emcrisostomo/semver-utils

    checksums           md5    e65be62dc9e25af8aa467aa99cde1e00 \
                        rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
                        sha256 888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
                        size   514429

    livecheck.url       ${github.homepage}/releases/latest


The template is intuitive:

    $ cat /usr/local/share/macports-utils/templates/github.in
    # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

    PortSystem          1.0
    PortGroup           github 1.0

    github.setup        ${GITHUB_AUTHOR} ${PORT_NAME} ${PORT_VERSION}
    github.tarball_from releases
   
    categories          changeme
    platforms           darwin
    license             GPL-3

    maintainers         obfuscated-maintainer-mail \\\\
                        openmaintainer
    description         Add a short port description here.
    long_description    Add a long port description here.

    homepage            ${GITHUB_REPO_URL}

    checksums           md5    ${CHECKSUM_MD5} \\\\
                        rmd160 ${CHECKSUM_RMD160} \\\\
                        sha256 ${CHECKSUM_SHA256} \\\\
                        size   ${CHECKSUM_SIZE}

    livecheck.url       \${github.homepage}/releases/latest

> On 7 Mar 2018, at 10:23, Enrico Maria Crisostomo <[hidden email]> wrote:
>
> Hi Ken,
>
> I think it's a great idea. I'm maintaining a lot of software both on my personal machines and at work using MacPorts and I've setup some local Makefile targets to do some of this work. Perhaps I could contribute some work on this area, but first I'd like to know what is it that we'd like to have.
>
> What is currently working for me is a very simple, template-based approach. Since 90% of the time I'm dealing with ports using the github portgroup, I'm basically calculating the relevant variables (port name, version, checksums, etc.) and fill in a template. The template is a file where shell expansion is performed (i.e.: variables in the form ${VAR}) are expanded.) I think this approach could be both very useful and simple to maintain if we had a list of pre-defined templates. It doesn't scale well with the number of different templates to support and their complexity. If we needed templates to have logic in it (flow control, expressions, etc.),then my guts say we should move to a suitable templating engine (or a different kind of solution altogether).
>
> I've had a look at portfile-gen and it's a nice source of inspiration. Yet, I'm not convinced that outputting text directly from the program flow is either easy to read or easily maintainable, especially if complexity goes up.
>
> portfile-gen, if I understand correctly, supports the following groups:
>
>  * perl5
>  * php
>  * python
>  * ruby
>  * github
>
> If we extracted a template for each one of those groups (or more templates in case a group may have multiple templates), then the effort required to come up with a working prototype would be implementing the functions required to populate the environment (i.e.: downloading a file if needed, calculating checksums, examining it). Perhaps we would have to split a Portfile in multiple segments, and generate each segment from a specific template (e.g.: if a port requires GNU Autotools).
>
> What are your gut feelings about a template-based solution? Do you think it would be able to manage the level of Portfile complexity we desire for this tool?
>
> Cheers,
> --
> Enrico
>
>
>> On 7 Mar 2018, at 01:27, Ken Cunningham <[hidden email]> wrote:
>>
>>
>> On 2018-03-06, at 7:19 AM, Joshua Root wrote:
>>
>>>
>>> There's a fairly basic tool called portfile-gen in contrib. Improvements
>>> are certainly welcome.
>>
>> That is indeed a step towards what I was thinking of.
>>
>> I'll look at this.
>>
>> Ken
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Rainer Müller-4
On 2018-03-09 20:22, Enrico Maria Crisostomo wrote:

> I pushed to GitHub a skeleton of the idea:
>
>     https://github.com/emcrisostomo/macports-utils
>
> I've moved what I'm using to a new script to see what the end result looked like. If you want to try it, just grab the release tarball here (if you don't have the Autotools installed):
>
>     ./configure && make install
>
> otherwise just clone the repo, bootstrap it and use it:
>
>     ./autogen.sh && ./configure && make install
>
> An example:
>
>     $ port-gen --url https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz

Nice work! Looks already very polished for a prototype.

I think it would be even nicer to just specify the URL to the project
(or to the tag) and it will find the latest distfile to download itself.

However, when we are going to add more templates, lots of options will
be repeated in each of them. If we are later going to change any of
these, we have to do this in all templates. I am not sure how much of a
problem that will be.

Also, such a template approach will never allow to combine different
features, like using github port group for fetching, but the python port
group for building the port.

Rainer
Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Enrico Maria Crisostomo

> On 11 Mar 2018, at 16:40, Rainer Müller <[hidden email]> wrote:
>
> On 2018-03-09 20:22, Enrico Maria Crisostomo wrote:
>> I pushed to GitHub a skeleton of the idea:
>>
>>    https://github.com/emcrisostomo/macports-utils
>>
>> I've moved what I'm using to a new script to see what the end result looked like. If you want to try it, just grab the release tarball here (if you don't have the Autotools installed):
>>
>>    ./configure && make install
>>
>> otherwise just clone the repo, bootstrap it and use it:
>>
>>    ./autogen.sh && ./configure && make install
>>
>> An example:
>>
>>    $ port-gen --url https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
>
> Nice work! Looks already very polished for a prototype.

Thanks Rainer.

>
> I think it would be even nicer to just specify the URL to the project
> (or to the tag) and it will find the latest distfile to download itself.

Yes.

>
> However, when we are going to add more templates, lots of options will
> be repeated in each of them. If we are later going to change any of
> these, we have to do this in all templates. I am not sure how much of a
> problem that will be.
>
> Also, such a template approach will never allow to combine different
> features, like using github port group for fetching, but the python port
> group for building the port.
>

The prototype uses just one monolithic template, but I'd expect the following to be true:

  * Templates may not need to be one per possible output, they could be finer grained and get concatenated.

  * Dynamic content could be generated and interleaved with template output, if necessary.

I think I share your feelings about templates, and I have little idea about how complex the output can/should be.


> Rainer

Reply | Threaded
Open this post in threaded view
|

Re: request for port create command, to build a portfile from a URL

Ryan Schmidt-24
In reply to this post by Enrico Maria Crisostomo

On Mar 9, 2018, at 13:22, Enrico Maria Crisostomo wrote:

> I pushed to GitHub a skeleton of the idea:
>
>    https://github.com/emcrisostomo/macports-utils
>
> I've moved what I'm using to a new script to see what the end result looked like. If you want to try it, just grab the release tarball here (if you don't have the Autotools installed):
>
>    ./configure && make install
>
> otherwise just clone the repo, bootstrap it and use it:
>
>    ./autogen.sh && ./configure && make install
>
> An example:
>
>    $ port-gen --url https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
>
> outputs:
>
>    # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
>
>    PortSystem          1.0
>    PortGroup           github 1.0
>
>    github.setup        emcrisostomo semver-utils 1.1.3
>    github.tarball_from releases
>
>    categories          changeme
>    platforms           darwin
>    license             GPL-3
>
>    maintainers         obfuscated-maintainer-mail \
>                        openmaintainer
>    description         Add a short port description here.
>    long_description    Add a long port description here.
>
>    homepage            https://github.com/emcrisostomo/semver-utils
>
>    checksums           md5    e65be62dc9e25af8aa467aa99cde1e00 \
>                        rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
>                        sha256 888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
>                        size   514429
>
>    livecheck.url       ${github.homepage}/releases/latest

The github portgroup takes care of setting the homepage and livecheck for you.


12