"cask" ports just keep on rolling in...

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

"cask" ports just keep on rolling in...

Ken Cunningham
With no plan, we’ll just keep getting more and more of these.

<https://github.com/macports/macports-ports/pull/9936>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.

Yuk.

Ken

Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Blake Garner-2
I also think this would be highly desirable. Would it be another portgroup that needs to be written in tcl? 

On Fri, Feb 5, 2021 at 11:01 AM Ken Cunningham <[hidden email]> wrote:
With no plan, we’ll just keep getting more and more of these.

<https://github.com/macports/macports-ports/pull/9936>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.

Yuk.

Ken

Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
In reply to this post by Ken Cunningham
What would we really need to do this properly?

The current fetch, checksum phases are OK.

The extract phase can be used as is for some software (simple copies), but for other software we don’t want to extract it, we want to run the installer.

The configure and build phases need to be overridden and disappear.

The destroot phase needs to — what — record the files that will be installed in some kind of an index file? And also know about the stuff that gets installed into ~/Library, etc.

Then run the “cask” destroot without installing any files into the actual destroot:

copy the apps and stuff
or 
run the installer
or
extract from the pkg
or 
… 

and then NOT have the entire software repackaged into a MP archive file to be stored 12 different times…

Or some such plan

Best,

Ken





On Feb 5, 2021, at 11:00 AM, Ken Cunningham <[hidden email]> wrote:

With no plan, we’ll just keep getting more and more of these.

<https://github.com/macports/macports-ports/pull/9936>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.

Yuk.

Ken


Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Jason Liu
The destroot phase needs to — what — record the files that will be installed in some kind of an index file? And also know about the stuff that gets installed into ~/Library, etc.

You would probably need to get the list of files installed by the installer by running either pkgutil --files or lsbom, if we're talking about a PKG installer. (Or maybe it would be easier to just grab the bom file in its entirety.) Typically running an ls -R on the .app bundle inside a DMG installer would be sufficient for those kinds of installers.

But then this begs the question: what happens to all of the work that went into the build-from-source packages? Wouldn't this end up rendering the hundreds of hours I spent getting the Blender package to work a complete waste?

-- 
Jason Liu


On Fri, Feb 5, 2021 at 5:31 PM Ken Cunningham <[hidden email]> wrote:
What would we really need to do this properly?

The current fetch, checksum phases are OK.

The extract phase can be used as is for some software (simple copies), but for other software we don’t want to extract it, we want to run the installer.

The configure and build phases need to be overridden and disappear.

The destroot phase needs to — what — record the files that will be installed in some kind of an index file? And also know about the stuff that gets installed into ~/Library, etc.

Then run the “cask” destroot without installing any files into the actual destroot:

copy the apps and stuff
or 
run the installer
or
extract from the pkg
or 
… 

and then NOT have the entire software repackaged into a MP archive file to be stored 12 different times…

Or some such plan

Best,

Ken





On Feb 5, 2021, at 11:00 AM, Ken Cunningham <[hidden email]> wrote:

With no plan, we’ll just keep getting more and more of these.

<https://github.com/macports/macports-ports/pull/9936>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.

Yuk.

Ken


Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
The idea of MacPorts is to manage installing inter-related libraries and executables.

But people want to use it to install prebuilt binaries as well, like homebrew does for cask installs.

MP has not decided if it will or will not accept these, but they just keep rolling in more and more anyway.

Installing them as "ports" is kinda silly, and wasteful, but simple, practical, and slides them in under the radar.

Having an actual plan for these would be better.

If software _can_ be built, we want to do it that way, so (IMHO) your work is not wasted. 

We get many benefits, I believe, from building things ourselves.

Whether we do or don't accept binary "cask" installs ins not up to me -- but I'm just point out that they are coming in anyway so a plan would be good, esp for the multi-megabyte behemoths that are out there.


Ken




On 2021-02-05, at 2:58 PM, Jason Liu wrote:

The destroot phase needs to — what — record the files that will be installed in some kind of an index file? And also know about the stuff that gets installed into ~/Library, etc.

You would probably need to get the list of files installed by the installer by running either pkgutil --files or lsbom, if we're talking about a PKG installer. (Or maybe it would be easier to just grab the bom file in its entirety.) Typically running an ls -R on the .app bundle inside a DMG installer would be sufficient for those kinds of installers.

But then this begs the question: what happens to all of the work that went into the build-from-source packages? Wouldn't this end up rendering the hundreds of hours I spent getting the Blender package to work a complete waste?

-- 
Jason Liu


On Fri, Feb 5, 2021 at 5:31 PM Ken Cunningham <[hidden email]> wrote:
What would we really need to do this properly?

The current fetch, checksum phases are OK.

The extract phase can be used as is for some software (simple copies), but for other software we don’t want to extract it, we want to run the installer.

The configure and build phases need to be overridden and disappear.

The destroot phase needs to — what — record the files that will be installed in some kind of an index file? And also know about the stuff that gets installed into ~/Library, etc.

Then run the “cask” destroot without installing any files into the actual destroot:

copy the apps and stuff
or 
run the installer
or
extract from the pkg
or 
… 

and then NOT have the entire software repackaged into a MP archive file to be stored 12 different times…

Or some such plan

Best,

Ken





On Feb 5, 2021, at 11:00 AM, Ken Cunningham <[hidden email]> wrote:

With no plan, we’ll just keep getting more and more of these.

<https://github.com/macports/macports-ports/pull/9936>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.

Yuk.

Ken



Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

ryandesign2
Administrator
In reply to this post by Ken Cunningham


On Feb 5, 2021, at 13:00, Ken Cunningham wrote:

> With no plan, we’ll just keep getting more and more of these.

I'm not aware of a huge influx of these, but I'm also not really paying attention to the PR queue. And I'm not intending to get drawn into this discussion of binary ports again.


> <https://github.com/macports/macports-ports/pull/9936>
>
> This kind of port just repackages the DMG into many tgz archives.
>
> It’s wasteful. People want them.
>
> What we should have instead of this is some kinda tech that
>
> 1. downloads the DMG
> 2. installs the app
> 3. records some way of uninstalling everything

MacPorts already does all these things... The submitted Portfile works fine, presumably. There's no need to reinvent the existing MacPorts functionality that does all these things.


> What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.
>
> Yuk.

If your objection is that we waste server space storing several copies of the same thing, then we could invent a new way for a portfile to indicate that it does not want binary archives stored on the server. It could be a new separate keyword or a new pseudo license type, like we already have for "nomirror" which tells the buildbot not to mirror the distfiles.

A port like the one we're talking about in the above PR could set e.g. "license GPL-2 nopackage", and buildbot could be modified to understand that this means that it should not upload the packages.

MacPorts base would still try to download the nonexistent package. MacPorts base currently does not use license values except to display to the user and has no idea if a port is distributable. Distributability is handled by a separate script. It should be integrated into base so that base can know if something could have been distributed, and if it could not have, then it shouldn't even try to download it. https://trac.macports.org/ticket/60315

We might also want to modify the MacPorts base command line "-b" binary only mode to allow installing these "nopackage" ports rather than error out as it would currently do.

If the few steps like disabling the configure and build phases and adding the hypothetical "nopackage" to the license line are too much work for a portfile submitter to do, a portgroup could be created that does those things. But you have often complained about "magic" portgroups doing things you didn't know about or didn't expect, so there is something to be said for ports doing what they need to do explicitly, when it's not so many steps, and when it's not always the same steps needing to be done. For example, surely each port will still need to specify what the archs of the binary files are, what license they're under, what type of distfile it is and where it is, and which files inside needs to be copied where.

Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham


> On Feb 5, 2021, at 10:00 PM, Ryan Schmidt <[hidden email]> wrote:
>
>
>
> On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
>
>> With no plan, we’ll just keep getting more and more of these.
>
> I'm not aware of a huge influx of these, but I'm also not really paying attention to the PR queue. And I'm not intending to get drawn into this discussion of binary ports again.

The last discussion didn’t get anywhere past the appropriate naming scheme :> We never even got into implementation details.

>> This kind of port just repackages the DMG into many tgz archives.
>>
>> It’s wasteful. People want them.
>>
>> What we should have instead of this is some kinda tech that
>>
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
>
> MacPorts already does all these things... The submitted Portfile works fine, presumably. There's no need to reinvent the existing MacPorts functionality that does all these things.
>

Well, IMHO there is, but I’ve looked at it quite a bit.

Look back a month or so and see my post about a “cask” port for SageMath for an idea of why this won’t work (well) in general.

For trivially simple ports, a few MB or less for a *.app that copies into /Applications/MacPorts — yeah, who cares?

>
>> What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.
>>
>> Yuk.
>
> If your objection is that we waste server space storing several copies of the same thing, then we could invent a new way for a portfile to indicate that it does not want binary archives stored on the server. It could be a new separate keyword or a new pseudo license type, like we already have for "nomirror" which tells the buildbot not to mirror the distfiles.
> A port like the one we're talking about in the above PR could set e.g. "license GPL-2 nopackage", and buildbot could be modified to understand that this means that it should not upload the packages.

Yes, that would help, if these binary ports start getting to any size.

>
> MacPorts base would still try to download the nonexistent package. MacPorts base currently does not use license values except to display to the user and has no idea if a port is distributable. Distributability is handled by a separate script. It should be integrated into base so that base can know if something could have been distributed, and if it could not have, then it shouldn't even try to download it. https://trac.macports.org/ticket/60315
>

Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup could set “archive_sites” to “” I guess, perhaps more easily.

> We might also want to modify the MacPorts base command line "-b" binary only mode to allow installing these "nopackage" ports rather than error out as it would currently do.
>
> If the few steps like disabling the configure and build phases and adding the hypothetical "nopackage" to the license line are too much work for a portfile submitter to do, a portgroup could be created that does those things. But you have often complained about "magic" portgroups doing things you didn't know about or didn't expect, so there is something to be said for ports doing what they need to do explicitly, when it's not so many steps, and when it's not always the same steps needing to be done. For example, surely each port will still need to specify what the archs of the binary files are, what license they're under, what type of distfile it is and where it is, and which files inside needs to be copied where.
>

It’s not that it’s too much work. It’s that these things are very simple, and people submit them but don’t know all these details you mention.

And —

this won’t work for larger ports too well (see SageMath message for an extreme example)
we don’t want to run rev-upgrade on them (surprises, surprises, surprises — eg SageMath again)
we want to be able to run a pkg-installer into some destination…. and nobody except three people know how to do that right
we want to be able to uninstall all the cruft the software installs in ~/Library, ~/Preferences, etc easily

So — it’s doable. What we do now is — meh.

IF we are going to do it, we should do it right.

And — we STILL have no naming scheme, so a user will have NO IDEA if he’s downloading an app from some website on the internet rather than something build by MacPorts.

And I think we should have a good way of identifying them, whatever it is. And yes, I still think identifying them by using a “+prebuilt” variant name is not the way to do it, if for no other reason than that might propagate down through all kinds of ports as “+prebuilt” and nobody wants that, it carries over from installation to upgrade and nobody wants that, and it is not (IMHO) logical to have something specified as “-prebuilt” if there is no “-prebuilt” and on and on and on for why I think this is just not clear enough …

but the NAME is the least of the issues, TBH.

Ken


Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

ryandesign2
Administrator
On Feb 6, 2021, at 02:28, Ken Cunningham wrote:

>>> This kind of port just repackages the DMG into many tgz archives.
>>>
>>> It’s wasteful. People want them.
>>>
>>> What we should have instead of this is some kinda tech that
>>>
>>> 1. downloads the DMG
>>> 2. installs the app
>>> 3. records some way of uninstalling everything
>>
>> MacPorts already does all these things... The submitted Portfile works fine, presumably. There's no need to reinvent the existing MacPorts functionality that does all these things.
>
> Well, IMHO there is, but I’ve looked at it quite a bit.
>
> Look back a month or so and see my post about a “cask” port for SageMath for an idea of why this won’t work (well) in general.

I had not seen that part of the previous discussion because I stopped following it, but going back and searching for "sagemath" in those messages now, I still don't see any reason why what MacPorts can already do would not work for these types of ports.

In that example, the sagemath dmg was 1.5GB and the installed size was 4GB. This is a stupid size for a software package to be, but that's up to its developers. Hopefully there are not more than one or two prospective ports that are so exorbitantly large, so I would not let this extreme example dictate what we do.

The complaint was that MacPorts extracts that 4GB into the work directory, then compresses the contents of the destroot into a package, then activates the package, using unnecessary time and disk space. Let's start by acknowledging that this works just fine, so it does not need to be changed. It is true that this takes a little time and temporarily disk space, but as you know the reason why MacPorts does this is so that the user can deactivate and later re-activate the port as needed.

It was proposed there that MacPorts should instead somehow use the original dmg distfile as the archive, skipping the normal extract, patch, configure, build, destroot, install (create package) and activate (extract package) steps. Based on what I've read so far, that doesn't sound like a great idea, since you'd be replacing a large amount of already-working MacPorts base code with new code that you'd have to test.

You're either requiring the user to keep that distfile around in order to be able to reactivate the port (which would be highly unexpected), or you have MacPorts copy the distfile into the software directory, which could be done, but then the port has to do all of that stuff during the activate phase, has to contain code that knows what files to pull off the dmg and where to put them at activate time, losing the safety net that destroot provides, making development of the port more difficult. When developing a port you want to be able to "sudo port destroot" and then check the contents of destroot to make sure everything is correct, and you want to be able to "sudo port clean" to delete it all with the assurance that nothing outside of the work directory was touched. We already have that today; I don't see how the proposed new mechanism would allow that to continue to be the case.

A point was raised that the original dmg is compressed already, therefore keeping and using it was desirable, but I have to point out that dmgs can be uncompressed, or could be compressed with less-efficient methods than our bz2 (though, granted, these days most of them probably are compressed, some probably with more-efficient methods like lzma). There's also the possibility that the original dmg contains additional files that we don't care about, so requiring the user keep the original dmg around might be requiring them to keep unneeded files around.

Another suggestion was raised there of preventing distfiles from being mirrored. I see no reason why that should be tied to this type of port. On the contrary, if we don't provide packages, then we absolutely want to mirror distfiles (license permitting, of course) for all the usual reasons, such as in case the original site is offline. With most ports, most users use our packages and it doesn't matter if the distfile is inaccessible. But with these ports, if we are talking about not providing our own packages and forcing all users to use the original distfile, then it very much matters if the distfile is inaccessible.


> It’s not that it’s too much work. It’s that these things are very simple, and people submit them but don’t know all these details you mention.

They won't know about the hypothetical new portgroup either unless they read about it or we tell them about it. So it comes down to a matter of keeping our documentation current and readable, and I'm the first to admit that our documentation is not in the best shape that it could be in.


> this won’t work for larger ports too well (see SageMath message for an extreme example)

> we don’t want to run rev-upgrade on them (surprises, surprises, surprises — eg SageMath again)

This might be a valid point. If so, we might enhance MacPorts to provide a way for a port to opt out of rev-upgrade. That would be simple enough. On the other hand, are you saying that sagemath is reported as broken by rev-upgrade, and that either we want to accept that brokenness? (why?) or that rev-upgrade is wrong? (in what way? why don't we fix that?)

It is not inconceivable, in fact it seems likely to me, that a port developer might want to patch a binary so that it finds MacPorts libraries. Given the popularity of Homebrew, someone might distribute a binary that links with Homebrew openssl, for example, and we might want to edit that with install_name_tool to point to MacPorts openssl instead. Or the binary we install might include libraries, and we would want to change their install_names to MacPorts paths. I do this in the oracle-instantclient and libxl ports for example. I would want rev-upgrade to let me know if something is amiss with that.


> we want to be able to run a pkg-installer into some destination…. and nobody except three people know how to do that right

True, some ports might want to install using the installer. macOS includes the `installer` command for doing this. I don't recall the right arguments offhand but I'm sure it's described in the manpage. If it is though that a portgroup that runs installer for you would be an easier interface to this, by all means create one. Or the functionality could be added to base, if that's more appropriate.

It would be nice if MacPorts base "decompress this file" functionality were more generalized (automatically figure out what program to use) and abstracted out to be available to ports at any time. Base already contains two separate implementations, one to extract distfiles and one to extract archives. Adding installer pkg/mpkg files to the list of file types that we know how to extract would be reasonable. But installers can also contain scripts which it might be important to run, and using `installer` would be the way to ensure that that happens.


> we want to be able to uninstall all the cruft the software installs in ~/Library, ~/Preferences, etc easily

I did not see in the messages about sagemath any mention of ~/Library or ~/Preferences cruft. ~/Preferences is where user preferences go; just as with settings files that other ports might install into /opt/local/etc, it would be unexpected for a port to delete them. I would have to take any additions to ~/Library on a case by case basis, but I would assume that any files that any program installed by a port puts into ~/Library after installation are put there intentionally and that it is right and proper for it to be doing so and that it's not up to us to delete them. This would be no different from a user installing an app manually by dragging and dropping. Anything that a port installs during installation would be handled as part of destroot of course, though of course it wouldn't be installing things in user-specific paths like ~.


Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
In reply to this post by Ken Cunningham
try the sagemath one if you like. It's all set, for a newish system at present:

https://github.com/kencu/myports/blob/master/binary/sagemath-binary/Portfile

It is a stress test, for sure -- and I fully admit you may know simple tweaks to make it much more efficient.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

ryandesign2
Administrator


On Feb 6, 2021, at 04:15, Ken Cunningham wrote:

> try the sagemath one if you like. It's all set, for a newish system at present:
>
> https://github.com/kencu/myports/blob/master/binary/sagemath-binary/Portfile
>
> It is a stress test, for sure -- and I fully admit you may know simple tweaks to make it much more efficient.

I did see that portfile in the previous discussion. I don't feel a need to try it myself. The portfile doesn't look overly more complicated than most ports.

Some tweaks could of course be made to the portfile.

To save some time and disk space usage, the app could be moved to the destroot rather than copied.

Something should be done to ensure that the port errors out on earlier OS versions on which it won't work. This is not a binary-port-specific issue; we've long wanted MacPorts base to be improved to handle this better.

If the binary is x86_64 only supported_archs should reflect that. If the binary is arm64/x86_64 universal then the universal variant should be enabled and for nonuninversal builds the "other" arch should be removed from all files with lipo. The libxl port gives an example. This could be tedious if there are many files, and could be a candidate for including in a hypothetical portgroup.

You're currently setting version conditionally. version is a required keyword; it must be set in all cases or portindex will probably fail.

Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
In reply to this post by Ken Cunningham
those are the small things.

I stopped when the broad strokes, as it is now, were a unworkable disaster.

K
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

ryandesign2
Administrator


On Feb 6, 2021, at 09:17, Ken Cunningham wrote:

> those are the small things.
>
> I stopped when the broad strokes, as it is now, were a unworkable disaster.

I don't understand which aspects are an unworkable disaster. As I said, based on what you've said so far, it seems like everything should work fine the way it is now, albeit in a possibly non-optimal way due to some excessive disk use during install.

Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
In reply to this post by Ken Cunningham
true, without ever trying it, you couldn't know
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Mark Anderson
In reply to this post by Ken Cunningham
Yeah, it seems like a lot of the stuff is in place, but we just need some tweaks. I like the idea of a portgroup like BinaryOnly or something, but what else needs to happen?

I'd be more than happy to help, what needs to be done? Should we maybe take to Trac to get a to-do and proposal going?
 
—Mark
_______________________
Mark E. Anderson <[hidden email]>



On Sat, Feb 6, 2021 at 3:28 AM Ken Cunningham <[hidden email]> wrote:


> On Feb 5, 2021, at 10:00 PM, Ryan Schmidt <[hidden email]> wrote:
>
>
>
> On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
>
>> With no plan, we’ll just keep getting more and more of these.
>
> I'm not aware of a huge influx of these, but I'm also not really paying attention to the PR queue. And I'm not intending to get drawn into this discussion of binary ports again.

The last discussion didn’t get anywhere past the appropriate naming scheme :> We never even got into implementation details.

>> This kind of port just repackages the DMG into many tgz archives.
>>
>> It’s wasteful. People want them.
>>
>> What we should have instead of this is some kinda tech that
>>
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
>
> MacPorts already does all these things... The submitted Portfile works fine, presumably. There's no need to reinvent the existing MacPorts functionality that does all these things.
>

Well, IMHO there is, but I’ve looked at it quite a bit.

Look back a month or so and see my post about a “cask” port for SageMath for an idea of why this won’t work (well) in general.

For trivially simple ports, a few MB or less for a *.app that copies into /Applications/MacPorts — yeah, who cares?

>
>> What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.
>>
>> Yuk.
>
> If your objection is that we waste server space storing several copies of the same thing, then we could invent a new way for a portfile to indicate that it does not want binary archives stored on the server. It could be a new separate keyword or a new pseudo license type, like we already have for "nomirror" which tells the buildbot not to mirror the distfiles.
> A port like the one we're talking about in the above PR could set e.g. "license GPL-2 nopackage", and buildbot could be modified to understand that this means that it should not upload the packages.

Yes, that would help, if these binary ports start getting to any size.

>
> MacPorts base would still try to download the nonexistent package. MacPorts base currently does not use license values except to display to the user and has no idea if a port is distributable. Distributability is handled by a separate script. It should be integrated into base so that base can know if something could have been distributed, and if it could not have, then it shouldn't even try to download it. https://trac.macports.org/ticket/60315
>

Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup could set “archive_sites” to “” I guess, perhaps more easily.

> We might also want to modify the MacPorts base command line "-b" binary only mode to allow installing these "nopackage" ports rather than error out as it would currently do.
>
> If the few steps like disabling the configure and build phases and adding the hypothetical "nopackage" to the license line are too much work for a portfile submitter to do, a portgroup could be created that does those things. But you have often complained about "magic" portgroups doing things you didn't know about or didn't expect, so there is something to be said for ports doing what they need to do explicitly, when it's not so many steps, and when it's not always the same steps needing to be done. For example, surely each port will still need to specify what the archs of the binary files are, what license they're under, what type of distfile it is and where it is, and which files inside needs to be copied where.
>

It’s not that it’s too much work. It’s that these things are very simple, and people submit them but don’t know all these details you mention.

And —

this won’t work for larger ports too well (see SageMath message for an extreme example)
we don’t want to run rev-upgrade on them (surprises, surprises, surprises — eg SageMath again)
we want to be able to run a pkg-installer into some destination…. and nobody except three people know how to do that right
we want to be able to uninstall all the cruft the software installs in ~/Library, ~/Preferences, etc easily

So — it’s doable. What we do now is — meh.

IF we are going to do it, we should do it right.

And — we STILL have no naming scheme, so a user will have NO IDEA if he’s downloading an app from some website on the internet rather than something build by MacPorts.

And I think we should have a good way of identifying them, whatever it is. And yes, I still think identifying them by using a “+prebuilt” variant name is not the way to do it, if for no other reason than that might propagate down through all kinds of ports as “+prebuilt” and nobody wants that, it carries over from installation to upgrade and nobody wants that, and it is not (IMHO) logical to have something specified as “-prebuilt” if there is no “-prebuilt” and on and on and on for why I think this is just not clear enough …

but the NAME is the least of the issues, TBH.

Ken


Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
although I was concerned about getting this pattern right before we had too many of these to fix, it does seem the admins feel there's really no issue to worry about here.

So I guess we just open the gate and let them in. There is no recommendation for a requirement for a naming convention or other identifier.

I have gone ahead and committed the ones in the PR queue I found there. Feel free to use those as a template.

K

On Feb 6, 2021, at 18:16, Mark Anderson <[hidden email]> wrote:

Yeah, it seems like a lot of the stuff is in place, but we just need some tweaks. I like the idea of a portgroup like BinaryOnly or something, but what else needs to happen?

I'd be more than happy to help, what needs to be done? Should we maybe take to Trac to get a to-do and proposal going?
 
—Mark
_______________________
Mark E. Anderson <[hidden email]>



On Sat, Feb 6, 2021 at 3:28 AM Ken Cunningham <[hidden email]> wrote:


> On Feb 5, 2021, at 10:00 PM, Ryan Schmidt <[hidden email]> wrote:
>
>
>
> On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
>
>> With no plan, we’ll just keep getting more and more of these.
>
> I'm not aware of a huge influx of these, but I'm also not really paying attention to the PR queue. And I'm not intending to get drawn into this discussion of binary ports again.

The last discussion didn’t get anywhere past the appropriate naming scheme :> We never even got into implementation details.

>> This kind of port just repackages the DMG into many tgz archives.
>>
>> It’s wasteful. People want them.
>>
>> What we should have instead of this is some kinda tech that
>>
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
>
> MacPorts already does all these things... The submitted Portfile works fine, presumably. There's no need to reinvent the existing MacPorts functionality that does all these things.
>

Well, IMHO there is, but I’ve looked at it quite a bit.

Look back a month or so and see my post about a “cask” port for SageMath for an idea of why this won’t work (well) in general.

For trivially simple ports, a few MB or less for a *.app that copies into /Applications/MacPorts — yeah, who cares?

>
>> What we have instead is a repackaging of the DMG into many, identical, system-specific archive bundles.
>>
>> Yuk.
>
> If your objection is that we waste server space storing several copies of the same thing, then we could invent a new way for a portfile to indicate that it does not want binary archives stored on the server. It could be a new separate keyword or a new pseudo license type, like we already have for "nomirror" which tells the buildbot not to mirror the distfiles.
> A port like the one we're talking about in the above PR could set e.g. "license GPL-2 nopackage", and buildbot could be modified to understand that this means that it should not upload the packages.

Yes, that would help, if these binary ports start getting to any size.

>
> MacPorts base would still try to download the nonexistent package. MacPorts base currently does not use license values except to display to the user and has no idea if a port is distributable. Distributability is handled by a separate script. It should be integrated into base so that base can know if something could have been distributed, and if it could not have, then it shouldn't even try to download it. https://trac.macports.org/ticket/60315
>

Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup could set “archive_sites” to “” I guess, perhaps more easily.

> We might also want to modify the MacPorts base command line "-b" binary only mode to allow installing these "nopackage" ports rather than error out as it would currently do.
>
> If the few steps like disabling the configure and build phases and adding the hypothetical "nopackage" to the license line are too much work for a portfile submitter to do, a portgroup could be created that does those things. But you have often complained about "magic" portgroups doing things you didn't know about or didn't expect, so there is something to be said for ports doing what they need to do explicitly, when it's not so many steps, and when it's not always the same steps needing to be done. For example, surely each port will still need to specify what the archs of the binary files are, what license they're under, what type of distfile it is and where it is, and which files inside needs to be copied where.
>

It’s not that it’s too much work. It’s that these things are very simple, and people submit them but don’t know all these details you mention.

And —

this won’t work for larger ports too well (see SageMath message for an extreme example)
we don’t want to run rev-upgrade on them (surprises, surprises, surprises — eg SageMath again)
we want to be able to run a pkg-installer into some destination…. and nobody except three people know how to do that right
we want to be able to uninstall all the cruft the software installs in ~/Library, ~/Preferences, etc easily

So — it’s doable. What we do now is — meh.

IF we are going to do it, we should do it right.

And — we STILL have no naming scheme, so a user will have NO IDEA if he’s downloading an app from some website on the internet rather than something build by MacPorts.

And I think we should have a good way of identifying them, whatever it is. And yes, I still think identifying them by using a “+prebuilt” variant name is not the way to do it, if for no other reason than that might propagate down through all kinds of ports as “+prebuilt” and nobody wants that, it carries over from installation to upgrade and nobody wants that, and it is not (IMHO) logical to have something specified as “-prebuilt” if there is no “-prebuilt” and on and on and on for why I think this is just not clear enough …

but the NAME is the least of the issues, TBH.

Ken


Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Clemens Lang-2
Hi Ken,

On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
> although I was concerned about getting this pattern right before we
> had too many of these to fix, it does seem the admins feel there's
> really no issue to worry about here.

I don't like this tone, Ken. "The admins" have as much obligation to
provide infrastructure as anybody else in this project, which is none.

If you feel repackaging binary archives is a thing MacPorts should
support better, please invest the time to come up with patches that do
this, or find somebody that will.


> So I guess we just open the gate and let them in. There is no
> recommendation for a requirement for a naming convention or other
> identifier.

Personally, I don't like this trend at all. It always used to be
MacPorts' policy to compile from source except in cases where Apple's
limitations made this impossible (e.g. because valid signatures with a
developer certificate were required and an ad-hoc signature would not
work).

Now, we're apparently shipping binaries compiled by other people with
other people's toolchains. When I previously installed things from
MacPorts, I knew that I'd either compile those with my own toolchain
locally, or that they had been compiled on MacPorts' buildbots.
Repackaging binaries breaks that assumption and adds additional trusted
third parties. If such parties were infiltrated by supply chain attacks
such as Xcode Ghost, we'd now ship malware via 'port install'.

I do know that we have recently started making more and more exceptions
to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
these approaches to a broken build of a very outdated version, but I'd
like to argue that we should keep asking ourselves the question "should
we really trust this person's toolchain" before merging such ports and
keep pushing for builds from source where those are feasible.

Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
source code that was used to compile a certain binary. Our distfiles
server does that automatically for everything that's compiled from
source, but repackaging a binary that contains compiled GPL or LGPL code
puts us in legal muddy water very quickly.

--
Clemens
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ruben Di Battista
If I can express my opinion, I agree with Clemens.

In my opinion binary only ports should be allowed only for a very restricted subset, where the effort is justified by the extreme complexity of the build (and I'm also I'm not sure about that even, if someone managed to build it, we should be able too ☺) or the port bering binary only (e.g. osxfuse).



On Sun, 7 Feb 2021, 12:17 Clemens Lang, <[hidden email]> wrote:
Hi Ken,

On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
> although I was concerned about getting this pattern right before we
> had too many of these to fix, it does seem the admins feel there's
> really no issue to worry about here.

I don't like this tone, Ken. "The admins" have as much obligation to
provide infrastructure as anybody else in this project, which is none.

If you feel repackaging binary archives is a thing MacPorts should
support better, please invest the time to come up with patches that do
this, or find somebody that will.


> So I guess we just open the gate and let them in. There is no
> recommendation for a requirement for a naming convention or other
> identifier.

Personally, I don't like this trend at all. It always used to be
MacPorts' policy to compile from source except in cases where Apple's
limitations made this impossible (e.g. because valid signatures with a
developer certificate were required and an ad-hoc signature would not
work).

Now, we're apparently shipping binaries compiled by other people with
other people's toolchains. When I previously installed things from
MacPorts, I knew that I'd either compile those with my own toolchain
locally, or that they had been compiled on MacPorts' buildbots.
Repackaging binaries breaks that assumption and adds additional trusted
third parties. If such parties were infiltrated by supply chain attacks
such as Xcode Ghost, we'd now ship malware via 'port install'.

I do know that we have recently started making more and more exceptions
to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
these approaches to a broken build of a very outdated version, but I'd
like to argue that we should keep asking ourselves the question "should
we really trust this person's toolchain" before merging such ports and
keep pushing for builds from source where those are feasible.

Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
source code that was used to compile a certain binary. Our distfiles
server does that automatically for everything that's compiled from
source, but repackaging a binary that contains compiled GPL or LGPL code
puts us in legal muddy water very quickly.

--
Clemens
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Jonathan Stickel-5
In reply to this post by Ken Cunningham
On 2/7/21 04:48 , [hidden email] wrote:

> Hi Ken,
>
> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
>> although I was concerned about getting this pattern right before we
>> had too many of these to fix, it does seem the admins feel there's
>> really no issue to worry about here.
> I don't like this tone, Ken. "The admins" have as much obligation to
> provide infrastructure as anybody else in this project, which is none.
>
> If you feel repackaging binary archives is a thing MacPorts should
> support better, please invest the time to come up with patches that do
> this, or find somebody that will.
>

If my interject, as a long-time user and observer of open-source
software and occasional contributor, this bit of conflict is a common
theme. There is always a tension between "doing it right" and "getting
it done". Ken seems to be leaning towards "getting it done". I
appreciate this a lot. I've been frustrated, at times, over the years by
languishing broken ports, especially when I offered patches/PRs that
received no response. What to do when there isn't consensus on a
particular issue or a new policy, like this one about binary-only ports?
"Doing it right" then means doing nothing at all. When volunteers are
willing to work on something but face this sentiment, they tend to stop
contributing. I don't know the answers but urge caution before things
get personal.

Regarding binary ports, I guess I don't understand what is the point.
Why not require users install the upstream binaries themselves and use
`path:` dependencies for things that depend on them? I know this isn't
ideal because it isn't automated, but I consider that a secondary issue.

Regards,
Jonathan
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

Ken Cunningham
In reply to this post by Clemens Lang-2


> On Feb 7, 2021, at 03:17, Clemens Lang <[hidden email]> wrote:
>
> Hi Ken,
>
>> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
>> although I was concerned about getting this pattern right before we
>> had too many of these to fix, it does seem the admins feel there's
>> really no issue to worry about here.
>
> I don't like this tone, Ken. "The admins" have as much obligation to
> provide infrastructure as anybody else in this project, which is none.

It wasn't meant to be a tone. Just stating the exact facts.

>
> If you feel repackaging binary archives is a thing MacPorts should
> support better, please invest the time to come up with patches that do
> this, or find somebody that will.
>

I did wonder about that, started some thought about how that might proceed, but the clear message was the current system is fine and there is nothing of significance that needs fixing.

>
>> So I guess we just open the gate and let them in. There is no
>> recommendation for a requirement for a naming convention or other
>> identifier.
>
> Personally, I don't like this trend at all. It always used to be
> MacPorts' policy to compile from source except in cases where Apple's
> limitations made this impossible (e.g. because valid signatures with a
> developer certificate were required and an ad-hoc signature would not
> work).
>
> Now, we're apparently shipping binaries compiled by other people with
> other people's toolchains. When I previously installed things from
> MacPorts, I knew that I'd either compile those with my own toolchain
> locally, or that they had been compiled on MacPorts' buildbots.
> Repackaging binaries breaks that assumption and adds additional trusted
> third parties. If such parties were infiltrated by supply chain attacks
> such as Xcode Ghost, we'd now ship malware via 'port install'.

Now that the reality of what this really means comes to the fore, people are starting the be more vocal about their thoughts on it. This is good. These ports have been coming in, with no plan so far.


>
> I do know that we have recently started making more and more exceptions
> to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
> these approaches to a broken build of a very outdated version, but I'd
> like to argue that we should keep asking ourselves the question "should
> we really trust this person's toolchain" before merging such ports and
> keep pushing for builds from source where those are feasible.
>
> Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
> source code that was used to compile a certain binary. Our distfiles
> server does that automatically for everything that's compiled from
> source, but repackaging a binary that contains compiled GPL or LGPL code
> puts us in legal muddy water very quickly.
>
> --
> Clemens



All true.

So new policy coming then?

As I stated at the very beginning months ago, with no plan or guidance, these just keep coming in. I am not championing this, but raising the flag here that there is a potential problem.

If it took a slightly more obvious message about what this really meant to get everyone to notice, I guess I accept that.

K
Reply | Threaded
Open this post in threaded view
|

Re: "cask" ports just keep on rolling in...

ryandesign2
Administrator
On Feb 7, 2021, at 09:59, Ken Cunningham wrote:

> On Feb 7, 2021, at 03:17, Clemens Lang wrote:
>
>> Hi Ken,
>>
>>> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
>>> although I was concerned about getting this pattern right before we
>>> had too many of these to fix, it does seem the admins feel there's
>>> really no issue to worry about here.
>>
>> I don't like this tone, Ken. "The admins" have as much obligation to
>> provide infrastructure as anybody else in this project, which is none.
>
> It wasn't meant to be a tone. Just stating the exact facts.
>

>> If you feel repackaging binary archives is a thing MacPorts should
>> support better, please invest the time to come up with patches that do
>> this, or find somebody that will.
>
> I did wonder about that, started some thought about how that might proceed, but the clear message was the current system is fine and there is nothing of significance that needs fixing.

To clarify a couple things... The managers (Josh, Rainer and myself) are here to guide the project. We've been here longer than most so we can provide historical context about why things are done the way they're done and recommend consistency in how we do things. But our opinions are not meant to be directives that can never be broken. We've granted commit access to you and many others who have demonstrated competency and understanding of MacPorts traditions and we trust you to make sound judgements when it comes to making changes in MacPorts.

I did not mean to convey that I "feel there's really no issue to worry about here" or that "the current system is fine and there is nothing of significance that needs fixing". What I meant to convey was that *based on the concerns you had raised so far* in this thread, which I addressed in turn, I did not understand why you feel that the current situation is an "unworkable disaster". You declined to elaborate, saying "without ever trying it, you couldn't know". I've already spent hours on this thread that I didn't plan on spending, and I don't want to spend more time trying your portfile. If you believe your portfile exposes a flaw in MacPorts base or some other need for changes to base, please describe it.


>>> So I guess we just open the gate and let them in. There is no
>>> recommendation for a requirement for a naming convention or other
>>> identifier.
>>
>> Personally, I don't like this trend at all. It always used to be
>> MacPorts' policy to compile from source except in cases where Apple's
>> limitations made this impossible (e.g. because valid signatures with a
>> developer certificate were required and an ad-hoc signature would not
>> work).
>>
>> Now, we're apparently shipping binaries compiled by other people with
>> other people's toolchains. When I previously installed things from
>> MacPorts, I knew that I'd either compile those with my own toolchain
>> locally, or that they had been compiled on MacPorts' buildbots.
>> Repackaging binaries breaks that assumption and adds additional trusted
>> third parties. If such parties were infiltrated by supply chain attacks
>> such as Xcode Ghost, we'd now ship malware via 'port install'.
>
> Now that the reality of what this really means comes to the fore, people are starting the be more vocal about their thoughts on it. This is good. These ports have been coming in, with no plan so far.

I thought that binary ports were exactly for what Clemens says above: "cases where Apple's limitations made [compiling from source] impossible" (or at least sufficiently difficult). I've said many times that I don't have a problem with that, and we've long had such ports in MacPorts already (e.g. for closed-source software like oracle-instantclient and libxl; osxfuse should be one too). But if we're proposing a more general acceptance of precompiled software, even if we could build it from source, then I think it's reasonable to ask why we would want to allow that.


> So new policy coming then?
>
> As I stated at the very beginning months ago, with no plan or guidance, these just keep coming in. I am not championing this, but raising the flag here that there is a potential problem.
>
> If it took a slightly more obvious message about what this really meant to get everyone to notice, I guess I accept that.

It's not going to be me or the other managers who comes down from high with an edict telling everyone else how to handle this. I for one have demonstrated that I don't yet understand what the concerns are, so I'm not yet equipped to advise what to do. I recommend interested parties have a discussion, as in this mailing list thread, to come to an agreement about what the issues are and how each of them should be addressed. I will warn you that I may not personally be participating in this discussion, which does not mean the issues are not important, just that we all have to decide what we have time to work on, and I may not have time for this.


12