invisible universal variant and merger_must_run_binaries

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

invisible universal variant and merger_must_run_binaries

Ken Cunningham
Just wanted to point out a potential issue with the universal variant
handling.


Let's assume a user on a BigSur Intel system wants to build some
software +universal to support arm64.


One port needed +universal has some tricks in it, and requires
"merger_must_run_binaries". That port cannot be built universal on an
Intel system (which can't run arm64) but can be built universal on an
arm64 system (which can run Intel). So let's say that universal port is
built, and uploaded to the buildbot or similar.


Joe Schmoe on BigSur Intel could download and install that port
+universal, and be on his way.


But now, his system will never see that port could be universal (because
he can't built it universal), and the chain collapses, some would say
needlessly.


So -- how to elegantly handle this issue, if possible?


Ken

Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

ryandesign2
Administrator


On Feb 2, 2021, at 11:18, Ken Cunningham wrote:

> One port needed +universal has some tricks in it, and requires "merger_must_run_binaries". That port cannot be built universal on an Intel system (which can't run arm64) but can be built universal on an arm64 system (which can run Intel).

Ideally of course the port would be fixed so that it can build on any system. The problem only occurs when using the muniversal portgroup, so investigating removing the portgroup is one possibility, though of course the portgroup exists to solve a certain type of problem and that may be needed for that port.

Parts of the build that only need to be run on the build machine should be built for the build machine's arch. For example, not using -arch flags when building those parts would do it. Of course if the parts that will be run during the build will also be installed, it's trickier. Since the muniversal portgroup builds multiple times, once for each arch, you may be able to modify the build so that it uses the version of that program that was built for the machine's build arch. This would require that the build for the machine's build arch happen before the build for foreign archs; I don't know if the muniversal portgroup currently arranges for that to be the case, but if not, that would be a good improvement to make there.


> Joe Schmoe on BigSur Intel could download and install that port +universal, and be on his way.
>
> But now, his system will never see that port could be universal (because he can't built it universal), and the chain collapses, some would say needlessly.

Josh recently made changes to how universal is handled in MacPorts, so I'm not sure how much of your question is related to that. But if "his system will never see that port could be universal" then how could he "download and install that port +universal" in the first place?

Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

Ken Cunningham
In reply to this post by Ken Cunningham
But if "his system will never see that port could be universal" then how could he "download and install that port +universal" in the first place?

that is the crux of it, indeed. 

Should our poor soul, who cannot build the port +universal, still be able to download it +universal from the buildbot and use it, if it exists (built by the arm64 machine).

K
Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

ryandesign2
Administrator
On Feb 3, 2021, at 01:12, Ken Cunningham wrote:

>> But if "his system will never see that port could be universal" then how could he "download and install that port +universal" in the first place?
>
> that is the crux of it, indeed.
>
> Should our poor soul, who cannot build the port +universal, still be able to download it +universal from the buildbot and use it, if it exists (built by the arm64 machine).

At present, no, that would not be possible.

I consider the buildbot a "nice to have". It makes life easier by precompiling things but it is not essential. When something is not available precompiled, MacPorts builds from source. If something doesn't build correctly from source, let's fix that.

Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

Ken Cunningham


> On Feb 2, 2021, at 11:22 PM, Ryan Schmidt <[hidden email]> wrote:
>
> On Feb 3, 2021, at 01:12, Ken Cunningham wrote:
>
>>> But if "his system will never see that port could be universal" then how could he "download and install that port +universal" in the first place?
>>
>> that is the crux of it, indeed.
>>
>> Should our poor soul, who cannot build the port +universal, still be able to download it +universal from the buildbot and use it, if it exists (built by the arm64 machine).
>
> At present, no, that would not be possible.
>
> I consider the buildbot a "nice to have". It makes life easier by precompiling things but it is not essential. When something is not available precompiled, MacPorts builds from source. If something doesn't build correctly from source, let's fix that.
>

OK — for users out there, then, we should begin to make it clear that building +universal on a BigSur Intel machine is not going to work out for a number of ports, and although they might find some ports can build universal on BigSur Intel, for comprehensive +universal support, they should expect to use BigSur arm64.

That is fine with me — IMHO nobody should really be putting anything out that hasn’t been actually tried on an actual BigSur arm64 machine anyway.

But should be clear to people what to expect.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

ryandesign2
Administrator
On Feb 3, 2021, at 01:29, Ken Cunningham wrote:

> On Feb 2, 2021, at 11:22 PM, Ryan Schmidt wrote:
>
>> I consider the buildbot a "nice to have". It makes life easier by precompiling things but it is not essential. When something is not available precompiled, MacPorts builds from source. If something doesn't build correctly from source, let's fix that.
>
> OK — for users out there, then, we should begin to make it clear that building +universal on a BigSur Intel machine is not going to work out for a number of ports, and although they might find some ports can build universal on BigSur Intel, for comprehensive +universal support, they should expect to use BigSur arm64.
>
> That is fine with me — IMHO nobody should really be putting anything out that hasn’t been actually tried on an actual BigSur arm64 machine anyway.
>
> But should be clear to people what to expect.

I'm not aware of this "number of ports". The number of ports that both use the muniversal portgroup and need to run something they built should hopefully be small. For each one, I recommend trying to fix it as I outlined previously.

I wouldn't embark on a campaign of seeding users with the idea that advertised MacPorts features don't work. Instead, as always, I would recommend that users file bug reports when they find anything that doesn't work, and then we can fix those issues. For the situation you're talking about, that could include fixing or (conditionally?) disabling the universal variant.

Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

Ken Cunningham
In reply to this post by Ken Cunningham
Well, as per the title, it will be every port that lists merger_must_run_binaries that will not build universal on BigSur Intel, however many that is.

And all the ports that cascade down as dependents of those ports...

K
Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

Joshua Root-8
In reply to this post by ryandesign2
On 2021-2-3 18:46 , Ryan Schmidt wrote:
> I'm not aware of this "number of ports". The number of ports that both use the muniversal portgroup and need to run something they built should hopefully be small. For each one, I recommend trying to fix it as I outlined previously.
>
> I wouldn't embark on a campaign of seeding users with the idea that advertised MacPorts features don't work. Instead, as always, I would recommend that users file bug reports when they find anything that doesn't work, and then we can fix those issues. For the situation you're talking about, that could include fixing or (conditionally?) disabling the universal variant.

The universal variant *is* conditionally disabled in this specific
situation. That's what Ken was asking about changing, so that users
could download a universal archive of a port they would not be able to
build locally.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: invisible universal variant and merger_must_run_binaries

Ken Cunningham
In reply to this post by Ken Cunningham
The good news is I currently only see 7 ports that set "merger_must_run_binaries" grepping the repo.

If we don't have to add too many more to the list as we explore more universal builds, then perhaps we can fix these to not need it, and solve our problem cleanly that way.

Several are core ports, though.

icu, readline, libgpg-error are probably the three biggest to fix, if possible.

Ken