Mirorring distfiles

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Mirorring distfiles

Mojca Miklavec-2
Dear Ryan,

What is the current situaton with mirorring distfiles and some other
tasks? (Yes, I wanted to put that on the list as well. The other
semi-related issue is integration of xz library of course [and
renaming binaries for legacy systems :].)

On 21 February 2018 at 21:22, Clemens Lang wrote:

> Hi,
>
> On Wed, Feb 21, 2018 at 05:33:53PM +0100, Mojca Miklavec wrote:
>> I have a few ideas, but I would like to gather more input about what
>> you consider the most burning issues that might need some discussions
>> or hacking sessions during the MacPorts meeting. We'll try to fit
>> those on the schedule.
>
> Oh, and we might also want to set up build jobs for mirroring of
> distfiles to get the ball rolling on this topic, too. The move to newer
> TLS is quite the issue for old systems.

From what I recall Rainer or someone else has already written up the
code and it was only a matter of deploying the changes. Or am I
missing something? (I didn't check right now.)

It would be good to have a clear idea of what we can and cannot do
(what needs to be done by Ryan or in case he needs someone's help).

Mojca
Reply | Threaded
Open this post in threaded view
|

Re: Mirorring distfiles

Ryan Schmidt-24

On Feb 22, 2018, at 04:57, Mojca Miklavec wrote:

> Dear Ryan,
>
> What is the current situaton with mirorring distfiles

Unchanged. I need to create a buildworker on the buildmaster. I then need to deploy the buildbot configuration that does the mirroring, figure out what configuration it needs if any, and see if it works, and if not, how it needs to be fixed. And it all needs to happen when the buildbot is idle.

> and some other
> tasks? (Yes, I wanted to put that on the list as well. The other
> semi-related issue is integration of xz library of course

Is this for the old idea of using xz instead of bz2 for compressing our binary archives? I assumed we had given up on that. I mean I'd love to do it because xz files would be smaller, but we either have to make that change and introduce it for the next major macOS version, and keep the old versions on bz2, or we have to develop a strategy for migrating existing systems from bz2 to xz and for recompressing our existing bz2 archives to xz and for eventually phasing out the bz2 archives.

> [and
> renaming binaries for legacy systems :].)

You're talking about choosing how to differentiate the libc++ binaries from the libstdc++ binaries?


> On 21 February 2018 at 21:22, Clemens Lang wrote:
>> Hi,
>>
>> On Wed, Feb 21, 2018 at 05:33:53PM +0100, Mojca Miklavec wrote:
>>> I have a few ideas, but I would like to gather more input about what
>>> you consider the most burning issues that might need some discussions
>>> or hacking sessions during the MacPorts meeting. We'll try to fit
>>> those on the schedule.
>>
>> Oh, and we might also want to set up build jobs for mirroring of
>> distfiles to get the ball rolling on this topic, too. The move to newer
>> TLS is quite the issue for old systems.
>
> From what I recall Rainer or someone else has already written up the
> code and it was only a matter of deploying the changes. Or am I
> missing something? (I didn't check right now.)
>
> It would be good to have a clear idea of what we can and cannot do
> (what needs to be done by Ryan or in case he needs someone's help).
>
> Mojca

Reply | Threaded
Open this post in threaded view
|

Re: Mirorring distfiles

Mojca Miklavec-2
On 22 February 2018 at 21:14, Ryan Schmidt wrote:
> On Feb 22, 2018, at 04:57, Mojca Miklavec wrote:
>
>> Dear Ryan,
>>
>> What is the current situaton with mirorring distfiles
>
> Unchanged. I need to create a buildworker on the buildmaster. I then need to deploy the buildbot configuration that does the mirroring, figure out what configuration it needs if any, and see if it works, and if not, how it needs to be fixed.

So does that mean that the code is there and merely needs testing? Or
do you also need to write the code.

> And it all needs to happen when the buildbot is idle.

Is anything in particular preventing you from testing the code on
another (maybe even linux) server? Potentially even your laptop?

>> and some other
>> tasks? (Yes, I wanted to put that on the list as well. The other
>> semi-related issue is integration of xz library of course
>
> Is this for the old idea of using xz instead of bz2 for compressing our binary archives?

Yes.

> I assumed we had given up on that.

I did not.

> I mean I'd love to do it because xz files would be smaller, but we either have to make that change and introduce it for the next major macOS version, and keep the old versions on bz2, or we have to develop a strategy for migrating existing systems from bz2 to xz and for recompressing our existing bz2 archives to xz and for eventually phasing out the bz2 archives.

I would start by merely supporting xz compression in the first place.
By adding a library to the sources. Migration strategies can always
happen later on.

>> [and
>> renaming binaries for legacy systems :].)
>
> You're talking about choosing how to differentiate the libc++ binaries from the libstdc++ binaries?

Yes. I still find it super painful that we did not manage to migrate
to libc++ over all those years due to a basically "trivial" issue.

My suggestion would be to use one hacking session during the meeting
to implement a few lines of code that take care of renaming the
binaries + test whether users could switch to libc++ using migration
tool written by Umesh.

Mojca
Reply | Threaded
Open this post in threaded view
|

Re: Mirorring distfiles

Ryan Schmidt-24

On Mar 1, 2018, at 16:09, Mojca Miklavec wrote:

> On 22 February 2018 at 21:14, Ryan Schmidt wrote:
>> On Feb 22, 2018, at 04:57, Mojca Miklavec wrote:
>>
>>> Dear Ryan,
>>>
>>> What is the current situaton with mirorring distfiles
>>
>> Unchanged. I need to create a buildworker on the buildmaster. I then need to deploy the buildbot configuration that does the mirroring, figure out what configuration it needs if any, and see if it works, and if not, how it needs to be fixed.
>
> So does that mean that the code is there and merely needs testing? Or
> do you also need to write the code.

I looked at it briefly long ago and don't know / don't remember what has been done and what still needs to be done.


>> And it all needs to happen when the buildbot is idle.
>
> Is anything in particular preventing you from testing the code on
> another (maybe even linux) server? Potentially even your laptop?

I don't do Linux. I don't have a buildbot setup elsewhere for testing. I may need to make one.



Reply | Threaded
Open this post in threaded view
|

Re: Mirorring distfiles

Joshua Root-8
In reply to this post by Mojca Miklavec-2
On 2018-3-2 09:09 , Mojca Miklavec wrote:

> On 22 February 2018 at 21:14, Ryan Schmidt wrote:
>> On Feb 22, 2018, at 04:57, Mojca Miklavec wrote:
>>
>>> Dear Ryan,
>>>
>>> What is the current situaton with mirorring distfiles
>>
>> Unchanged. I need to create a buildworker on the buildmaster. I then need to deploy the buildbot configuration that does the mirroring, figure out what configuration it needs if any, and see if it works, and if not, how it needs to be fixed.
>
> So does that mean that the code is there and merely needs testing? Or
> do you also need to write the code.

Much of the code exists, but a bit more is needed. The script needs to
filter out ports that have 'nomirror' in their licenses, and it needs to
get a ports tree.

Last I remember I was waiting for details of where the files should go,
and asked whether it would make sense to simply fold mprsyncup into the
mirroring job, rather than maintaining yet another ports tree or
synchronising two users of the same one.

(This discussion was on macports-infra back in September BTW.)

>> You're talking about choosing how to differentiate the libc++ binaries from the libstdc++ binaries?
>
> Yes. I still find it super painful that we did not manage to migrate
> to libc++ over all those years due to a basically "trivial" issue.
>
> My suggestion would be to use one hacking session during the meeting
> to implement a few lines of code that take care of renaming the
> binaries + test whether users could switch to libc++ using migration
> tool written by Umesh.

IMO the way forward is to just say libc++ is the new normal on all
platforms where it can work. For this we need a few things:

Fields in the registry indicating whether each port links with a C++
runtime and which one, and whether the port explicitly chose it or went
with the default. The former can be filled in by rev-upgrade when it
scans for broken linking.

Rev-upgrade then also needs to check whether c++ ports that didn't
explicitly choose a runtime are using the current default, and treat
them as broken if not.

Then we can release a new base version that defaults to libc++ on the
affected platforms, and at the same time delete the archives for those
platforms of all c++ ports from the packages server.

None of this is particularly hard, and will minimise disruption and
rebuilding. I'm sure you should be able to implement all the code in the
hacking session. :)

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: Mirorring distfiles

Ryan Schmidt-24
In reply to this post by Mojca Miklavec-2

On Mar 1, 2018, at 16:09, Mojca Miklavec wrote:

> Yes. I still find it super painful that we did not manage to migrate
> to libc++ over all those years due to a basically "trivial" issue.
>
> My suggestion would be to use one hacking session during the meeting
> to implement a few lines of code that take care of renaming the
> binaries + test whether users could switch to libc++ using migration
> tool written by Umesh.

Please also take a look at this related issue about having separate libc++ and libstdc++ portindexes, and what name to use to differentiate them:

https://trac.macports.org/ticket/55471

It doesn't have to use the same naming scheme as the archives we distribute, but it could, so it could relate.

Reply | Threaded
Open this post in threaded view
|

Re: Mirorring distfiles

Mojca Miklavec-2
On 2 March 2018 at 15:02, Ryan Schmidt wrote:

> On Mar 1, 2018, at 16:09, Mojca Miklavec wrote:
>
>> Yes. I still find it super painful that we did not manage to migrate
>> to libc++ over all those years due to a basically "trivial" issue.
>>
>> My suggestion would be to use one hacking session during the meeting
>> to implement a few lines of code that take care of renaming the
>> binaries + test whether users could switch to libc++ using migration
>> tool written by Umesh.
>
> Please also take a look at this related issue about having separate libc++ and libstdc++ portindexes, and what name to use to differentiate them:
>
> https://trac.macports.org/ticket/55471
>
> It doesn't have to use the same naming scheme as the archives we distribute, but it could, so it could relate.

I wrote some comments to that ticket.

Mojca