Build base archives for CI in a separate repository

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Build base archives for CI in a separate repository

Zero King-2
Hi,

In [1], I patched MacPorts in an attempt to fix a bug (port(1) failed
randomly on Travis). As it seems to be a Travis-specific bug, I plan to
use a separate repository to generate MacPorts archives used in CI and
keep the patch there. This way we can update the CI-specific archives
without a new release in macports-base (e.g. when new releases of macOS
become available on Travis).

[1]: https://github.com/macports-staging/macports-base/commit/282e498ac51ba40bdfd43008ce430ca20a7d54ce#diff-d7db55f70d83fc9dba4ef14de9febe71

--
Best regards,
Zero King

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Rainer Müller-4
On 2017-07-22 13:26, Zero King wrote:
> In [1], I patched MacPorts in an attempt to fix a bug (port(1) failed
> randomly on Travis). As it seems to be a Travis-specific bug, I plan to
> use a separate repository to generate MacPorts archives used in CI and
> keep the patch there. This way we can update the CI-specific archives
> without a new release in macports-base (e.g. when new releases of macOS
> become available on Travis).
>
> [1]:
> https://github.com/macports-staging/macports-base/commit/282e498ac51ba40bdfd43008ce430ca20a7d54ce#diff-d7db55f70d83fc9dba4ef14de9febe71

Should we keep a separate branch in the base repository for this? We
could cherry-pick such hotfixes to that branch (could also be restricted
to infrastructure team). I don't think we need a whole other repository
for this and we should keep everything in one organization.

Besides that, I see no reason that this change could not go into regular
base with a comment explaining why it is necessary.

To debug the issue at hand, you could try to inject a
  system "/bin/ls -laeO@ ${target_dir}"
right before the 'file delete' to find out if there is anything like an
immutable flag on this file or directory.

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Clemens Lang-2
Hi,

----- On 25 Jul, 2017, at 17:27, Rainer Müller [hidden email] wrote:

> On 2017-07-22 13:26, Zero King wrote:
>> In [1], I patched MacPorts in an attempt to fix a bug (port(1) failed
>> randomly on Travis). As it seems to be a Travis-specific bug, I plan to
>> use a separate repository to generate MacPorts archives used in CI and
>> keep the patch there. This way we can update the CI-specific archives
>> without a new release in macports-base (e.g. when new releases of macOS
>> become available on Travis).
>>
>> [1]:
>> https://github.com/macports-staging/macports-base/commit/282e498ac51ba40bdfd43008ce430ca20a7d54ce#diff-d7db55f70d83fc9dba4ef14de9febe71
>
> Should we keep a separate branch in the base repository for this? We
> could cherry-pick such hotfixes to that branch (could also be restricted
> to infrastructure team). I don't think we need a whole other repository
> for this and we should keep everything in one organization.

Exactly my thoughts as well. Having a separate branch allows us to quickly
deploy fixes without the overhead of a separate repository not used for
anything else.

Ideally, we should set up the macports-base Travis CI to automatically build
and publish the latest commit on that branch and the macports-ports CI to
automatically use the latest archive from that branch.

For example, we could upload a generated archive to Bintray and update a
file that always contains the URL to the latest archive and just download
that in the Travis CI job.


> Besides that, I see no reason that this change could not go into regular
> base with a comment explaining why it is necessary.

Agree here, please commit the change in macports-base.


> To debug the issue at hand, you could try to inject a
>  system "/bin/ls -laeO@ ${target_dir}"
> right before the 'file delete' to find out if there is anything like an
> immutable flag on this file or directory.

This might also happen if our builds are run under a sandbox, which could
denies deleting files in this location.

--
Clemens Lang
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Ryan Schmidt-24

On Jul 26, 2017, at 11:19, Clemens Lang wrote:

> For example, we could upload a generated archive to Bintray and update a
> file that always contains the URL to the latest archive and just download
> that in the Travis CI job.
>

If we want to use BinTray for any MacPorts things, I already registered the "macports" organization there.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Zero King-2
In reply to this post by Clemens Lang-2
On Wed, Jul 26, 2017 at 06:19:16PM +0200, Clemens Lang wrote:

>Hi,
>
>----- On 25 Jul, 2017, at 17:27, Rainer Müller [hidden email] wrote:
>
>> On 2017-07-22 13:26, Zero King wrote:
>>> In [1], I patched MacPorts in an attempt to fix a bug (port(1) failed
>>> randomly on Travis). As it seems to be a Travis-specific bug, I plan to
>>> use a separate repository to generate MacPorts archives used in CI and
>>> keep the patch there. This way we can update the CI-specific archives
>>> without a new release in macports-base (e.g. when new releases of macOS
>>> become available on Travis).
>>>
>>> [1]:
>>> https://github.com/macports-staging/macports-base/commit/282e498ac51ba40bdfd43008ce430ca20a7d54ce#diff-d7db55f70d83fc9dba4ef14de9febe71
>>
>> Should we keep a separate branch in the base repository for this? We
>> could cherry-pick such hotfixes to that branch (could also be restricted
>> to infrastructure team). I don't think we need a whole other repository
>> for this and we should keep everything in one organization.
>
>Exactly my thoughts as well. Having a separate branch allows us to quickly
>deploy fixes without the overhead of a separate repository not used for
>anything else.
I prefer using a separate repository (only containing .travis.yml and
maybe patches, not MacPorts source) because I can update the binaries by
creating a new release (tag) on the same commit (MacPorts version can be
read from tag name) and keep the setup for ports CI away from
macports-base (they aren't related in my opinion).

>Ideally, we should set up the macports-base Travis CI to automatically build
>and publish the latest commit on that branch and the macports-ports CI to
>automatically use the latest archive from that branch.

From https://bintray.com/docs/terms_of_service.html:
You may not, whether by yourself or anyone on your behalf, unless otherwise explicitly permitted under these Terms:
(xxvii) in connection with Contributions, upload and/or store on the Site continuous integration artifacts or nightly builds, (i.e. you may only upload and store content in direct connection to official software releases).

A tag counts as official software releases. But the latest commit
doesn't.

>For example, we could upload a generated archive to Bintray and update a
>file that always contains the URL to the latest archive and just download
>that in the Travis CI job.
>
>
>> Besides that, I see no reason that this change could not go into regular
>> base with a comment explaining why it is necessary.
>
>Agree here, please commit the change in macports-base.

That change didn't work and I committed another fix (not verified yet):
https://github.com/macports-staging/macports-base/commit/84b040fbcb1134e5cab1cc10cfc991c2d05c4824#diff-d7db55f70d83fc9dba4ef14de9febe71

Also, the other bug (Warning: Failed to copy com.apple.dt.Xcode.plist
... [1]) is not fixed.

[1] https://travis-ci.org/macports/macports-ports/jobs/249233655

>> To debug the issue at hand, you could try to inject a
>>  system "/bin/ls -laeO@ ${target_dir}"
>> right before the 'file delete' to find out if there is anything like an
>> immutable flag on this file or directory.
>
>This might also happen if our builds are run under a sandbox, which could
>denies deleting files in this location.
>
>--
>Clemens Lang
--
Best regards,
Zero King

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Clemens Lang-2
Hi,

On Thu, Jul 27, 2017 at 01:51:31AM +0000, Zero King wrote:
> I prefer using a separate repository (only containing .travis.yml and
> maybe patches, not MacPorts source) because I can update the binaries
> by creating a new release (tag) on the same commit (MacPorts version
> can be read from tag name) and keep the setup for ports CI away from
> macports-base (they aren't related in my opinion).

I would argue that the repository should in fact contain the MacPorts
source code. Ideally, without change in the repository, the build
results should be exactly the same, so there should be no need to
rebuild without a change in the repository. Remember that a change in
.travis.yml is also a code change worth a commit.

I also think that macports-base is very much related to the CI for
macports-ports just in the same way that macports-base in general is
related to macports-ports.

What's your use case for re-tagging on the same commit? I.e. what change
do you expect from rebuilding without changing something in the
repository (i.e. the .travis.yml file or the source).

Additionally, having a separate repository is very visible to users,
which do not care about the internals of our CI system implementation.


> From https://bintray.com/docs/terms_of_service.html:
> You may not, whether by yourself or anyone on your behalf, unless
> otherwise explicitly permitted under these Terms:
> (xxvii) in connection with Contributions, upload and/or store on the
> Site continuous integration artifacts or nightly builds, (i.e. you may
> only upload and store content in direct connection to official
> software releases).
>
> A tag counts as official software releases. But the latest commit
> doesn't.

If we deploy the latest commit to production, that's a release, whether
it has a tag or not. In practice, I expect us to update this repository
once per MacPorts Base release and maybe once or twice in between, so
their main concern of huge disk space usage is probably not an issue.

Alternatively, we can also use a different service from the list
supported by Travis: https://docs.travis-ci.com/user/deployment/.


> That change didn't work and I committed another fix (not verified yet):
> https://github.com/macports-staging/macports-base/commit/84b040fbcb1134e5cab1cc10cfc991c2d05c4824#diff-d7db55f70d83fc9dba4ef14de9febe71

That would match my guess of sandboxing (or similar) being in place.

> Also, the other bug (Warning: Failed to copy com.apple.dt.Xcode.plist
> ... [1]) is not fixed.
>
> [1] https://travis-ci.org/macports/macports-ports/jobs/249233655

It's a warning only, so we might be able to just ignore it.

> > > To debug the issue at hand, you could try to inject a
> > >  system "/bin/ls -laeO@ ${target_dir}"
> > > right before the 'file delete' to find out if there is anything like an
> > > immutable flag on this file or directory.

Have you tried this?

--
Clemens
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Zero King-2
On Thu, Jul 27, 2017 at 07:56:27PM +0200, Clemens Lang wrote:

>Hi,
>
>On Thu, Jul 27, 2017 at 01:51:31AM +0000, Zero King wrote:
>> I prefer using a separate repository (only containing .travis.yml and
>> maybe patches, not MacPorts source) because I can update the binaries
>> by creating a new release (tag) on the same commit (MacPorts version
>> can be read from tag name) and keep the setup for ports CI away from
>> macports-base (they aren't related in my opinion).
>
>I would argue that the repository should in fact contain the MacPorts
>source code. Ideally, without change in the repository, the build
>results should be exactly the same, so there should be no need to
>rebuild without a change in the repository. Remember that a change in
>.travis.yml is also a code change worth a commit.
>
>I also think that macports-base is very much related to the CI for
>macports-ports just in the same way that macports-base in general is
>related to macports-ports.
>
>What's your use case for re-tagging on the same commit? I.e. what change
>do you expect from rebuilding without changing something in the
>repository (i.e. the .travis.yml file or the source).
The tag name changed. Currently, I'm using MacPorts version + a letter
for tags (e.g. v2.4.1a). So when v2.4.2 is released, I can simply tag
v2.4.2a and the build script can read the MacPorts version from the tag
and fetch the source from distfiles.macports.org. This doesn't cover new
macOS versions though (require changing .travis.yml).

Were we to use a branch, should we rebase onto or merge the release
branch? I don't want to distribute API keys (though encrypted) in
MacPorts releases, so I prefer that the CI branch not be merged into
master or release branches.

>Additionally, having a separate repository is very visible to users,
>which do not care about the internals of our CI system implementation.

We already have many macports-user-* repositories that are very visible
but inactive.

>> > > To debug the issue at hand, you could try to inject a
>> > >  system "/bin/ls -laeO@ ${target_dir}"
>> > > right before the 'file delete' to find out if there is anything like an
>> > > immutable flag on this file or directory.
>
>Have you tried this?

https://travis-ci.org/macports-staging/macports-ports/jobs/257682551#L102

Tried once by adding it to .travis.yml, nothing abnormal. I'll try
adding that to source later. It's harder to redeploy the archives than
changing .travis.yml, but not as accurate for debugging.

>--
>Clemens

--
Best regards,
Zero King

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Mojca Miklavec-2
In reply to this post by Clemens Lang-2
Just my 2c. I would also prefer to do the necessary changes in the
same repository unless there's a valid technical reason that would
prevent us from being able to do it (like Travis not allowing us to
build/upload often enough).

In case there's a bug in base, it has to be fixed anyway.

Mojca
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Clemens Lang-2
In reply to this post by Zero King-2
Hi,

----- On 28 Jul, 2017, at 04:06, Zero King [hidden email] wrote:

> On Thu, Jul 27, 2017 at 07:56:27PM +0200, Clemens Lang wrote:
> > What's your use case for re-tagging on the same commit? I.e. what change
> > do you expect from rebuilding without changing something in the
> > repository (i.e. the .travis.yml file or the source).
>
> The tag name changed. Currently, I'm using MacPorts version + a letter
> for tags (e.g. v2.4.1a). So when v2.4.2 is released, I can simply tag
> v2.4.2a and the build script can read the MacPorts version from the tag
> and fetch the source from distfiles.macports.org. This doesn't cover new
> macOS versions though (require changing .travis.yml).

But we would get the same behavior by merging the 2.4.2 tag (or the
release-2.4 branch) into the ci branch, wouldn't we?


> Were we to use a branch, should we rebase onto or merge the release
> branch? I don't want to distribute API keys (though encrypted) in
> MacPorts releases, so I prefer that the CI branch not be merged into
> master or release branches.

I'd say we should merge into the CI branch. I also agree that API keys
and the specific .travis.yml should never be merged from the CI branch
back to master (i.e. merging is only ever allowed in one direction). Since
close to no development that's also relevant for master should happen in
this branch, I think that's fine.


> > Additionally, having a separate repository is very visible to users,
> > which do not care about the internals of our CI system implementation.
>
> We already have many macports-user-* repositories that are very visible
> but inactive.

Those are an side effect of the move because we didn't have a different
place to put them. Ideally, we want users to claim them and remove them
from our org.


--
Clemens Lang
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Rainer Müller-4
In reply to this post by Zero King-2
On 2017-07-28 04:06, Zero King wrote:

> On Thu, Jul 27, 2017 at 07:56:27PM +0200, Clemens Lang wrote:
>> I would argue that the repository should in fact contain the MacPorts
>> source code. Ideally, without change in the repository, the build
>> results should be exactly the same, so there should be no need to
>> rebuild without a change in the repository. Remember that a change in
>> .travis.yml is also a code change worth a commit.
>>
>> I also think that macports-base is very much related to the CI for
>> macports-ports just in the same way that macports-base in general is
>> related to macports-ports.
>>
>> What's your use case for re-tagging on the same commit? I.e. what change
>> do you expect from rebuilding without changing something in the
>> repository (i.e. the .travis.yml file or the source).
>
> The tag name changed. Currently, I'm using MacPorts version + a letter
> for tags (e.g. v2.4.1a). So when v2.4.2 is released, I can simply tag
> v2.4.2a and the build script can read the MacPorts version from the tag
> and fetch the source from distfiles.macports.org. This doesn't cover new
> macOS versions though (require changing .travis.yml).

v2.4.1a seems confusing to me, as it looks like a re-release.

My suggestion would be a travis-2.4 branch (to match release-2.4) with
tags named like travis-v2.4.1 in the macports-base repository. That
should sufficiently distinguish that this branch/tag is meant for CI only.

> Were we to use a branch, should we rebase onto or merge the release
> branch? I don't want to distribute API keys (though encrypted) in
> MacPorts releases, so I prefer that the CI branch not be merged into
> master or release branches.

I guess we cannot add the secrets to the Travis execution environment,
as there can only be one configuration per repository and that would
expose the secrets on pull requests?

If keeping secrets is a big problem, using a separate repository might
indeed be a solution, but I would keep it in the MacPorts organization.

Although we could also use the buildbot to recreate and deploy the base
archive, the track record for deploying stuff on build.macports.org
suggests that it would take at least months...

>> Additionally, having a separate repository is very visible to users,
>> which do not care about the internals of our CI system implementation.
>
> We already have many macports-user-* repositories that are very visible
> but inactive.

Which is just old leftovers from Subversion. We wanted keep around for
some time before we remove them.

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build base archives for CI in a separate repository

Zero King-2
On Fri, Jul 28, 2017 at 02:20:47PM +0200, Rainer Müller wrote:

>On 2017-07-28 04:06, Zero King wrote:
>> On Thu, Jul 27, 2017 at 07:56:27PM +0200, Clemens Lang wrote:
>>> I would argue that the repository should in fact contain the MacPorts
>>> source code. Ideally, without change in the repository, the build
>>> results should be exactly the same, so there should be no need to
>>> rebuild without a change in the repository. Remember that a change in
>>> .travis.yml is also a code change worth a commit.
>>>
>>> I also think that macports-base is very much related to the CI for
>>> macports-ports just in the same way that macports-base in general is
>>> related to macports-ports.
>>>
>>> What's your use case for re-tagging on the same commit? I.e. what change
>>> do you expect from rebuilding without changing something in the
>>> repository (i.e. the .travis.yml file or the source).
>>
>> The tag name changed. Currently, I'm using MacPorts version + a letter
>> for tags (e.g. v2.4.1a). So when v2.4.2 is released, I can simply tag
>> v2.4.2a and the build script can read the MacPorts version from the tag
>> and fetch the source from distfiles.macports.org. This doesn't cover new
>> macOS versions though (require changing .travis.yml).
>
>v2.4.1a seems confusing to me, as it looks like a re-release.
I may need to update the CI archives between MacPorts releases. That's
why I used an extra letter.

>My suggestion would be a travis-2.4 branch (to match release-2.4) with
>tags named like travis-v2.4.1 in the macports-base repository. That
>should sufficiently distinguish that this branch/tag is meant for CI only.
>
>> Were we to use a branch, should we rebase onto or merge the release
>> branch? I don't want to distribute API keys (though encrypted) in
>> MacPorts releases, so I prefer that the CI branch not be merged into
>> master or release branches.
>
>I guess we cannot add the secrets to the Travis execution environment,
>as there can only be one configuration per repository and that would
>expose the secrets on pull requests?
No, PRs won't have access to the secrets.

>If keeping secrets is a big problem, using a separate repository might
>indeed be a solution, but I would keep it in the MacPorts organization.

Of course I'll keep it in the MacPorts organization. The discussion
isn't over yet and right now I don't have a working implementation for
either approach.

--
Best regards,
Zero King

smime.p7s (4K) Download Attachment
Loading...