[MacPorts] #52742: buildbot: add support for building variants

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[MacPorts] #52742: buildbot: add support for building variants

MacPorts
#52742: buildbot: add support for building variants
----------------------------+----------------------
 Reporter:  mojca           |      Owner:  admin@…
     Type:  enhancement     |     Status:  new
 Priority:  Normal          |  Milestone:
Component:  server/hosting  |    Version:
 Keywords:                  |       Port:  buildbot
----------------------------+----------------------
 It would be very handy to have some limited support for building with
 variants.

 Before trashing the subversion checkout on my buildbot, I wanted to attach
 partial patches.

 See #51995.

--
Ticket URL: <https://trac.macports.org/ticket/52742>
MacPorts <https://www.macports.org/>
Ports system for OS X
_______________________________________________
macports-tickets mailing list
[hidden email]
https://lists.macosforge.org/mailman/listinfo/macports-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building variants

MacPorts
#52742: buildbot: add support for building variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------
Changes (by mojca):

 * Attachment "variants-master.cfg.diff" added.

 patches for master.cfg that add support for building with arbitrary
 variants (no sanity checking yet!)

--
Ticket URL: <https://trac.macports.org/ticket/52742>
MacPorts <https://www.macports.org/>
Ports system for OS X
_______________________________________________
macports-tickets mailing list
[hidden email]
https://lists.macosforge.org/mailman/listinfo/macports-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------
Changes (by mojca):

 * Attachment "variants-mpbb-fully-outdated.diff" added.

 outdated patch for basic support of variants in mpbb

--
Ticket URL: <https://trac.macports.org/ticket/52742>
MacPorts <https://www.macports.org/>
Ports system for OS X
_______________________________________________
macports-tickets mailing list
[hidden email]
https://lists.macosforge.org/mailman/listinfo/macports-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------

Comment (by mojca):

 The patches are pretty outdated, but they should give the first
 impression. I wanted to fix the second one, but I should probably write
 some code to parse options between `install-port` and `<portname>` first.

 Some hints by Clemens about how to proceed:
 > `tools/dependencies.tcl` just opens all ports
 > without any specified variants. See `tools/dependencies.tcl` line 66;
 the
 > last parameter to `mportopen`, which is currently `{}`, is the list of
 > variants. The format is easy to create:
 >
 > {{{
 > array set variations {}
 > set variations(quartz) "+"
 > set variations(x11) "-"
 > mportopen ... [array get variations]
 > }}}
 >
 > gives you an mport reference as if the port was installed with
 > `+quartz-x11`. You should be able to find some suitable parsing code for
 > variants given on the command line in `src/port/port.tcl`, line 1695 and
 > following.
 >
 >> Looking at that file I have an impression that we forgot to
 >> consider global settings (`[array get global_variations]`):
 >>
 >> {{{
 >> if {!$index_only} {
 >>     # Add any global_variations to the variations
 >>     # specified for the port (so we get e.g. dependencies right)
 >>     array unset merged_variations
 >>     array set merged_variations [array get variations]
 >>     foreach { variation value } [array get global_variations] {
 >>         if { ![info exists merged_variations($variation)] } {
 >>             set merged_variations($variation) $value
 >>         }
 >>     }
 >> }}}
 >>
 >> This means that even if someone would (theoretically) set up a buildbot
 >> with `+quartz -x11` in configuration file, those setting would probably
 >> be ignored. Or do I misunderstand the concept?
 >
 > Yes, I think the file doesn't handle global variants. I think it's also
 > bad design that `port/port.tcl` has to do the merging of variant specs
 > without at least some library support from `macports1.0`, but that's a
 > story for a different refactoring.


 An additional warning:
 > However, this output will only list positive variants, it will not list
 > default variants that have been explicitly disabled. This might be a
 > problem, because specifying `-x11 +quartz` will just give you a list of
 > all dependencies with `+quartz`, but not without `+x11`, so if a port is
 > written in a way that it does not automatically disable `x11` if
 `+quartz`
 > is given (for example if both are supported simultaneously and `x11` is
 > the default), this might lead to unexpected results. I haven't tested
 > any of this, but we probably should.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:1>
MacPorts <https://www.macports.org/>
Ports system for OS X
_______________________________________________
macports-tickets mailing list
[hidden email]
https://lists.macosforge.org/mailman/listinfo/macports-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants (was: buildbot: add support for building variants)

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:2>
MacPorts <https://www.macports.org/>
Ports system for OS X
_______________________________________________
macports-tickets mailing list
[hidden email]
https://lists.macosforge.org/mailman/listinfo/macports-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------

Comment (by mojca):

 It looks like the core of the needed functionality was already added in
 r152489.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:3>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------

Comment (by jmroot):

 Some of the logging probably needs to be enhanced with variant info, but
 that's about it AFAICT.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:4>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+---------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:  buildbot        |
-----------------------------+---------------------

Comment (by mojca):

 I didn't check the logs, but among other things we also need:
 * (straightforward/trivial) a revised version of [attachment:variants-
 master.cfg.diff my old attempt] to add a field for variants to
 `master.cfg`
 * (lower priority) some basic logic to automatically build `+quartz` for
 certain ports

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:5>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------
Changes (by mojca):

 * keywords:   => buildbot
 * port:  buildbot =>


--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:6>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------
Changes (by mojca):

 * Attachment "variants-master.cfg.2.diff" added.

 add a field to support variants

--
Ticket URL: <https://trac.macports.org/ticket/52742>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 The new (trivial) patch for `master.cfg` seems to work for me.

 We need some basic safeguard (regexp?) though to make sure that only valid
 variant names are entered, otherwise people could end up writing port
 names or any other weird things. (I could imagine things equivalent to sql
 injections ending up here. Nothing to do with sql, but I hope you get the
 idea.)

 This will be sufficient to get `gkt3 +quartz` and other binaries on the
 buildbots. The next (independent) step would be to write the logic to
 build that automatically for some ports.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:7>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+----------------------
  Reporter:  mojca          |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+----------------------

Comment (by neverpanic):

 I've been working on getting this working, ideally even without
 modifications to the master.cfg of the buildbot, but I've hit some
 problems. [f6e46815a5033358212ce383956fd567eb51e53c/mpbb] changed the
 variant handling code in mpbb in a way that will cause failures when
 requesting specific variants to be installed. See the commit on GitHub for
 a description of the problem. We would have to revert this commit for this
 effort to proceed, but I'd like to wait for feedback from Joshua before
 doing that.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:14>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+----------------------
  Reporter:  mojca          |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+----------------------

Comment (by neverpanic):

 See https://github.com/macports/mpbb/pull/5.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:15>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+----------------------
  Reporter:  mojca          |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+----------------------

Comment (by jmroot):

 I don't doubt the current code needs changes to work right when specifying
 arbitrary variants. Simply reverting that commit is not right either,
 though. The distinction between the canonical_active_variants and the
 variants that are being requested needs to be maintained.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:16>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+----------------------
  Reporter:  mojca          |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+----------------------

Comment (by neverpanic):

 You're right. I thought about this yesterday and came to the conclusion
 that this is really a bug in base, since base propagates these variants
 down the dependency tree and attempts to install a combination that will
 not work.

 Either the `glib2` Portfile should ignore the user's request to change the
 variant and forcefully set a variant, or base should not request this
 invalid combination.

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:17>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+----------------------
  Reporter:  mojca          |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+----------------------

Comment (by neverpanic):

 In [changeset:"45c59fe0a66e382927fbf6fe629d37898eb42b6b/macports-ports"
 45c59fe0a66e382927fbf6fe629d37898eb42b6b/macports-ports]:
 {{{
 #!ConfigurableCommitTicketReference repository="macports-ports"
 revision="45c59fe0a66e382927fbf6fe629d37898eb42b6b"
 glib2: Default to +quartz when -x11 is given

 When building pango -x11 in a clean prefix, glib2 will fail to install,
 because the -x11 variant is being passed down to glib2, but glib2
 requires either +quartz or +x11, and +quartz was not enabled by default
 when x11 was disabled.

 This caused problems when implementing variant support for the buildbot
 in https://github.com/macports/mpbb/pull/5 and initially caused me to
 think that I would have to revert
 https://github.com/macports/mpbb/commit/f6e46815a5033358212ce383956fd567eb51e53c
 which was added due to https://github.com/macports/mpbb/pull/4 and
 https://lists.macports.org/pipermail/macports-dev/2017-June/035978.html.

 This solution should instead work without the revert and still allow the
 buildbot to build both wine and +quartz-x11 ports.

 See: https://trac.macports.org/ticket/52742
 }}}

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:18>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+------------------------
  Reporter:  mojca          |      Owner:  neverpanic
      Type:  enhancement    |     Status:  assigned
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+------------------------
Changes (by neverpanic):

 * owner:  admin@… => neverpanic
 * status:  new => assigned


--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:19>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #52742: buildbot: add support for building ports with variants

MacPorts
In reply to this post by MacPorts
#52742: buildbot: add support for building ports with variants
----------------------------+------------------------
  Reporter:  mojca          |      Owner:  neverpanic
      Type:  enhancement    |     Status:  assigned
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:  buildbot
      Port:                 |
----------------------------+------------------------

Comment (by ryandesign):

 In [changeset:"32fd667ad3d4cb8fe9be7bf2dc5058cc73d8e530/macports-ports"
 32fd667ad3d4cb8fe9be7bf2dc5058cc73d8e530/macports-ports]:
 {{{
 #!ConfigurableCommitTicketReference repository="macports-ports"
 revision="32fd667ad3d4cb8fe9be7bf2dc5058cc73d8e530"
 glib2-devel: Default to +quartz when -x11 is given

 When building pango -x11 in a clean prefix, glib2 will fail to install,
 because the -x11 variant is being passed down to glib2, but glib2
 requires either +quartz or +x11, and +quartz was not enabled by default
 when x11 was disabled.

 This caused problems when implementing variant support for the buildbot
 in https://github.com/macports/mpbb/pull/5 and initially caused me to
 think that I would have to revert
 https://github.com/macports/mpbb/commit/f6e46815a5033358212ce383956fd567eb51e53c
 which was added due to https://github.com/macports/mpbb/pull/4 and
 https://lists.macports.org/pipermail/macports-dev/2017-June/035978.html.

 This solution should instead work without the revert and still allow the
 buildbot to build both wine and +quartz-x11 ports.

 See: https://trac.macports.org/ticket/52742
 }}}

--
Ticket URL: <https://trac.macports.org/ticket/52742#comment:20>
MacPorts <https://www.macports.org/>
Ports system for macOS