Re: [MacPorts] #51995: buildbot improvements

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #51995: buildbot improvements

MacPorts
#51995: buildbot improvements
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  assigned
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Old description:

> I'm opening a single ticket to list and brainstorm ideas for further
> improvements of the new buildbot setup. Please note that each individual
> item listed here would deserve its own ticket, but it makes no sense to
> open so many tickets at this early stage.
>
> * avoid activating the port and its dependencies if the port was already
> installed (and is only deactivated), just continue from the upload step
> * create "fail cache" and don't build the port if it hasn't been changed
> and is known to fail; don't even install any dependency if one of the
> dependencies is know to fail and hasn't been modified since (the email
> about build failure should still be sent though)
> * provide a force flag to override the cache and attempt the build
> despite of cache predicting the failure
> * provide means to build just a single port without any subports (for
> example, allow building just `wxWidgets-3.0` without `wxgtk-3.0` and
> others)
> * on top of the `portlist` (`portname`) variable, introduce a new
> variable that holds variants (say `+quartz -x11`) and allow building the
> port with those variants (the idea is not to come up with a complete list
> of useful builds, but to set the necessary building blocks to allow
> building quartz variants of ports or maybe other tricks that might
> occasionally be needed); see also #35897
> * add a new variable to allow running the tests (we could later try to
> enable that by default for perl modules for example)
> * triggering the individual builds on the builder is currently in random
> order (not sure if this is possible to fix)
> * provide a variable to allow rebuilding all the ports committed between
> two commit numbers (useful to overcome problems due to temporary broken
> dependencies or broken build slaves); currently this is only possible for
> a single commit
> * Would it make any sense to merge portwatcher requests or rather not?
> The slight optimization might reduce the need to rebuild ports with
> several commits one after the other.
>
> ==== Display / UI ====
> * create a hyperlink from the builder to the watcher (currently there's
> just url, but it's not a clickable hyperlink)
> * create hypherlinks to individual builds from the console page (there's
> currently just a hyperlink that points to an useless `#` on the same
> page)
> * create navigation button for individual builds (one build back and
> forward)
> * ~~(DONE; r152536) disable tracking of remaining build time based on
> statistics (unless the statistics gets considerably better and actually
> tracks builds of individual ports)~~
> * Is it possible to display URLs (on the watcher) to triggered builds as
> soon as the builds start (rather than waiting until all builds are
> finished)?
> * (not sure if doable) pending forced build requests don't display any
> info at all until they are started (one doesn't know which port builds
> were forced)
> * it would be nice if hyperlinks from the watcher to the builders would
> also contain the port name being built
> * (the port builder could theoretically display a new property with all
> ports being built by the same watcher; no added benefit, just giving
> extra info to anyone browsing the logs; super low priority, if desired at
> all)
> * it would be handy if the SVN revision (when applicable) was also shown
> on the port builder (with all the sugars ... hyperlinks, commit message
> etc.), not just the port watcher
>
> ==== Log files ====
> * ~~(DONE; r151264, r151266) provide a link to `main.log`~~
> * provide a link to `config.log` if one exists (we can worry about more
> advanced situations with multiple logs later)
> * ~~(DONE; r152521) remove `stdio` from the first step in builder with
> the sole purpose of displaying port name~~
> * ~~(DONE; r151264, r151266) add some very basic build statistics (build
> time, size of build dir, size of destroot)~~
>   * missing: size of binary archive, ...
>   * perhaps more accuracy in timing (currently seconds are used)
> * ~~(DONE; r151215, r151216) the builder should provide a list of
> dependencies to be installed as a separate log file, for example in the
> following format:~~
> {{{
> Installing dependencies:
>  - expat
>  - jpeg
>  - libiconv
>  - zlib
>  - libpng
>  - ncurses
>  - gettext
>  - xz
>  - tiff
>
> Installing dependency 1 of 9: expat ... [OK]
> Installing dependency 2 of 9: jpeg ... [OK]
> Installing dependency 3 of 9: libiconv ... [OK]
> Installing dependency 4 of 9: libpng ... [OK]
> Installing dependency 5 of 9: ncurses ... [OK]
> Installing dependency 6 of 9: gettext ... [FAIL]
> }}}
> * optionally print the last few lines from `main.log` of the failed
> dependency
> * provide a shorter log about the build progress, the equivalent of
> running `port install` without verbose/debug mode; we could do that
> either by filtering the logs and displaying just the relevant lines or by
> running the installation without debug more and get debug information
> from the `main.log` instead; needs some thinking
> * `steps/svn/logs/stdio` on the watcher now contains email addresses
> (that's slightly bad for web spiders)
> * ~~(DONE; r151267, r151268) the watcher should provide a summary of
> successful / broken triggered builds similar to the old `status`, for
> example something like:~~
>   * (lots of room for improvements)
> {{{
> Building 1 of 20 foo ... [OK]
> Building 2 of 20 foo-devel ... broken dependency baz [FAIL]
> Building 3 of 20 bar ... broken dependency bar-2.0 [FAIL]
> Building - of 20 bar-1.0 ... obsolete [SKIP]
> Building 4 of 20 bar-2.0 ... failed before, no attempt to rebuild [FAIL]
> Building 5 of 20 bar-3.0 ... [OK]
> ...
> }}}
> * see the following link for some ideas of how to make custom (html)
> reports (it uses an old version of buildbot, but ...)
>   * https://blogs.igalia.com/itoral/2007/02/13/howto-integrate-unit-
> tests-with-buildbot/
>   * https://blogs.igalia.com/itoral/2007/02/05/gnome-buildbot-and-
> integration-of-unit-tests/
>
> ==== Sending e-mails ====
> * portwatcher should send notification about broken builds to all
> maintainers of the broken ports + the committer (or the user who forced
> the build)
> * include the aggregated statistics in the email
>   * which ports failed to build
>   * whether they failed because of broken dependencies (which one) or due
> to a build error in port itself (maybe even the phase where the error
> happened), see the log file proposal above
> * include the URL to the build (both the watcher and the builder if
> possible)
>
> ==== Improvements in the core ====
> * if the port builds locally (rather that being fetched from the binary
> archive), it would be nice to store the build time in the database (with
> some caution when builds are broken)
>   * I would like to run a command that would print the list of installed
> port sorted by the build time (say, to determine which ports are the most
> demanding/important to build on the buildbot)
>   * this could be useful input for determining the remaining build time
> on the builder (if we ever figure out how to hack the buildbot to read
> that piece of information from an external source)
>
> ==== Statistics ====
>
> * create a simple web site that collects statistics in a table,
> containing the following variables:
>   * port name
>   * version
>   * revision
>   * explicit variants (usually none, but `+universal` could be one)
>   * all variants
>   * OS version
>   * architecture
>   * builder name
>   * number or link to the watcher
>   * number or link to the builder
>   * success: OK | FAIL | (FAIL-skip)
>   * uploaded: True | False
> * and then prints out the list of latest builds grouped by `Portfile` +
> URLs to binary archives
> * also collect the lists of files + size of compressend & uncompressed
> archive (and build bir) + potentially the build time just for the latest
> successful builds
>
> ==== Organisation ====
> * move the old `buildbot` and `mpab` to a place of historic interest and
> rename `buildbot-test` to `buildbot`, update `contrib/README` to reflect
> the changes
> * potentially move more logic of `master.cfg` to a python module
> * move the list of builders from hardcoded python code to a configuration
> file

New description:

 I'm opening a single ticket to list and brainstorm ideas for further
 improvements of the new buildbot setup. Please note that each individual
 item listed here would deserve its own ticket, but it makes no sense to
 open so many tickets at this early stage.

 * (DONE, [changeset:62834a4212c5b1c59df22221d896cb2c4aab3b44/mpbb
 62834a42]) ~~avoid activating the port and its dependencies if the port
 was already installed (and is only deactivated),~~
   * but continue from the upload step (partially addressed by
 [changeset:4162046ce0f29867d71cca6778d65dbb39b37d05/mpbb 4162046c], still
 a problem for `wine`)
 * ~~create "fail cache" and don't build the port if it hasn't been changed
 and is known to fail; don't even install any dependency if one of the
 dependencies is know to fail and hasn't been modified since (the email
 about build failure should still be sent though)~~
 * provide a force flag to override the cache and attempt the build despite
 of cache predicting the failure
 * (see see #53976) provide means to build just a single port without any
 subports (for example, allow building just `wxWidgets-3.0` without
 `wxgtk-3.0` and others)
 * (see #35897, #52742) on top of the `portlist` (`portname`) variable,
 introduce a new variable that holds variants (say `+quartz -x11`) and
 allow building the port with those variants (the idea is not to come up
 with a complete list of useful builds, but to set the necessary building
 blocks to allow building quartz variants of ports or maybe other tricks
 that might occasionally be needed)
 * (see #53977) add a new variable to allow running the tests (we could
 later try to enable that by default for perl modules for example)
 * (see #52766) triggering the individual builds on the builder is
 currently in random order (not sure if this is possible to fix)
 * (see #53978) provide a variable to allow rebuilding all the ports
 committed between two commit numbers (useful to overcome problems due to
 temporary broken dependencies or broken build slaves); currently this is
 only possible for a single commit
 * (see #53979) Would it make any sense to merge portwatcher requests or
 rather not? The slight optimization might reduce the need to rebuild ports
 with several commits one after the other.

 ==== Display / UI ====
 * create a hyperlink from the builder to the watcher (currently there's
 just url, but it's not a clickable hyperlink)
 * create hypherlinks to individual builds from the console page (there's
 currently just a hyperlink that points to an useless `#` on the same page)
 * create navigation button for individual builds (one build back and
 forward)
 * ~~(DONE; r152536) disable tracking of remaining build time based on
 statistics (unless the statistics gets considerably better and actually
 tracks builds of individual ports)~~
 * Is it possible to display URLs (on the watcher) to triggered builds as
 soon as the builds start (rather than waiting until all builds are
 finished)?
 * (not sure if doable) pending forced build requests don't display any
 info at all until they are started (one doesn't know which port builds
 were forced)
 * it would be nice if hyperlinks from the watcher to the builders would
 also contain the port name being built
 * (the port builder could theoretically display a new property with all
 ports being built by the same watcher; no added benefit, just giving extra
 info to anyone browsing the logs; super low priority, if desired at all)
 * it would be handy if the SVN revision (when applicable) was also shown
 on the port builder (with all the sugars ... hyperlinks, commit message
 etc.), not just the port watcher

 ==== Log files ====
 * ~~(DONE; r151264, r151266) provide a link to `main.log`~~
 * provide a link to `config.log` if one exists (we can worry about more
 advanced situations with multiple logs later)
 * ~~(DONE; r152521) remove `stdio` from the first step in builder with the
 sole purpose of displaying port name~~
 * ~~(DONE; r151264, r151266) add some very basic build statistics (build
 time, size of build dir, size of destroot)~~
   * missing: size of binary archive, ...
   * perhaps more accuracy in timing (currently seconds are used)
 * ~~(DONE; r151215, r151216) the builder should provide a list of
 dependencies to be installed as a separate log file, for example in the
 following format:~~
 {{{
 Installing dependencies:
  - expat
  - jpeg
  - libiconv
  - zlib
  - libpng
  - ncurses
  - gettext
  - xz
  - tiff

 Installing dependency 1 of 9: expat ... [OK]
 Installing dependency 2 of 9: jpeg ... [OK]
 Installing dependency 3 of 9: libiconv ... [OK]
 Installing dependency 4 of 9: libpng ... [OK]
 Installing dependency 5 of 9: ncurses ... [OK]
 Installing dependency 6 of 9: gettext ... [FAIL]
 }}}
 * optionally print the last few lines from `main.log` of the failed
 dependency
 * provide a shorter log about the build progress, the equivalent of
 running `port install` without verbose/debug mode; we could do that either
 by filtering the logs and displaying just the relevant lines or by running
 the installation without debug more and get debug information from the
 `main.log` instead; needs some thinking
 * `steps/svn/logs/stdio` on the watcher now contains email addresses
 (that's slightly bad for web spiders)
 * ~~(DONE; r151267, r151268) the watcher should provide a summary of
 successful / broken triggered builds similar to the old `status`, for
 example something like:~~
   * (lots of room for improvements)
 {{{
 Building 1 of 20 foo ... [OK]
 Building 2 of 20 foo-devel ... broken dependency baz [FAIL]
 Building 3 of 20 bar ... broken dependency bar-2.0 [FAIL]
 Building - of 20 bar-1.0 ... obsolete [SKIP]
 Building 4 of 20 bar-2.0 ... failed before, no attempt to rebuild [FAIL]
 Building 5 of 20 bar-3.0 ... [OK]
 ...
 }}}
 * see the following link for some ideas of how to make custom (html)
 reports (it uses an old version of buildbot, but ...)
   * https://blogs.igalia.com/itoral/2007/02/13/howto-integrate-unit-tests-
 with-buildbot/
   * https://blogs.igalia.com/itoral/2007/02/05/gnome-buildbot-and-
 integration-of-unit-tests/

 ==== Sending e-mails ====
 * portwatcher should send notification about broken builds to all
 maintainers of the broken ports + the committer (or the user who forced
 the build)
 * include the aggregated statistics in the email
   * which ports failed to build
   * whether they failed because of broken dependencies (which one) or due
 to a build error in port itself (maybe even the phase where the error
 happened), see the log file proposal above
 * include the URL to the build (both the watcher and the builder if
 possible)

 ==== Improvements in the core ====
 * if the port builds locally (rather that being fetched from the binary
 archive), it would be nice to store the build time in the database (with
 some caution when builds are broken)
   * I would like to run a command that would print the list of installed
 port sorted by the build time (say, to determine which ports are the most
 demanding/important to build on the buildbot)
   * this could be useful input for determining the remaining build time on
 the builder (if we ever figure out how to hack the buildbot to read that
 piece of information from an external source)

 ==== Statistics ====

 * create a simple web site that collects statistics in a table, containing
 the following variables:
   * port name
   * version
   * revision
   * explicit variants (usually none, but `+universal` could be one)
   * all variants
   * OS version
   * architecture
   * builder name
   * number or link to the watcher
   * number or link to the builder
   * success: OK | FAIL | (FAIL-skip)
   * uploaded: True | False
 * and then prints out the list of latest builds grouped by `Portfile` +
 URLs to binary archives
 * also collect the lists of files + size of compressend & uncompressed
 archive (and build bir) + potentially the build time just for the latest
 successful builds

 ==== Organisation ====
 * ~~move the old `buildbot` and `mpab` to a place of historic interest and
 rename `buildbot-test` to `buildbot`, update `contrib/README` to reflect
 the changes~~
 * potentially move more logic of `master.cfg` to a python module
 * ~~move the list of builders from hardcoded python code to a
 configuration file~~

--

Comment (by mojca):

 cross some items that have been done, open a few new tickets

--
Ticket URL: <https://trac.macports.org/ticket/51995#comment:19>
MacPorts <https://www.macports.org/>
Ports system for macOS
Loading...