[MacPorts] #57464: Builtbot: Don't keep huge ports installed

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[MacPorts] #57464: Builtbot: Don't keep huge ports installed

MacPorts
#57464: Builtbot: Don't keep huge ports installed
---------------------------+---------------------
 Reporter:  ryandesign     |      Owner:  admin@…
     Type:  enhancement    |     Status:  new
 Priority:  Normal         |  Milestone:
Component:  buildbot/mpbb  |    Version:
 Keywords:                 |       Port:
---------------------------+---------------------
 Our buildbot setup currently keeps all built ports installed on each
 worker. This is supposed to speed up builds so that we don't waste time
 constantly downloading and installing ports that are dependencies of a lot
 of other ports.

 But there are also some inordinately large ports, such as some games,
 which aren't commonly used as dependencies. Here are some of the larger
 ports installed on the highsierra buildbot worker:

 {{{
 $ cd /opt/local/var/macports/software
 $ du -sch alienarena-data/* geant*-data/* knp/* supertuxkart/* texlive-
 latex-extra/* typesafe-activator/* wesnoth/*
 551M    alienarena-data/alienarena-
 data-7.66-20130827_0.darwin_16.noarch.tbz2
 427M    geant4.10.0-data/geant4.10.0-data-4.10.0_0.darwin_16.noarch.tbz2
 428M    geant4.10.1-data/geant4.10.1-data-4.10.1_0.darwin_16.noarch.tbz2
 432M    geant4.10.2-data/geant4.10.2-data-4.10.2_1.darwin_16.noarch.tbz2
 1005M   geant4.10.3-data/geant4.10.3-data-4.10.3_0.darwin_16.noarch.tbz2
 1.1G    geant4.10.4-data/geant4.10.4-data-4.10.4_0.darwin_16.noarch.tbz2
 266M    geant4.9.6-data/geant4.9.6-data-4.9.6_0.darwin_16.noarch.tbz2
 1.0G    knp/knp-4.14_0.darwin_16.x86_64.tbz2
 512M    supertuxkart/supertuxkart-0.9.3_0.darwin_16.x86_64.tbz2
 453M    texlive-latex-extra/texlive-latex-extra-
 47410_0+doc.darwin_16.noarch.tbz2
 663M    typesafe-activator/typesafe-
 activator-1.3.12_0.darwin_16.noarch.tbz2
 365M    wesnoth/wesnoth-1.12.6_2.darwin_16.x86_64.tbz2
 7.1G    total
 }}}

 It's getting to the point where builds are failing because there's not
 enough free disk space, and I am manually uninstalling some of these ports
 on the buildbot workers to free up space. If we could have mpbb
 automatically uninstall large ports, it would free up a good deal of disk
 space.

 I'm not sure what do about dependencies. For example, geant4.10.4 is not
 large, but it depends on geant4.10.4-data which is.

 Do we force the uninstallation of the large ports, leaving the ports
 depending on them broken, on the assumption that when the buildbot asks to
 install them again it will reinstall the dependencies and fix them?

 Or do we have the buildbot uninstall not only the large ports but also all
 ports depending on them? That would be good for geant, but bad for
 texlive-latex-extra, on which lots of things depend. We could base the
 decision of whether to uninstall on how many ports would be uninstalled.
 (It it's, say, 5 or fewer ports, uninstall them, otherwise leave them
 installed.)

 Or do we instead make a general rule (not based on size) that we don't
 keep a port installed if nothing depends on it?

 Some of these ports might not be distributable. Once we have [ticket:54800
 nondistributable archives available on a private server] that won't be a
 big problem anymore. We can wait until we resolve that ticket before
 deploying an implementation for this one.

--
Ticket URL: <https://trac.macports.org/ticket/57464>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #57464: Builtbot: Don't keep huge ports installed

MacPorts
#57464: Builtbot: Don't keep huge ports installed
----------------------------+---------------------
  Reporter:  ryandesign     |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:
      Port:                 |
----------------------------+---------------------
Description changed by ryandesign:

Old description:

> Our buildbot setup currently keeps all built ports installed on each
> worker. This is supposed to speed up builds so that we don't waste time
> constantly downloading and installing ports that are dependencies of a
> lot of other ports.
>
> But there are also some inordinately large ports, such as some games,
> which aren't commonly used as dependencies. Here are some of the larger
> ports installed on the highsierra buildbot worker:
>
> {{{
> $ cd /opt/local/var/macports/software
> $ du -sch alienarena-data/* geant*-data/* knp/* supertuxkart/* texlive-
> latex-extra/* typesafe-activator/* wesnoth/*
> 551M    alienarena-data/alienarena-
> data-7.66-20130827_0.darwin_16.noarch.tbz2
> 427M    geant4.10.0-data/geant4.10.0-data-4.10.0_0.darwin_16.noarch.tbz2
> 428M    geant4.10.1-data/geant4.10.1-data-4.10.1_0.darwin_16.noarch.tbz2
> 432M    geant4.10.2-data/geant4.10.2-data-4.10.2_1.darwin_16.noarch.tbz2
> 1005M   geant4.10.3-data/geant4.10.3-data-4.10.3_0.darwin_16.noarch.tbz2
> 1.1G    geant4.10.4-data/geant4.10.4-data-4.10.4_0.darwin_16.noarch.tbz2
> 266M    geant4.9.6-data/geant4.9.6-data-4.9.6_0.darwin_16.noarch.tbz2
> 1.0G    knp/knp-4.14_0.darwin_16.x86_64.tbz2
> 512M    supertuxkart/supertuxkart-0.9.3_0.darwin_16.x86_64.tbz2
> 453M    texlive-latex-extra/texlive-latex-extra-
> 47410_0+doc.darwin_16.noarch.tbz2
> 663M    typesafe-activator/typesafe-
> activator-1.3.12_0.darwin_16.noarch.tbz2
> 365M    wesnoth/wesnoth-1.12.6_2.darwin_16.x86_64.tbz2
> 7.1G    total
> }}}
>
> It's getting to the point where builds are failing because there's not
> enough free disk space, and I am manually uninstalling some of these
> ports on the buildbot workers to free up space. If we could have mpbb
> automatically uninstall large ports, it would free up a good deal of disk
> space.
>
> I'm not sure what do about dependencies. For example, geant4.10.4 is not
> large, but it depends on geant4.10.4-data which is.
>
> Do we force the uninstallation of the large ports, leaving the ports
> depending on them broken, on the assumption that when the buildbot asks
> to install them again it will reinstall the dependencies and fix them?
>
> Or do we have the buildbot uninstall not only the large ports but also
> all ports depending on them? That would be good for geant, but bad for
> texlive-latex-extra, on which lots of things depend. We could base the
> decision of whether to uninstall on how many ports would be uninstalled.
> (It it's, say, 5 or fewer ports, uninstall them, otherwise leave them
> installed.)
>
> Or do we instead make a general rule (not based on size) that we don't
> keep a port installed if nothing depends on it?
>
> Some of these ports might not be distributable. Once we have
> [ticket:54800 nondistributable archives available on a private server]
> that won't be a big problem anymore. We can wait until we resolve that
> ticket before deploying an implementation for this one.
New description:

 Our buildbot setup currently keeps all built ports installed on each
 worker. This is supposed to speed up builds so that we don't waste time
 constantly downloading and installing ports that are dependencies of a lot
 of other ports.

 But there are also some inordinately large ports, such as some games,
 which aren't commonly used as dependencies. Here are some of the larger
 ports installed on the highsierra buildbot worker:

 {{{
 $ cd /opt/local/var/macports/software
 $ du -sch alienarena-data/* geant*-data/* knp/* supertuxkart/*
 texlive-{fonts,latex}-extra/* typesafe-activator/* wesnoth/*
 551M    alienarena-data/alienarena-
 data-7.66-20130827_0.darwin_17.noarch.tbz2
 427M    geant4.10.0-data/geant4.10.0-data-4.10.0_0.darwin_17.noarch.tbz2
 428M    geant4.10.1-data/geant4.10.1-data-4.10.1_0.darwin_17.noarch.tbz2
 432M    geant4.10.2-data/geant4.10.2-data-4.10.2_1.darwin_17.noarch.tbz2
 1005M   geant4.10.3-data/geant4.10.3-data-4.10.3_0.darwin_17.noarch.tbz2
 1.1G    geant4.10.4-data/geant4.10.4-data-4.10.4_0.darwin_17.noarch.tbz2
 266M    geant4.9.6-data/geant4.9.6-data-4.9.6_0.darwin_17.noarch.tbz2
 1.0G    knp/knp-4.14_0.darwin_17.x86_64.tbz2
 512M    supertuxkart/supertuxkart-0.9.3_0.darwin_17.x86_64.tbz2
 534M    texlive-fonts-extra/texlive-fonts-extra-
 47288_0+doc.darwin_17.noarch.tbz2
 453M    texlive-latex-extra/texlive-latex-extra-
 47410_0+doc.darwin_17.noarch.tbz2
 663M    typesafe-activator/typesafe-
 activator-1.3.12_0.darwin_17.noarch.tbz2
 365M    wesnoth/wesnoth-1.12.6_2.darwin_17.x86_64.tbz2
 7.7G    total
 }}}

 It's getting to the point where builds are failing because there's not
 enough free disk space, and I am manually uninstalling some of these ports
 on the buildbot workers to free up space. If we could have mpbb
 automatically uninstall large ports, it would free up a good deal of disk
 space.

 I'm not sure what do about dependencies. For example, geant4.10.4 is not
 large, but it depends on geant4.10.4-data which is.

 Do we force the uninstallation of the large ports, leaving the ports
 depending on them broken, on the assumption that when the buildbot asks to
 install them again it will reinstall the dependencies and fix them?

 Or do we have the buildbot uninstall not only the large ports but also all
 ports depending on them? That might be ok for geant, but would be bad for
 texlive-latex-extra, on which lots of things depend. We could base the
 decision of whether to uninstall on how many ports would be uninstalled.
 (It it's, say, 5 or fewer ports, uninstall them, otherwise leave them
 installed.)

 Or do we instead make a general rule (not based on size) that we don't
 keep a port installed if nothing depends on it?

 Some of these ports might not be distributable. Once we have [ticket:54800
 nondistributable archives available on a private server] that won't be a
 big problem anymore. We can wait until we resolve that ticket before
 deploying an implementation for this one.

--

--
Ticket URL: <https://trac.macports.org/ticket/57464#comment:1>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #57464: Builtbot: Don't keep huge ports installed

MacPorts
In reply to this post by MacPorts
#57464: Builtbot: Don't keep huge ports installed
----------------------------+---------------------
  Reporter:  ryandesign     |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:
      Port:                 |
----------------------------+---------------------

Comment (by mojca):

 Just a short note. I'm a maintainer of geant4 and the data files are
 nothing else but "fetch that gigabyte of data & extract it". There's
 basically no penalty in building them (there are no build steps) and the
 files are exactly the same for all OS versions. After you bought this up
 ... maybe I should also add some restrictive licence just for the sake of
 decreasing the space we use on our mirror for binaries. This wouldn't
 solve the issue with storing them on the builder, but it would at least
 help a tiny tiny bit somewhere else. I would be totally in favour of
 simply always deleting geant4 data files from the builders, even if I'm
 not sure about the best way. If you run some kind of a cleanup job ...
 simply put the geant-data files uncoditionally there.

 The only dependent port (Gate) is broken as I'm stuck in the upgrade
 process (the last few times when I tried it failed to build and I didn't
 spend enough time trying to come up with a proper fix; that said, I'm not
 aware of anyone else offering Gate as a dmg, so it would be cool to fix it
 rather than deleting it).

--
Ticket URL: <https://trac.macports.org/ticket/57464#comment:2>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #57464: Builtbot: Don't keep huge ports installed

MacPorts
In reply to this post by MacPorts
#57464: Builtbot: Don't keep huge ports installed
----------------------------+---------------------
  Reporter:  ryandesign     |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:
      Port:                 |
----------------------------+---------------------

Comment (by mojca):

 Btw: a rule that says "uninstall the port unless something depends on it"
 won't help you a tiny bit in case of Geant since:
 * `geant4.x` depends on `geant4.x-data`
 * `gate` depends on `geant4.x`
 So in case of geant: feel free to delete both (times different versions of
 geant, so probably 12 ports). We should keep `geant4.x` on the binary
 package server, but it does absolutely nothing useful on the builder. I
 may also remove the older versions one day.

--
Ticket URL: <https://trac.macports.org/ticket/57464#comment:4>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|

Re: [MacPorts] #57464: Builtbot: Don't keep huge ports installed

MacPorts
In reply to this post by MacPorts
#57464: Builtbot: Don't keep huge ports installed
----------------------------+---------------------
  Reporter:  ryandesign     |      Owner:  admin@…
      Type:  enhancement    |     Status:  new
  Priority:  Normal         |  Milestone:
 Component:  buildbot/mpbb  |    Version:
Resolution:                 |   Keywords:
      Port:                 |
----------------------------+---------------------

Comment (by ryandesign):

 Replying to [comment:4 mojca]:
 > Btw: a rule that says "uninstall the port unless something depends on
 it" won't help you a tiny bit in case of Geant

 I understand, I'm just trying to come up with multiple ideas for saving
 space on the workers. One idea is to uninstall huge ports. Another idea is
 to uninstall ports that nothing in MacPorts depends on. I'm not sure which
 would save more space. We could implement one or the other or both. We're
 already saving some space by [ticket:57396 using hfsCompression] on the
 tools prefix. We could save a bit more space by using it on the main
 prefix as well, though that would probably impact performance.

 What would be a good way to identify whether anything in MacPorts depends
 on a port? I would want "anything" to include ports that are not yet
 installed. Do we do something like this?

 {{{
 port=wine-crossover
 if [ -z "$(/opt/local/bin/port echo depends:":$port(\s|$)")" ]; then
     echo uninstall it
 fi
 }}}

--
Ticket URL: <https://trac.macports.org/ticket/57464#comment:5>
MacPorts <https://www.macports.org/>
Ports system for macOS