Quantcast

[MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
----------------------------+---------------------
 Reporter:  mojca           |      Owner:  admin@…
     Type:  enhancement     |     Status:  new
 Priority:  Normal          |  Milestone:
Component:  server/hosting  |    Version:
 Keywords:  buildbot        |       Port:
----------------------------+---------------------
 The 10.6-i386 buildbot avoids building many packages:
 {{{
 #!tcl
 if [[ $exclude -eq 0 && "${os_version}" = "10.6" && "${is_64bit_capable}"
 = "0" ]]; then
     supported_archs=$("${option_prefix}/bin/port-tclsh" "${thisdir}/tools
 /supported-archs.tcl" "${port}")
     if [[ -n "${supported_archs}" && ! ("${supported_archs}" == *"x86_64"*
 && "${supported_archs}" == *"i386"*) ]]; then
         exclude=1
         exclude_reasons+=("the ${os_version}_x86_64 builder will build
 it")
     fi
 fi
 }}}
 I believe that a similar strategy could be used to avoid building packages
 that are declared not to be supported on PPC.

 See [https://build.macports.org/builders/ports-10.5_ppc_legacy-
 builder/builds/11580/steps/install-port/logs/stdio an example] of the
 libvpx port which doesn't even start to build:
 {{{
 Error: libvpx cannot be installed for the configured build_arch 'ppc'
 because it only supports the arch(s) 'x86_64 i386'.
 Error: Processing of port libvpx failed
 Build of 'libvpx' failed.
 }}}

--
Ticket URL: <https://trac.macports.org/ticket/52986>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Old description:

> The 10.6-i386 buildbot avoids building many packages:
> {{{
> #!tcl
> if [[ $exclude -eq 0 && "${os_version}" = "10.6" && "${is_64bit_capable}"
> = "0" ]]; then
>     supported_archs=$("${option_prefix}/bin/port-tclsh" "${thisdir}/tools
> /supported-archs.tcl" "${port}")
>     if [[ -n "${supported_archs}" && ! ("${supported_archs}" ==
> *"x86_64"* && "${supported_archs}" == *"i386"*) ]]; then
>         exclude=1
>         exclude_reasons+=("the ${os_version}_x86_64 builder will build
> it")
>     fi
> fi
> }}}
> I believe that a similar strategy could be used to avoid building
> packages that are declared not to be supported on PPC.
>
> See [https://build.macports.org/builders/ports-10.5_ppc_legacy-
> builder/builds/11580/steps/install-port/logs/stdio an example] of the
> libvpx port which doesn't even start to build:
> {{{
> Error: libvpx cannot be installed for the configured build_arch 'ppc'
> because it only supports the arch(s) 'x86_64 i386'.
> Error: Processing of port libvpx failed
> Build of 'libvpx' failed.
> }}}

New description:

 The 10.6-i386 buildbot avoids building many packages:
 {{{
 #!bash
 if [[ $exclude -eq 0 && "${os_version}" = "10.6" && "${is_64bit_capable}"
 = "0" ]]; then
     supported_archs=$("${option_prefix}/bin/port-tclsh" "${thisdir}/tools
 /supported-archs.tcl" "${port}")
     if [[ -n "${supported_archs}" && ! ("${supported_archs}" == *"x86_64"*
 && "${supported_archs}" == *"i386"*) ]]; then
         exclude=1
         exclude_reasons+=("the ${os_version}_x86_64 builder will build
 it")
     fi
 fi
 }}}
 I believe that a similar strategy could be used to avoid building packages
 that are declared not to be supported on PPC.

 See [https://build.macports.org/builders/ports-10.5_ppc_legacy-
 builder/builds/11580/steps/install-port/logs/stdio an example] of the
 libvpx port which doesn't even start to build:
 {{{
 Error: libvpx cannot be installed for the configured build_arch 'ppc'
 because it only supports the arch(s) 'x86_64 i386'.
 Error: Processing of port libvpx failed
 Build of 'libvpx' failed.
 }}}

--

Comment (by larryv):

 Most of this fiddling should be done in the Buildbot configuration and not
 in mpbb.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:1>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 (Thanks for fixing my stupid mistake.)

 No, I don't believe that the buildbot can possibly do that. Buildbot is
 just a stupid python script that doesn't know anything about ports unless
 we create an up-to-date database of all ports with all the required meta
 information on the build master ... or install MacPorts on the build
 master, but then it has to be able to switch between different OS versions
 for the portindex etc.

 Until we implement that, the decision about whether or not a port is
 installable should be done by the build slave in my opinion.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:2>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by larryv):

 The build step that invokes `mpbb list-subports` can specify the
 architecture(s) it can build, and mpbb can compare that to the ports’
 `supported_archs` and filter appropriately.

 mpbb’s knowledge of which buildslaves we are running (and any limitations
 thereof) is needlessly tight coupling that we should eliminate.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:3>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 I only partially agree. The above mentioned step only avoids building
 ports like `wine` on 10.6-i386 because 10.6-x86_64 already builds it and
 you cannot delegate the job of building wine to the i386 builder as you
 wouldn't get all the dependencies built universally. It's not really clear
 to me how you would simplify the code by telling the i386 builder to build
 for i386. Just about the only thing that could be simplified is the need
 for testing:
 {{{
 #!bash
 is_64bit_capable="$(sysctl -n hw.cpu64bit_capable)"
 ...
 if [[ ... "${is_64bit_capable}" = "0"... ]]; then
 }}}

 For 10.5 we would currently run `uname -m`. With your suggested changes we
 would have to call
 {{{
 mpbb list-subports -arch ppc ...
 }}}
 or provide an environmental variable, but I don't see how that simplifies
 things in any way.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:4>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by jmroot):

 One note: `uname -m` is not correct here (or in the buildbot README.md).
 It tells you the architecture of the kernel, and K64 is sufficient but not
 necessary for running 64-bit programs.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:5>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 That's already nitpicking. I'm talking about determining whether
 computer's architecture is PPC.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:6>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 The problem with `tools/supported-archs.tcl` is that it only returns
 something when the port specifies `supported_archs` explicitly.

 So one can get `i386 ppc` or `i386 x86_64` or `i386 ppc x86_64`, ... or
 `noarch` or an empty string as the answer. I'm not sure how to reliably
 tell whether a port builds on PPC. Just guessing:
 * If the answer is not empty and not `noarch`, then `ppc` must be present,
 else skip building the port.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:7>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 Off-topic.

 I just wanted to add one strange observation:
 {{{
 > time /opt/local/bin/port-tclsh ./tools/supported-archs.tcl wine
 i386

 real    0m26.122s
 }}}
 What's the deal with the super long waiting time? Sure, I don't have SSD
 yet and any subsequent run is much faster, but I'm still completely
 puzzled why it was so long in the first place (and we are experiencing
 super long time for trivial tasks on the buildbot all the time).

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:8>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by ryandesign):

 Replying to [ticket:52986 mojca]:
 > The 10.6-i386 buildbot avoids building many packages:
 > {{{
 > #!bash
 > if [[ $exclude -eq 0 && "${os_version}" = "10.6" &&
 "${is_64bit_capable}" = "0" ]]; then
 >     supported_archs=$("${option_prefix}/bin/port-tclsh"
 "${thisdir}/tools/supported-archs.tcl" "${port}")
 >     if [[ -n "${supported_archs}" && ! ("${supported_archs}" ==
 *"x86_64"* && "${supported_archs}" == *"i386"*) ]]; then
 >         exclude=1
 >         exclude_reasons+=("the ${os_version}_x86_64 builder will build
 it")
 >     fi
 > fi
 > }}}

 This code is there because it is buildbot-specific. The i386 builder
 ''could'' build i386-only ports, but we don't want it to, because want to
 build them on the x86_64 builder, so that their dependencies get built
 universal.

 > I believe that a similar strategy could be used to avoid building
 packages that are declared not to be supported on PPC.

 We've discussed before that this is a generic MacPorts problem, not a
 buildbot-specific one. Instead of adding code for this to our buildbot
 system, we should write code in MacPorts base so that a client (e.g. a
 user or the buildbot) can determine in advance whether a port is buildable
 on a particular system. Then the buildbot can simply avoid attempting to
 build ports that are known not to be buildable on a particular system.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:9>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 I fully agree that the base should tell whether a port is "buildable". But
 until we have a working code to do that (which might take quite a while -
 both discussing and implementing), it might be a matter of just a few
 lines of code to avoid building some ports – similar to what we do with
 obsolete ports.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:10>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by jmroot):

 Replying to [comment:8 mojca]:
 > Off-topic.
 >
 > I just wanted to add one strange observation:
 > {{{
 > > time /opt/local/bin/port-tclsh ./tools/supported-archs.tcl wine
 > i386
 >
 > real  0m26.122s
 > }}}
 > What's the deal with the super long waiting time? Sure, I don't have SSD
 yet and any subsequent run is much faster, but I'm still completely
 puzzled why it was so long in the first place (and we are experiencing
 super long time for trivial tasks on the buildbot all the time).

 I would again suggest becoming familiar with using the profiler package.
 This series of blog posts isn't a bad introduction, though the sample code
 may need some tweaking:

 https://wuhrr.wordpress.com/2011/01/14/simple-profile-for-your-tcl-script/
 https://wuhrr.wordpress.com/2011/01/15/profiling-your-tcl-show-profilers-
 summary/
 https://wuhrr.wordpress.com/2011/01/19/profiling-your-tcl-output-to-csv/

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:11>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 (Sorry to everyone for the off-topic discussion.)

 Thank you, I ended up with the following script (suggestions welcome):

 {{{
 #!tcl
 #!/opt/local/bin/port-tclsh

 # File: profile_me
 package require profiler
 profiler::init

 proc srcx {} {
   set argv $::argv
   set argc $::argc

   set file   [lindex $::argv 0]
   set ::argv [lreplace $::argv 0 0]
   set ::argc [llength $::argv]
   set code [catch {uplevel [list source $file]} return]

   set ::argv $argv
   set ::argc $argc
   return -code $code $return
 }

 srcx

 # Display profiler info
 puts "\n[string repeat - 40]"
 puts [::profiler::print]
 }}}
 and `time ./profileme.tcl ./tools/supported-archs.tcl wine`

 but I need to wait. There seems to be some cache somewhere and only the
 first call is super slow, any subsequent calls are fast.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:12>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------

Comment (by mojca):

 I absolutely love the results of profiling (this was a fast run though):
 {{{
 Functions that are sorted by exclusiveRuntime:
 -   15312828 - ::lshift
 -   12601898 - ::puts
 -    7224617 - ::ldindex
 -    2379654 - ::lpush
 -    1840049 - ::lpop
 -    1336408 - ::_source_is_snapshot
 -     816458 - ::ditem_key
 ...
 -          0 - ::intersect
 -    -138321 - ::macports::setxcodeinfo
 -    -270916 - ::macports::set_developer_dir
 -    -479661 - ::mportinit
 -    -554829 - ::_mports_load_quickindex
 -    -558463 - ::mportclose
 -    -564781 - ::macports::ui_init
 -    -666930 - ::macports::getindex
 -    -688013 - ::mportlookup
 -   -1302686 - ::msgcat::Init
 -   -2878285 - ::mportopen
 -  -26476148 - ::try
 }}}

 We have negative times, yay!

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:13>
MacPorts <https://www.macports.org/>
Ports system for macOS
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [MacPorts] #52986: buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

MacPorts
In reply to this post by MacPorts
#52986: buildbot: don't add ports that declare not to support ppc to the build
queue of 10.5-ppc
-----------------------------+----------------------
  Reporter:  mojca           |      Owner:  admin@…
      Type:  enhancement     |     Status:  closed
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:  fixed           |   Keywords:  buildbot
      Port:                  |
-----------------------------+----------------------
Changes (by mojca):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 This was fixed in
 [changeset:"258d416705164f7f2a272ef3d5eb520d2a484229/mpbb" 258d4167].

 Long delays were fixed (worked around) as well.

--
Ticket URL: <https://trac.macports.org/ticket/52986#comment:14>
MacPorts <https://www.macports.org/>
Ports system for macOS
Loading...