Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Ryan Schmidt-24

On Apr 10, 2018, at 15:39, Mojca Miklavec wrote:

> Mojca Miklavec (mojca) pushed a commit to branch master
> in repository macports-ports.
>
>
> https://github.com/macports/macports-ports/commit/6c05e9852d0b08b2a8f14e7ee622031196a9d699
>
> The following commit(s) were added to refs/heads/master by this push:
>
>      new 6c05e98  py-*: add missing subports for python 3.5 and 3.6

The thousands of builds this commit has scheduled on the buildbot have brought it to its knees. For whatever reason, the more scheduled builds there are, the longer the buildmaster takes to decide which build to start next, to the extent that for the past 28 hours, most of the buildbot's time has been spend deciding what to build next, rather than actually building anything. At this rate, it'll take a week or more to finish the builds for this commit. Meanwhile, requests to build commits that happened later are piling up.

We're not specifying any priority for the builders [1], and so because of the order in which the builders are initialized in our buildmaster config, it appears that most of the builds that have been completed for this commit are on the High Sierra builder. I'm going to cancel the pending builds for this commit on all the other builders, and I'll reschedule them later in smaller batches, after the other pending builds have had a change to finish.

The developer of buildbot thinks we may be able to improve buildbot's performance when there are so many pending builds, by switching to a postgresql database [2]. I intend to do that, when I have time.

[1] https://trac.macports.org/ticket/56273

[2] https://lists.buildbot.net/pipermail/users/2016-October/000829.html

Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Joshua Root-8
On 2018-4-12 10:47 , Ryan Schmidt wrote:
> The developer of buildbot thinks we may be able to improve buildbot's performance when there are so many pending builds, by switching to a postgresql database [2]. I intend to do that, when I have time.

If switching databases would help, that would mean we're doing a lot of
queries in getNextBuildOnPortBuilder, which would mean
request.properties.getProperty involves a query. So that means we should:

(a) only do the queries once for each set of build requests, and
(b) use Deferred so the main thread isn't waiting for every query to finish.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Ryan Schmidt-24

On Apr 11, 2018, at 20:47, Joshua Root wrote:

> On 2018-4-12 10:47 , Ryan Schmidt wrote:
>> The developer of buildbot thinks we may be able to improve buildbot's performance when there are so many pending builds, by switching to a postgresql database [2]. I intend to do that, when I have time.
>
> If switching databases would help, that would mean we're doing a lot of
> queries in getNextBuildOnPortBuilder, which would mean
> request.properties.getProperty involves a query. So that means we should:
>
> (a) only do the queries once for each set of build requests, and
> (b) use Deferred so the main thread isn't waiting for every query to finish.

These performance problems predate the existence of our getNextBuildOnPortBuilder function.

It smells to me like a missing database index problem. I don't know how to determine that though.

Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Joshua Root-8
On 2018-4-12 11:49 , Ryan Schmidt wrote:

>
> On Apr 11, 2018, at 20:47, Joshua Root wrote:
>
>> On 2018-4-12 10:47 , Ryan Schmidt wrote:
>>> The developer of buildbot thinks we may be able to improve buildbot's performance when there are so many pending builds, by switching to a postgresql database [2]. I intend to do that, when I have time.
>>
>> If switching databases would help, that would mean we're doing a lot of
>> queries in getNextBuildOnPortBuilder, which would mean
>> request.properties.getProperty involves a query. So that means we should:
>>
>> (a) only do the queries once for each set of build requests, and
>> (b) use Deferred so the main thread isn't waiting for every query to finish.
>
> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>
> It smells to me like a missing database index problem. I don't know how to determine that though.

This thing may be of use: <https://pypi.python.org/pypi/buildbot-profiler>

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Joshua Root-8
In reply to this post by Ryan Schmidt-24
On 2018-4-12 11:49 , Ryan Schmidt wrote:
>
> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>
> It smells to me like a missing database index problem. I don't know how to determine that though.

I'm guessing we probably run into trouble when the number of pending
builds exceeds our cachedbuildrequests number. Is that at the default
1000 as set in master.cfg?

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Ryan Schmidt-24

On Apr 11, 2018, at 21:22, Joshua Root wrote:

> On 2018-4-12 11:49 , Ryan Schmidt wrote:
>>
>> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>>
>> It smells to me like a missing database index problem. I don't know how to determine that though.
>
> I'm guessing we probably run into trouble when the number of pending
> builds exceeds our cachedbuildrequests number. Is that at the default
> 1000 as set in master.cfg?

Aha! Sure, I don't know why its value would be anything other than what we've told it to be in master.cfg. So we should increase that to a much higher number. Along with maybe some of the other values.

Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Joshua Root-8
On 2018-4-12 12:23 , Ryan Schmidt wrote:

>
> On Apr 11, 2018, at 21:22, Joshua Root wrote:
>
>> On 2018-4-12 11:49 , Ryan Schmidt wrote:
>>>
>>> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>>>
>>> It smells to me like a missing database index problem. I don't know how to determine that though.
>>
>> I'm guessing we probably run into trouble when the number of pending
>> builds exceeds our cachedbuildrequests number. Is that at the default
>> 1000 as set in master.cfg?
>
> Aha! Sure, I don't know why its value would be anything other than what we've told it to be in master.cfg. So we should increase that to a much higher number. Along with maybe some of the other values.

Well you can change it in config.json. But yes, if we have the memory to
spare, that would help.

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Ryan Schmidt-24

On Apr 11, 2018, at 21:36, Joshua Root wrote:

> On 2018-4-12 12:23 , Ryan Schmidt wrote:
>>
>> On Apr 11, 2018, at 21:22, Joshua Root wrote:
>>
>>> On 2018-4-12 11:49 , Ryan Schmidt wrote:
>>>>
>>>> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>>>>
>>>> It smells to me like a missing database index problem. I don't know how to determine that though.
>>>
>>> I'm guessing we probably run into trouble when the number of pending
>>> builds exceeds our cachedbuildrequests number. Is that at the default
>>> 1000 as set in master.cfg?
>>
>> Aha! Sure, I don't know why its value would be anything other than what we've told it to be in master.cfg. So we should increase that to a much higher number. Along with maybe some of the other values.
>
> Well you can change it in config.json. But yes, if we have the memory to
> spare, that would help.

It seems to be much happier now that cachedbuildrequests has been increased. Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Ryan Schmidt-24
Mojca, I think you're going to need to add some files (in the files directory) for these ports:

py-django-htmlmin
py-hieroglyph
py-ioflo
py-local-pipelines
py-memprof
py-obspy-devel
py-pss


Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Mojca Miklavec-2
In reply to this post by Ryan Schmidt-24
On 12 April 2018 at 04:23, Ryan Schmidt wrote:

> On Apr 11, 2018, at 21:22, Joshua Root wrote:
>> On 2018-4-12 11:49 , Ryan Schmidt wrote:
>>>
>>> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>>>
>>> It smells to me like a missing database index problem. I don't know how to determine that though.
>>
>> I'm guessing we probably run into trouble when the number of pending
>> builds exceeds our cachedbuildrequests number. Is that at the default
>> 1000 as set in master.cfg?
>
> Aha! Sure, I don't know why its value would be anything other than what we've told it to be in master.cfg. So we should increase that to a much higher number.

Joshua, thank you very much for figuring out what the problem was and
saving our buildbot.

> Along with maybe some of the other values.

You might want to increase the number of builds retained in history. I
suspect that currently the builds get deleted after roughly 10.000 new
builds come in.


I owe a deep apology. I did commit perl updates (with more than 1k
ports) a couple of times without ever experience a problem as big as
this one. This list seemed super short in comparison, but I didn't
anticipate that so many perl subports would be broken and that the
list would end up containing more than 1k ports per builder.

The queue for 10.13 has finally been digested, so I'll go over the
list of failed ports and probably open a ticket to track progress on
fixing the issues.

Mojca
Reply | Threaded
Open this post in threaded view
|

Re: [macports-ports] branch master updated: py-*: add missing subports for python 3.5 and 3.6

Ryan Schmidt-24

On Apr 12, 2018, at 02:53, Mojca Miklavec wrote:

> On 12 April 2018 at 04:23, Ryan Schmidt wrote:
>> On Apr 11, 2018, at 21:22, Joshua Root wrote:
>>> On 2018-4-12 11:49 , Ryan Schmidt wrote:
>>>>
>>>> These performance problems predate the existence of our getNextBuildOnPortBuilder function.
>>>>
>>>> It smells to me like a missing database index problem. I don't know how to determine that though.
>>>
>>> I'm guessing we probably run into trouble when the number of pending
>>> builds exceeds our cachedbuildrequests number. Is that at the default
>>> 1000 as set in master.cfg?
>>
>> Aha! Sure, I don't know why its value would be anything other than what we've told it to be in master.cfg. So we should increase that to a much higher number.
>
> Joshua, thank you very much for figuring out what the problem was and
> saving our buildbot.
>
>> Along with maybe some of the other values.
>
> You might want to increase the number of builds retained in history. I
> suspect that currently the builds get deleted after roughly 10.000 new
> builds come in.

Yes, we do only keep the 10,000 most recent builds per builder. Do you really think we need to keep more than that?


> I owe a deep apology. I did commit perl updates (with more than 1k
> ports) a couple of times without ever experience a problem as big as
> this one. This list seemed super short in comparison, but I didn't
> anticipate that so many perl subports would be broken and that the
> list would end up containing more than 1k ports per builder.

No worries. I appreciate the work you're doing, and we need to fix infrastructure such as buildbot to be compatible with how we work, so I'm glad we figured out how to let the buildbot deal with a larger number of pending builds.

I'm not sure why the mass perl changes before worked better...


> The queue for 10.13 has finally been digested, so I'll go over the
> list of failed ports and probably open a ticket to track progress on
> fixing the issues.

Dave has been adding some missing select files and missing setuptools dependencies.