Switching open source Debug bots to building and testing with configuration --force-opt=O3

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Switching open source Debug bots to building and testing with configuration --force-opt=O3

Mark Lam
Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Alexey Proskuryakov-4

I frequently find it critically useful to see stack traces from debug builds, because of no inlining. So I don't think that we should do this. A local build does not help when the issue is not readily reproducible.

- Alexey


17 июня 2020 г., в 1:36 PM, Mark Lam <[hidden email]> написал(а):

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Yusuke Suzuki-3
For EWS, we are running Release build right now. So in terms of stack traces, nothing is changed.

-Yusuke

On Jun 17, 2020, at 5:21 PM, Alexey Proskuryakov <[hidden email]> wrote:


I frequently find it critically useful to see stack traces from debug builds, because of no inlining. So I don't think that we should do this. A local build does not help when the issue is not readily reproducible.

- Alexey


17 июня 2020 г., в 1:36 PM, Mark Lam <[hidden email]> написал(а):

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Yusuke Suzuki-3
In JSC EWS bots. For the other Debug EWS bots, (Like, WebCore Debug builds), I have no strong opinion.

-Yusuke

On Jun 17, 2020, at 6:07 PM, Yusuke Suzuki <[hidden email]> wrote:

For EWS, we are running Release build right now. So in terms of stack traces, nothing is changed.

-Yusuke

On Jun 17, 2020, at 5:21 PM, Alexey Proskuryakov <[hidden email]> wrote:


I frequently find it critically useful to see stack traces from debug builds, because of no inlining. So I don't think that we should do this. A local build does not help when the issue is not readily reproducible.

- Alexey


17 июня 2020 г., в 1:36 PM, Mark Lam <[hidden email]> написал(а):

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Simon Fraser-6
In reply to this post by Mark Lam
I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Alex Christensen-2
It sounds to me like Mark’s suggestion does not lose anything.  It’s just for JSC “Debug” which currently is not running because it’s too slow.  If he called it “ReleaseWithAssert” it would make it more clear what is going on and we would all appreciate the additional information those bots provide.

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Ryan Haddad

On Jun 18, 2020, at 9:44 AM, Alex Christensen <[hidden email]> wrote:

It sounds to me like Mark’s suggestion does not lose anything.  It’s just for JSC “Debug”

The post-commit JSC bots use the same build products as the other debug testers, so with our current setup it would have to apply to other queues as well.

Ryan

which currently is not running because it’s too slow.  If he called it “ReleaseWithAssert” it would make it more clear what is going on and we would all appreciate the additional information those bots provide.

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Geoffrey Garen
Better JSC debugging in exchange for worse debugging in non-JSC code that calls through to WTF is not a great tradeoff.

Is there a way to localize this change to only JSC?

Do we know for sure that this change would get stress tests running, or are we just guessing? Can we find out? It’s easier to weight a tradeoff between known quantities than it is to weigh a tradeoff between hoped-for quantities.

Thanks,
Geoff

On Jun 18, 2020, at 9:55 AM, Ryan Haddad <[hidden email]> wrote:


On Jun 18, 2020, at 9:44 AM, Alex Christensen <[hidden email]> wrote:

It sounds to me like Mark’s suggestion does not lose anything.  It’s just for JSC “Debug”

The post-commit JSC bots use the same build products as the other debug testers, so with our current setup it would have to apply to other queues as well.

Ryan

which currently is not running because it’s too slow.  If he called it “ReleaseWithAssert” it would make it more clear what is going on and we would all appreciate the additional information those bots provide.

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Saam barati-2
In reply to this post by Simon Fraser-6
Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Geoffrey Garen
Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?

And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.

And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.

Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Alexey Proskuryakov-4
In reply to this post by Saam barati-2


18 июня 2020 г., в 9:30 PM, Saam Barati <[hidden email]> написал(а):

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Replying to this point specifically, I wanted to point out that WebKit tests take 2x longer in debug, not 10x longer. JSC tests take 3.6x longer.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:

Last time this was discussed, we talked about -Og, which is specifically designed for the purpose I believe. Where do we stand on understanding and adopting that?

- Alexey

- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

- Alexey



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Mark Lam


On Jun 19, 2020, at 9:53 AM, Alexey Proskuryakov <[hidden email]> wrote:



18 июня 2020 г., в 9:30 PM, Saam Barati <[hidden email]> написал(а):

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Replying to this point specifically, I wanted to point out that WebKit tests take 2x longer in debug, not 10x longer. JSC tests take 3.6x longer.

Since I collected real data on this last night on the actual bot that runs the JSC stress tests, I’ll share the data here:

Build time
A clean build using buid-jsc for a normal Debug build on bot638 takes about 4.5 minutes.
A clean build using build-jsc for a --force-opt=O3 Debug build on bot638 takes about 6 minutes.

Test run time
Running with a regular Debug build, the test completed in about 4 hours 41 minutes with 1 timeout.
Running with a --force-opt=O3 Debug build, the test completed in about 39 minutes with 0 timeouts.

The difference in test run time is 281 minutes vs 29 minutes.  That is a 7.2x ratio, not 3.6x.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:

Last time this was discussed, we talked about -Og, which is specifically designed for the purpose I believe. Where do we stand on understanding and adopting that?

I tried -Og last time after your suggestion, and I remember thinking that the perf was not acceptable back then.  I’ll collect the data again and report back with real number later today.

Mark


- Alexey

- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

- Alexey


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Saam barati-2
In reply to this post by Geoffrey Garen


On Jun 19, 2020, at 8:04 AM, Geoffrey Garen <[hidden email]> wrote:

Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?
There doesn't exist a way to do this now, but it'd be trivial to add a way. I won't claim it fixes all stack traces differences, but I'd think compiling using "-fno-inline -fno-optimize-sibling-calls" would get us pretty far in crashing stack traces being similar enough.


And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.
I believe Mark's proposal, initially, is just to make JSC do this. So I don't see the point of compiling WTF differently. JSC can kick off its own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given people working on JSC want this, and people working on JSC defend these tests, and that these test results are more stable (see below), we should make this change for JSC.

I was trying to convince folks defending non-JSC testing, that they too, should want this. I'm not going to pull teeth here. If folks want their tests to take ~10x longer to run, they're entitled to make that tradeoff.


And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.
My experience running the tests locally supports this fully. I don't get timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd get timeouts all the time, and the total test run would take ~4-8 hours. We can wait for official confirmation from Mark.

- Saam


Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Alexey Proskuryakov-4
In reply to this post by Mark Lam


19 июня 2020 г., в 9:59 AM, Mark Lam <[hidden email]> написал(а):



On Jun 19, 2020, at 9:53 AM, Alexey Proskuryakov <[hidden email]> wrote:



18 июня 2020 г., в 9:30 PM, Saam Barati <[hidden email]> написал(а):

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Replying to this point specifically, I wanted to point out that WebKit tests take 2x longer in debug, not 10x longer. JSC tests take 3.6x longer.

Since I collected real data on this last night on the actual bot that runs the JSC stress tests, I’ll share the data here:

Build time
A clean build using buid-jsc for a normal Debug build on bot638 takes about 4.5 minutes.
A clean build using build-jsc for a --force-opt=O3 Debug build on bot638 takes about 6 minutes.

Test run time
Running with a regular Debug build, the test completed in about 4 hours 41 minutes with 1 timeout.
Running with a --force-opt=O3 Debug build, the test completed in about 39 minutes with 0 timeouts.

The difference in test run time is 281 minutes vs 29 minutes.  That is a 7.2x ratio, not 3.6x.


That's 3.6x.

- Alexey



Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:

Last time this was discussed, we talked about -Og, which is specifically designed for the purpose I believe. Where do we stand on understanding and adopting that?

I tried -Og last time after your suggestion, and I remember thinking that the perf was not acceptable back then.  I’ll collect the data again and report back with real number later today.

Mark


- Alexey

- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

- Alexey


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev




_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Saam barati-2
In reply to this post by Saam barati-2


On Jun 19, 2020, at 10:02 AM, Saam Barati <[hidden email]> wrote:



On Jun 19, 2020, at 8:04 AM, Geoffrey Garen <[hidden email]> wrote:

Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?
There doesn't exist a way to do this now, but it'd be trivial to add a way. I won't claim it fixes all stack traces differences, but I'd think compiling using "-fno-inline -fno-optimize-sibling-calls" would get us pretty far in crashing stack traces being similar enough.


And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.
I believe Mark's proposal, initially, is just to make JSC do this. So I don't see the point of compiling WTF differently. JSC can kick off its own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given people working on JSC want this, and people working on JSC defend these tests, and that these test results are more stable (see below), we should make this change for JSC.

I was trying to convince folks defending non-JSC testing, that they too, should want this. I'm not going to pull teeth here. If folks want their tests to take ~10x longer to run, they're entitled to make that tradeoff.
According to Alexey, this is ~2x longer. So please replace my use of 10x here with 2x.

- Saam



And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.
My experience running the tests locally supports this fully. I don't get timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd get timeouts all the time, and the total test run would take ~4-8 hours. We can wait for official confirmation from Mark.

- Saam


Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Geoffrey Garen
In reply to this post by Saam barati-2
On Jun 19, 2020, at 8:04 AM, Geoffrey Garen <[hidden email]> wrote:

Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?
There doesn't exist a way to do this now, but it'd be trivial to add a way. I won't claim it fixes all stack traces differences, but I'd think compiling using "-fno-inline -fno-optimize-sibling-calls" would get us pretty far in crashing stack traces being similar enough.

Sounds good.

I think we should try to refine the proposal along these lines, to minimize the downsides. I won’t speak for Simon, but for me, being able to ensure a clear backtrace from a bot would be a big improvement.

And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.
I believe Mark's proposal, initially, is just to make JSC do this. So I don't see the point of compiling WTF differently. JSC can kick off its own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given people working on JSC want this, and people working on JSC defend these tests, and that these test results are more stable (see below), we should make this change for JSC.

I was trying to convince folks defending non-JSC testing, that they too, should want this. I'm not going to pull teeth here. If folks want their tests to take ~10x longer to run, they're entitled to make that tradeoff.

Got it.

And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.
My experience running the tests locally supports this fully. I don't get timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd get timeouts all the time, and the total test run would take ~4-8 hours. We can wait for official confirmation from Mark.

Alexey, do the JSC stress tests run now on bots? If not, how fast would they need to run in order to be eligible to run on bots?

Thanks,
Geoff


- Saam


Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev




_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Mark Lam
In reply to this post by Alexey Proskuryakov-4


On Jun 19, 2020, at 10:02 AM, Alexey Proskuryakov <[hidden email]> wrote:



19 июня 2020 г., в 9:59 AM, Mark Lam <[hidden email]> написал(а):



On Jun 19, 2020, at 9:53 AM, Alexey Proskuryakov <[hidden email]> wrote:



18 июня 2020 г., в 9:30 PM, Saam Barati <[hidden email]> написал(а):

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Replying to this point specifically, I wanted to point out that WebKit tests take 2x longer in debug, not 10x longer. JSC tests take 3.6x longer.

Since I collected real data on this last night on the actual bot that runs the JSC stress tests, I’ll share the data here:

Build time
A clean build using buid-jsc for a normal Debug build on bot638 takes about 4.5 minutes.
A clean build using build-jsc for a --force-opt=O3 Debug build on bot638 takes about 6 minutes.

Test run time
Running with a regular Debug build, the test completed in about 4 hours 41 minutes with 1 timeout.
Running with a --force-opt=O3 Debug build, the test completed in about 39 minutes with 0 timeouts.

The difference in test run time is 281 minutes vs 29 minutes.  That is a 7.2x ratio, not 3.6x.


That's 3.6x.

Some details to consider:


2. You’re comparing a Release build run vs a Debug build run, not an O3 Debug build vs a regular Debug build.

3. Release builds do less work e.g. all debug ASSERTs are disabled.  Debug only validation code is disabled.

4. Not every file in a Release build is compiled with O3.  If I remember correctly, some are O3, others are not.  None are O0.

5. The collection of JSC tests run for Debug builds and Release builds are slightly different.  For example, Debug builds run 71793 jsc stress tests.  Release builds runs 69950 jsc stress tests.  Hence, the Debug builds run 1843 more tests.

My main point here is that Release builds are not the same as O3 Debug builds: some things are faster, some things are slower.  A Release build test run is at best an approximate representation of how a Debug O3 build test run will behave.  FWIW, my data came from actually running an O3 Debug build and a regular Debug build on the same bot.  

But my number are purely from timing the run of the tests.  If the numbers reported from the bot includes time for any bots scripts in addition to the test run times (I don’t know if there are any), then the total run time will also be different than the numbers I collected.  Feel free to use the 3.6x ratio if you prefer.

About -Og numbers (using the same measurement methology as my O3 Debug and regular Debug runs):

Build time: A clean build using build-jsc for a --force-opt=Og Debug build on bot638 takes about 5.5 minutes.
Test run time: Running with a --force-opt=Og Debug build, the test completed in about 2 hours 19 minute minutes with 0 timeouts.

That's about 3.5x slower than the Debug O3 run.

Mark



- Alexey



Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:

Last time this was discussed, we talked about -Og, which is specifically designed for the purpose I believe. Where do we stand on understanding and adopting that?

I tried -Og last time after your suggestion, and I remember thinking that the perf was not acceptable back then.  I’ll collect the data again and report back with real number later today.

Mark


- Alexey

- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

- Alexey


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev





_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Alexey Proskuryakov-4
In reply to this post by Geoffrey Garen


19 июня 2020 г., в 10:24 AM, Geoffrey Garen <[hidden email]> написал(а):

On Jun 19, 2020, at 8:04 AM, Geoffrey Garen <[hidden email]> wrote:

Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?
There doesn't exist a way to do this now, but it'd be trivial to add a way. I won't claim it fixes all stack traces differences, but I'd think compiling using "-fno-inline -fno-optimize-sibling-calls" would get us pretty far in crashing stack traces being similar enough.

Sounds good.

I think we should try to refine the proposal along these lines, to minimize the downsides. I won’t speak for Simon, but for me, being able to ensure a clear backtrace from a bot would be a big improvement.

Enabling some level of optimization is reasonable; whether it should be -O3 with inlining disabled or -Og is a technical question that probably can't be answered without hard data. Also, building locally in the same way as bots do could be a show stopper, as people don't like slow builds.

And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.
I believe Mark's proposal, initially, is just to make JSC do this. So I don't see the point of compiling WTF differently. JSC can kick off its own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given people working on JSC want this, and people working on JSC defend these tests, and that these test results are more stable (see below), we should make this change for JSC.

I was trying to convince folks defending non-JSC testing, that they too, should want this. I'm not going to pull teeth here. If folks want their tests to take ~10x longer to run, they're entitled to make that tradeoff.

Got it.

To do this only for JSC builds, we'd need separate builders and storage, so it becomes a question of allocating more resources, not just switching over to a different configuration. While EWS builds for JSC independently, post-commit testing shares build artifacts across all testers.

And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.
My experience running the tests locally supports this fully. I don't get timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd get timeouts all the time, and the total test run would take ~4-8 hours. We can wait for official confirmation from Mark.

Alexey, do the JSC stress tests run now on bots? If not, how fast would they need to run in order to be eligible to run on bots?

I don't think that there is a simple answer, as certain variations of stress tests get disabled on certain bots, JSC tests have a lot of variations that are handpicked. I wouldn't even know how to find the complex answer, but perhaps you can get the answer from <https://build.webkit.org/dashboard/>

- Alexey



Thanks,
Geoff


- Saam


Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Mark Lam
In reply to this post by Geoffrey Garen


On Jun 19, 2020, at 10:24 AM, Geoffrey Garen <[hidden email]> wrote:

On Jun 19, 2020, at 8:04 AM, Geoffrey Garen <[hidden email]> wrote:

Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?
There doesn't exist a way to do this now, but it'd be trivial to add a way. I won't claim it fixes all stack traces differences, but I'd think compiling using "-fno-inline -fno-optimize-sibling-calls" would get us pretty far in crashing stack traces being similar enough.

Sounds good.

I think we should try to refine the proposal along these lines, to minimize the downsides. I won’t speak for Simon, but for me, being able to ensure a clear backtrace from a bot would be a big improvement.

And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.
I believe Mark's proposal, initially, is just to make JSC do this. So I don't see the point of compiling WTF differently. JSC can kick off its own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given people working on JSC want this, and people working on JSC defend these tests, and that these test results are more stable (see below), we should make this change for JSC.

I was trying to convince folks defending non-JSC testing, that they too, should want this. I'm not going to pull teeth here. If folks want their tests to take ~10x longer to run, they're entitled to make that tradeoff.

Got it.

I'm of the same mind as Saam.  We want this change for the JSC bots, and from the time measurements I’ve collected, we can afford to do a clean build for the JSC Debug test runs using O3, and still come out way ahead.

As for non-JSC test runs, I have not actually measured what the time savings are.  Given there is resistance to going with O3 there, we don’t have to share the build artifacts for running the tests.  JSC test runs should be able to just build JSC for each O3 Debug JSC test run and it is still a win over the current status quo i.e. test runs never complete.

Regarding Geoff’s earlier question about whether I know for sure that switching to O3 will fix the current Debug JSC bot failures to run tests, the answer is I’m not certain.  The failure is a timeout due to the master bot not seeing any output from the tester bot for more than 20 minutes.  I’ve not been able to reproduce this yet.  But with a Debug build test run taking 4+ hours, it can only be a progression to switch the Debug JSC test bots to O3.

Mark



And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.
My experience running the tests locally supports this fully. I don't get timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd get timeouts all the time, and the total test run would take ~4-8 hours. We can wait for official confirmation from Mark.

Alexey, do the JSC stress tests run now on bots? If not, how fast would they need to run in order to be eligible to run on bots?

Thanks,
Geoff


- Saam


Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching open source Debug bots to building and testing with configuration --force-opt=O3

Saam barati-2
In reply to this post by Alexey Proskuryakov-4


On Jun 19, 2020, at 1:04 PM, Alexey Proskuryakov <[hidden email]> wrote:



19 июня 2020 г., в 10:24 AM, Geoffrey Garen <[hidden email]> написал(а):

On Jun 19, 2020, at 8:04 AM, Geoffrey Garen <[hidden email]> wrote:

Can you explain more about what "O3 with no-inlining” is? How does --force-opt=O3 avoid inlining? Would this fully resolve Simon concern about stack traces, or would something still be different about stack traces?
There doesn't exist a way to do this now, but it'd be trivial to add a way. I won't claim it fixes all stack traces differences, but I'd think compiling using "-fno-inline -fno-optimize-sibling-calls" would get us pretty far in crashing stack traces being similar enough.

Sounds good.

I think we should try to refine the proposal along these lines, to minimize the downsides. I won’t speak for Simon, but for me, being able to ensure a clear backtrace from a bot would be a big improvement.

Enabling some level of optimization is reasonable; whether it should be -O3 with inlining disabled or -Og is a technical question that probably can't be answered without hard data. Also, building locally in the same way as bots do could be a show stopper, as people don't like slow builds.
Just to be super clear, I'm not proposing turning off inlining for JSC debug testers. We on JSC are happy with stack traces that aren't perfect.

My no inlining is a proposal for folks if we wanted this more broadly in our testing. I don't know what the results of inlining are on the time Debug+O3 takes to. run. If it's negligible, we'd also be happy to run this way for JSC Debug. But if JSC is the only thing making a change, we're going to run with inlining on.


And again, I think this discussion would get a lot more focused if the change could apply only to JSC code, and not also to WTF code.
I believe Mark's proposal, initially, is just to make JSC do this. So I don't see the point of compiling WTF differently. JSC can kick off its own build, and run Debug+O3 tests faster than it can run Debug+O0 tests. Given people working on JSC want this, and people working on JSC defend these tests, and that these test results are more stable (see below), we should make this change for JSC.

I was trying to convince folks defending non-JSC testing, that they too, should want this. I'm not going to pull teeth here. If folks want their tests to take ~10x longer to run, they're entitled to make that tradeoff.

Got it.

To do this only for JSC builds, we'd need separate builders and storage, so it becomes a question of allocating more resources, not just switching over to a different configuration. While EWS builds for JSC independently, post-commit testing shares build artifacts across all testers.
As Mark has shown, we can build locally and run faster than using a plain debug build. So as a straw man, this should just work and be faster than what we have today.

- Saam



And again, on the run more tests front, it would be helpful to know whether this change was enough to get the stress tests running or not.
My experience running the tests locally supports this fully. I don't get timeouts when running O3+Debug locally. When running Debug+O0 locally, I'd get timeouts all the time, and the total test run would take ~4-8 hours. We can wait for official confirmation from Mark.

Alexey, do the JSC stress tests run now on bots? If not, how fast would they need to run in order to be eligible to run on bots?

I don't think that there is a simple answer, as certain variations of stress tests get disabled on certain bots, JSC tests have a lot of variations that are handpicked. I wouldn't even know how to find the complex answer, but perhaps you can get the answer from <https://build.webkit.org/dashboard/>

- Alexey



Thanks,
Geoff


- Saam


Thanks,
Geoff

On Jun 18, 2020, at 9:30 PM, Saam Barati <[hidden email]> wrote:

Why are we insisting on doing something on the bots that takes ~10x longer to run than necessary? I’d rather have that time spent running more tests.

Overall, how we’re doing things now feels like a bad allocation of bot resources. The differences I see between O3 with no-inlining vs O0 is:
- Some race conditions will behave differently. Race conditions are already non predictable. I don’t think we’re losing anything here.
- O0 vs O3 is a different compiler. We may encounter bugs in O3 we don’t in O0, and vice versa. In general, we probably care more about O3 compiler bugs than O0, since we don’t ship O0, but ship a lot of O3.

(And if we’re going to insist on “I want it to run what I build at my desk”: I run debug with O3 at my desk, and I can run debug tests in a reasonable amount of time now.)

In evaluating what’s the better setup, I think it’s helpful to think about this from the other side. Let’s imagine we had Debug+O3 as our current setup. And someone proposed to move it to O0 and make our tests take ~10x longer. I think that’d be a non-starter.

- Saam

On Jun 17, 2020, at 9:48 PM, Simon Fraser <[hidden email]> wrote:

I also object to losing good stack traces for crashes on Debug bots.

Also, I don't think Debug bots should build something different from what I build at my desk.

Simon

On Jun 17, 2020, at 1:36 PM, Mark Lam <[hidden email]> wrote:

Hi folks,

We're planning to switch the JSC EWS bot and build.webkit.org Debug build and test bots to building with the following set first:
./Tools/Scripts/set-webkit-configuration --force-opt=O3

This means the Debug builds will be built with optimization level forced to O3.

Why are we doing this?
1. So that the JSC EWS will start catching ASSERT failures.
2. JSC stress test Debug bots have been timing out and not running tests at all.  Hopefully, this change will fix this issue.
3. Tests will run to completion faster and we’ll catch regressions sooner.

The downside: crash stack traces will be like Release build stack traces.  But I don’t think we should let this deter us.  It’s not like there’s no stack information.  And just as we do with debugging Release build test failures, we can always do a Debug build locally to do our debugging.

We would like to apply this change to all Debug build and test bots, not just the JSC ones.  Does anyone strongly object to this change?

Thanks.

Cheers,
Mark

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
12