Another WPT bite

classic Classic list List threaded Threaded
71 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Another WPT bite

youenn fablet
Hi all,

Discussing with some WebKittens, testharness.js is more and more used in WebKit.
Is it time to make testharness.js the recommended way of writing LayoutTests?

To continue moving forward, some of us are proposing to serve all tests in LayoutTests/wpt through the WPT server [1].
This would serve some purposes like increasing the use of WPT goodies: file-specific headers, templated tests (*.any.js), IDLParser, server-side scripts...
It could also ease test migration from WebKit to W3C WPT.

Some rules can guide whether adding tests to LayoutTests/wpt or LayoutTests/imported/w3c/web-platform-tests:
- WebKit specific tests (crash tests, tests using internals...) in LayoutTests/wpt
- Spec conformance/interoperability tests in LayoutTests/imported/w3c/web-platform-tests

   y




_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Geoffrey Garen
Is it time to make testharness.js the recommended way of writing LayoutTests?

What are the costs and benefits of testharness.js?

We usually try to make regression tests reductions of some larger problem to aid debugging and to make testing fast. But testharness.js is 95kB. That's kind of the opposite of a reduction.

Geoff


To continue moving forward, some of us are proposing to serve all tests in LayoutTests/wpt through the WPT server [1].
This would serve some purposes like increasing the use of WPT goodies: file-specific headers, templated tests (*.any.js), IDLParser, server-side scripts...
It could also ease test migration from WebKit to W3C WPT.

Some rules can guide whether adding tests to LayoutTests/wpt or LayoutTests/imported/w3c/web-platform-tests:
- WebKit specific tests (crash tests, tests using internals...) in LayoutTests/wpt
- Spec conformance/interoperability tests in LayoutTests/imported/w3c/web-platform-tests

   y



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Chris Dumez


On May 8, 2017, at 9:44 PM, Geoffrey Garen <[hidden email]> wrote:

Is it time to make testharness.js the recommended way of writing LayoutTests?

What are the costs and benefits of testharness.js?

Benefit: 
- Tests would be more easily upstreamable to web-platform-tests, which are run by all major browser engines. This would help a lot in terms of interoperability. As previously discussed, Gecko and Blink already do automated export of tests to web-platform-tests. I believe we should do in the same direction and contribute more tests back.
- It is quite powerful (see documentation [1], has things like Promise tests, Worker tests and advanced assertions.

Cost:
- People are not necessarily used to it so there is a little bit of a learning curve. It is well documented though [1].


We usually try to make regression tests reductions of some larger problem to aid debugging and to make testing fast. But testharness.js is 95kB. That's kind of the opposite of a reduction.

It is proposed as a replacement to js-test.js, which a lot of us are already using in our layout tests. Using js-test.js or testharness.js has rarely interfered in my reductions, although it has happened to me.
For some tests, we’ll probably not use any framework. However, for most of them, I personally don’t see an issue.




Geoff


To continue moving forward, some of us are proposing to serve all tests in LayoutTests/wpt through the WPT server [1].
This would serve some purposes like increasing the use of WPT goodies: file-specific headers, templated tests (*.any.js), IDLParser, server-side scripts...
It could also ease test migration from WebKit to W3C WPT.

Some rules can guide whether adding tests to LayoutTests/wpt or LayoutTests/imported/w3c/web-platform-tests:
- WebKit specific tests (crash tests, tests using internals...) in LayoutTests/wpt
- Spec conformance/interoperability tests in LayoutTests/imported/w3c/web-platform-tests

   y



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Brady Eidson
In reply to this post by youenn fablet
On May 8, 2017, at 9:31 PM, youenn fablet <[hidden email]> wrote:

Hi all,

Discussing with some WebKittens, testharness.js is more and more used in WebKit.
Is it time to make testharness.js the recommended way of writing LayoutTests?

Setting aside the pros or cons of testharness.js itself, I disagree with the principle of "1 single way to write all regression tests"

In the past 11 years I've heard from multiple members of the team commenting on the benefits of different people writing regression tests in their own styles using their own techniques. We're capable of covering more edge cases when we don't have enforced uniformity. And I agree wholeheartedly.

But now talking about testharness.js directly, I object on the grounds of "a file:// regression test is dirt easy to hack on and work with, whereas anything that requires me to have an httpd running is a PITA"

Note: I don't intend for any of this to mean I discourage the use of testharness.js tests. I don't. By all means, write them.

I just object to making it the "recommended way" of writing tests.

Thanks,
 Brady


To continue moving forward, some of us are proposing to serve all tests in LayoutTests/wpt through the WPT server [1].
This would serve some purposes like increasing the use of WPT goodies: file-specific headers, templated tests (*.any.js), IDLParser, server-side scripts...
It could also ease test migration from WebKit to W3C WPT.

Some rules can guide whether adding tests to LayoutTests/wpt or LayoutTests/imported/w3c/web-platform-tests:
- WebKit specific tests (crash tests, tests using internals...) in LayoutTests/wpt
- Spec conformance/interoperability tests in LayoutTests/imported/w3c/web-platform-tests

   y



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

youenn fablet
testharness.js does not need an http server. Some WPT goodies need the WPT server.

I agree different frameworks offer different benefits. There is no reason we should mandate one framework in particular.

In case there is no specific needs, it makes sense to me to recommend using testharness.js, at least for those hacking WebCore. Chances are high that another browser community might like running (and improving and maintaining) it.
Chances are also high that one will have to debug/update such tests, so it is good to learn this framework anyway.
Le lun. 8 mai 2017 à 22:17, Brady Eidson <[hidden email]> a écrit :
On May 8, 2017, at 9:31 PM, youenn fablet <[hidden email]> wrote:

Hi all,

Discussing with some WebKittens, testharness.js is more and more used in WebKit.
Is it time to make testharness.js the recommended way of writing LayoutTests?

Setting aside the pros or cons of testharness.js itself, I disagree with the principle of "1 single way to write all regression tests"

In the past 11 years I've heard from multiple members of the team commenting on the benefits of different people writing regression tests in their own styles using their own techniques. We're capable of covering more edge cases when we don't have enforced uniformity. And I agree wholeheartedly.

But now talking about testharness.js directly, I object on the grounds of "a file:// regression test is dirt easy to hack on and work with, whereas anything that requires me to have an httpd running is a PITA"

Note: I don't intend for any of this to mean I discourage the use of testharness.js tests. I don't. By all means, write them.

I just object to making it the "recommended way" of writing tests.

Thanks,
 Brady


To continue moving forward, some of us are proposing to serve all tests in LayoutTests/wpt through the WPT server [1].
This would serve some purposes like increasing the use of WPT goodies: file-specific headers, templated tests (*.any.js), IDLParser, server-side scripts...
It could also ease test migration from WebKit to W3C WPT.

Some rules can guide whether adding tests to LayoutTests/wpt or LayoutTests/imported/w3c/web-platform-tests:
- WebKit specific tests (crash tests, tests using internals...) in LayoutTests/wpt
- Spec conformance/interoperability tests in LayoutTests/imported/w3c/web-platform-tests

   y



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Ryosuke Niwa-2
In reply to this post by Brady Eidson
On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:

> On May 8, 2017, at 9:31 PM, youenn fablet <[hidden email]> wrote:
>
> Hi all,
>
> Discussing with some WebKittens, testharness.js is more and more used in
> WebKit.
> Is it time to make testharness.js the recommended way of writing
> LayoutTests?
>
>
> Setting aside the pros or cons of testharness.js itself, I disagree with the
> principle of "1 single way to write all regression tests"
>
> In the past 11 years I've heard from multiple members of the team commenting
> on the benefits of different people writing regression tests in their own
> styles using their own techniques. We're capable of covering more edge cases
> when we don't have enforced uniformity. And I agree wholeheartedly.

Agreed.

> But now talking about testharness.js directly, I object on the grounds of "a
> file:// regression test is dirt easy to hack on and work with, whereas
> anything that requires me to have an httpd running is a PITA"

I think whether we use file:// or http:// is orthogonal point to using
testharness.js.  Many of the tests Chris and I have written using
testharness.js are checked into regular LayoutTests/ directories, and
they work just fine.

The imported web platform tests use WPT today because that's how
upstream works. But we're looking at ways to reduce the number of
tests that need to be ran under WPT for imported tests as well.

> I just object to making it the "recommended way" of writing tests.

Would you equally object to making js-test.js / js-test-pre.js the
recommended way of writing tests? If not, why?

What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.

- R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Brady Eidson
In reply to this post by youenn fablet

On May 8, 2017, at 10:42 PM, youenn fablet <[hidden email]> wrote:

testharness.js does not need an http server. Some WPT goodies need the WPT server.

I misunderstood since we were also discussing:

To continue moving forward, some of us are proposing to serve all tests in LayoutTests/wpt through the WPT server [1].

I think if somebody is writing a testharness.js test and it does *not* require the WPT server, it should be tested as a file URL.

This would imply putting it outside of LayoutTests/wpt though, right?

I agree different frameworks offer different benefits. There is no reason we should mandate one framework in particular.

Let me make it even more clear what I'm trying to defend. An ad-hoc custom test that uses no "framework" but is entirely self contained.

In case there is no specific needs, it makes sense to me to recommend using testharness.js, at least for those hacking WebCore.

I think it's valuable for engineers to be aware of the "frameworks" like testharness.js and js-test, and familiar with the facilities they provide, but to not necessarily imply that using any framework is expected.

Thanks,
 Brady

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Brady Eidson
In reply to this post by Ryosuke Niwa-2

> On May 8, 2017, at 10:44 PM, Ryosuke Niwa <[hidden email]> wrote:
>
> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:
>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>
> I think whether we use file:// or http:// is orthogonal point to using
> testharness.js.  Many of the tests Chris and I have written using
> testharness.js are checked into regular LayoutTests/ directories, and
> they work just fine.

Yes, I misunderstood this in Youenn's original message. Good to know!
>
>> I just object to making it the "recommended way" of writing tests.
>
> Would you equally object to making js-test.js / js-test-pre.js the
> recommended way of writing tests?

Yes.

> If not, why?

N/A

> What we're suggesting is to give preferential treatments to
> testharness.js over js-test.js / js-test-pre.js when you were already
> planning to write a test with the latter two scripts.

"It's okay to write your test however you'd like. If you were considering using js-test, maybe you should consider using testharness instead."

Is that's what's being proposed?

The WebKit project usually doesn't spend much time worrying about highly qualified non-binding suggestions. ;)

Thanks,
 Brady
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Ryosuke Niwa-2
On Mon, May 8, 2017 at 11:01 PM, Brady Eidson <[hidden email]> wrote:

>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa <[hidden email]> wrote:
>>
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:
>>
>>> But now talking about testharness.js directly, I object on the grounds of "a
>>> file:// regression test is dirt easy to hack on and work with, whereas
>>> anything that requires me to have an httpd running is a PITA"
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>
> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>>> I just object to making it the "recommended way" of writing tests.
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>
> Yes.
>
>> If not, why?
>
> N/A
>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>
> "It's okay to write your test however you'd like. If you were considering using js-test, maybe you should consider using testharness instead."
>
> Is that's what's being proposed?

The thing I specifically asked Youenn to ask is, whether we should
place a test inside LayoutTests/wpt like LayoutTests/http/tests when
we want to write a test using testharness.js which requires some sort
of network code.

Since people have had some opinions about directory structures in the past.

- R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Maciej Stachowiak


On May 8, 2017, at 11:15 PM, Ryosuke Niwa <[hidden email]> wrote:

On Mon, May 8, 2017 at 11:01 PM, Brady Eidson <[hidden email]> wrote:
<a class="apple-mail-expandLink" href="x-redundant-cluster-toggle://0" style="text-decoration: none; color: inherit;">
<a class="apple-mail-expandLink" href="x-redundant-cluster-toggle://0" style="text-decoration: none; color: inherit;">
<a class="apple-mail-expandLink" href="x-redundant-cluster-toggle://0" style="text-decoration: none; color: inherit;">
<a class="apple-mail-expandLink" href="x-redundant-cluster-toggle://0" style="text-decoration: none; color: inherit;">On May 8, 2017, at 10:44 PM, Ryosuke Niwa <[hidden email]> wrote:
On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:
But now talking about testharness.js directly, I object on the grounds of "a
file:// regression test is dirt easy to hack on and work with, whereas
anything that requires me to have an httpd running is a PITA"
I think whether we use file:// or http:// is orthogonal point to using
testharness.js.  Many of the tests Chris and I have written using
testharness.js are checked into regular LayoutTests/ directories, and
they work just fine.
Yes, I misunderstood this in Youenn's original message. Good to know!
I just object to making it the "recommended way" of writing tests.
Would you equally object to making js-test.js / js-test-pre.js the
recommended way of writing tests?
Yes.
If not, why?
N/A
What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.
"It's okay to write your test however you'd like. If you were considering using js-test, maybe you should consider using testharness instead."
Is that's what's being proposed?

On May 8, 2017, at 10:44 PM, Ryosuke Niwa <[hidden email]> wrote:

On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:

But now talking about testharness.js directly, I object on the grounds of "a
file:// regression test is dirt easy to hack on and work with, whereas
anything that requires me to have an httpd running is a PITA"

I think whether we use file:// or http:// is orthogonal point to using
testharness.js.  Many of the tests Chris and I have written using
testharness.js are checked into regular LayoutTests/ directories, and
they work just fine.

Yes, I misunderstood this in Youenn's original message. Good to know!

I just object to making it the "recommended way" of writing tests.

Would you equally object to making js-test.js / js-test-pre.js the
recommended way of writing tests?

Yes.

If not, why?

N/A

What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.

"It's okay to write your test however you'd like. If you were considering using js-test, maybe you should consider using testharness instead."

Is that's what's being proposed?

Besides other issues mentioned, testharness tends to result in more verbose tests compared to js-test, at least for simple cases.


The thing I specifically asked Youenn to ask is, whether we should
place a test inside LayoutTests/wpt like LayoutTests/http/tests when
we want to write a test using testharness.js which requires some sort
of network code.

Since people have had some opinions about directory structures in the past.

It seems like we need a few different directories, here are my opinions on them:

(1) Imported web platform tests that don't need a server
    Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
(2) Imported web platform tests that do need a server
    Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe under http/tests/ per point (4)
(3) Custom testharness.js tests that don't need a server 
    Probably these should just go in their normal topic-specific directories and should not need a special directory
(4) Custom testharness.js tests that do need a server
    Can these just be a subdirectory of http/tests/? We have websocket and ssl/tls tests in there too. Would be nice to not need a separate directory for networking tests that to use a particular test framework.

Regards,
Maciej


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Geoffrey Garen
In reply to this post by Ryosuke Niwa-2
What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.

OK, I think this makes sense.

But I still think the very best kind of test is a flat file with 10-20 lines of code in it. Particularly for debugging JavaScript issues, large wrapper frameworks get in the way.

- Tests would be more easily upstreamable to web-platform-tests, which are run by all major browser engines. This would help a lot in terms of interoperability. As previously discussed, Gecko and Blink already do automated export of tests to web-platform-tests. I believe we should do in the same direction and contribute more tests back.

I wonder why these other projects do automated export instead of incorporating testharness.js directly.

Geoff

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

youenn fablet
In reply to this post by Maciej Stachowiak

Besides other issues mentioned, testharness tends to result in more verbose tests compared to js-test, at least for simple cases.

For synchronous tests, I am not sure there is any big difference one way or the other.
With asynchronous tests, it might be true, but using testharness.js/promise_test usually improves things there.
 I personally find it easier to not wrap code-to-be-tested into quotes.

Another concern is the lack of verbose output which reduces the ability to debug failing tests.
This can be partially fixed by authoring tests with that issue in mind.
For instance, having a big promise_test to handle the asynchronous aspect of a test and nested test() inside it.

The thing I specifically asked Youenn to ask is, whether we should
place a test inside LayoutTests/wpt like LayoutTests/http/tests when
we want to write a test using testharness.js which requires some sort
of network code.

Since people have had some opinions about directory structures in the past.

It seems like we need a few different directories, here are my opinions on them:

(1) Imported web platform tests that don't need a server
    Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.

All WPT tests are expected to run behind the WPT server.
That is the way tests are authored and tested elsewhere.
If we have an issue with that, it is best to bring that and fix that directly in WPT.
I encountered several times small issues due to file:// based origins which makes me think defaulting to http is a safe choice.

One concern is efficiency. We should study that and improve on that.
Another concern is the ease of running tests for developers: drag&dropping tests into a browser instead of running a server.
We can partially accommodate this by rewriting testharness.js/testharnessreport.js urls.
A significant and growing amount of wpt tests will not behave as expected (other resources loaded, origins, need for specific headers, need for https...)

(2) Imported web platform tests that do need a server
    Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe under http/tests/ per point (4)

I don't think this will work, web-platform-tests is organized in terms of features.
There is no clear separation between file based compatible tests and http based tests like in WebKit.

(3) Custom testharness.js tests that don't need a server 
    Probably these should just go in their normal topic-specific directories and should not need a special directory

Right.
The only case where it might make sense to put such tests in a specific WPT-enabled directory is if the plan is to upstream these tests at some point.
Such tests could be added in imported/w3c/web-platform-tests directly but this requires coordination with resyncing tests at the moment.
In a not-too-far-future, I hope such tests would directly be authored in imported/w3c/web-platform-tests.
 
(4) Custom testharness.js tests that do need a server
    Can these just be a subdirectory of http/tests/? We have websocket and ssl/tls tests in there too. Would be nice to not need a separate directory for networking tests that to use a particular test framework.

I do not have strong feelings there, http/wpt might make sense if it is found easier to understand to everybody.
I'll update the patch accordingly and will land it sometimes this week if there is no additional feedback.

Thanks!
   y

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Geoffrey Garen
> Another concern is the ease of running tests for developers: drag&dropping tests into a browser instead of running a server.

Yeah, it’s a pretty big concern if you can’t just drop a simple test case into a browser.

> We can partially accommodate this by rewriting testharness.js/testharnessreport.js urls.
> A significant and growing amount of wpt tests will not behave as expected (other resources loaded, origins, need for specific headers, need for https…)

This might be a reason to prefer WPT tests when you know you’re testing something whose behavior depends on http. But in all other cases, the ability for a test to be a self-contained file that can run in any browser, upload to a bug tracker, and so on, is super valuable.

Geoff
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Maciej Stachowiak
In reply to this post by youenn fablet


On May 9, 2017, at 8:44 AM, youenn fablet <[hidden email]> wrote:


Besides other issues mentioned, testharness tends to result in more verbose tests compared to js-test, at least for simple cases.

For synchronous tests, I am not sure there is any big difference one way or the other.
With asynchronous tests, it might be true, but using testharness.js/promise_test usually improves things there.
 I personally find it easier to not wrap code-to-be-tested into quotes.

It seems like it's a matter of personal preference to some extent. So I am not sure we should recommend one or the other (besides for tests that are intended to be contributed to web-platform-tests).


Another concern is the lack of verbose output which reduces the ability to debug failing tests.
This can be partially fixed by authoring tests with that issue in mind.
For instance, having a big promise_test to handle the asynchronous aspect of a test and nested test() inside it.

I don't think we've ever had a problem with debugging js-test.js tests when they fail.


The thing I specifically asked Youenn to ask is, whether we should
place a test inside LayoutTests/wpt like LayoutTests/http/tests when
we want to write a test using testharness.js which requires some sort
of network code.

Since people have had some opinions about directory structures in the past.

It seems like we need a few different directories, here are my opinions on them:

(1) Imported web platform tests that don't need a server
    Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.

All WPT tests are expected to run behind the WPT server.
That is the way tests are authored and tested elsewhere.
If we have an issue with that, it is best to bring that and fix that directly in WPT.
I encountered several times small issues due to file:// based origins which makes me think defaulting to http is a safe choice.

One concern is efficiency. We should study that and improve on that.
Another concern is the ease of running tests for developers: drag&dropping tests into a browser instead of running a server.
We can partially accommodate this by rewriting testharness.js/testharnessreport.js urls.
A significant and growing amount of wpt tests will not behave as expected (other resources loaded, origins, need for specific headers, need for https...)

(2) Imported web platform tests that do need a server
    Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe under http/tests/ per point (4)

I don't think this will work, web-platform-tests is organized in terms of features.
There is no clear separation between file based compatible tests and http based tests like in WebKit.

If we run all the w3c-imported web platform tests under a web server, then obviously we only need one directory. My understanding is that we don't run them under a server at all. So it seems like one part of this proposal is "run everything under LayoutTests/imported/w3c/ from a web server".


(3) Custom testharness.js tests that don't need a server 
    Probably these should just go in their normal topic-specific directories and should not need a special directory

Right.
The only case where it might make sense to put such tests in a specific WPT-enabled directory is if the plan is to upstream these tests at some point.
Such tests could be added in imported/w3c/web-platform-tests directly but this requires coordination with resyncing tests at the moment.
In a not-too-far-future, I hope such tests would directly be authored in imported/w3c/web-platform-tests.

I think it would be cleaner to have a separate directory of tests intended for import, separate from imported tests. It could be right next to imported/w3c/web-platform-tests. I think mixing tests that are imported from upstream and tests intended for eventual upstreaming is more confusing than helpful.

 
(4) Custom testharness.js tests that do need a server
    Can these just be a subdirectory of http/tests/? We have websocket and ssl/tls tests in there too. Would be nice to not need a separate directory for networking tests that to use a particular test framework.

I do not have strong feelings there, http/wpt might make sense if it is found easier to understand to everybody.
I'll update the patch accordingly and will land it sometimes this week if there is no additional feedback.

One question I have is whether web platform tests can run under a regular HTTP server (maybe with appropriate configuration) or do we need something special? Is the WPT server more than just a web server with specific configuration settings?

Regards,
Maciej



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Maciej Stachowiak
In reply to this post by Geoffrey Garen


On May 9, 2017, at 8:11 AM, Geoffrey Garen <[hidden email]> wrote:

What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.

OK, I think this makes sense.

But I still think the very best kind of test is a flat file with 10-20 lines of code in it. Particularly for debugging JavaScript issues, large wrapper frameworks get in the way.

- Tests would be more easily upstreamable to web-platform-tests, which are run by all major browser engines. This would help a lot in terms of interoperability. As previously discussed, Gecko and Blink already do automated export of tests to web-platform-tests. I believe we should do in the same direction and contribute more tests back.

I wonder why these other projects do automated export instead of incorporating testharness.js directly.

I don't think that's an "also", not an "instead". My understanding is that they do two-way sync with the web-platform-tests GitHub, so there's a process for downloading tests and upstreaming tests authored by their team. But they still have their own copy.

Regards,
Maciej


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Simon Fraser-6
On May 9, 2017, at 11:10 AM, Maciej Stachowiak <[hidden email]> wrote:

On May 9, 2017, at 8:11 AM, Geoffrey Garen <[hidden email]> wrote:

What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.

OK, I think this makes sense.

But I still think the very best kind of test is a flat file with 10-20 lines of code in it. Particularly for debugging JavaScript issues, large wrapper frameworks get in the way.

- Tests would be more easily upstreamable to web-platform-tests, which are run by all major browser engines. This would help a lot in terms of interoperability. As previously discussed, Gecko and Blink already do automated export of tests to web-platform-tests. I believe we should do in the same direction and contribute more tests back.

I wonder why these other projects do automated export instead of incorporating testharness.js directly.

I don't think that's an "also", not an "instead". My understanding is that they do two-way sync with the web-platform-tests GitHub, so there's a process for downloading tests and upstreaming tests authored by their team. But they still have their own copy.

Another consideration here is "would my test be useful for other browser vendors". I don't think the answer is a unanimous "yes", so I think we should only use WPT for tests that will think are worth sharing.

I'm also concerned that with 4 vendors upstreaming their WPT tests, the WPT repo will just become a morass of partially overlapping tests that takes 4x longer to run than a curated repo.

Simon



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Ryosuke Niwa-2
In reply to this post by Maciej Stachowiak
On Tue, May 9, 2017 at 11:06 AM, Maciej Stachowiak <[hidden email]>
>
> If we run all the w3c-imported web platform tests under a web server, then
> obviously we only need one directory. My understanding is that we don't run
> them under a server at all. So it seems like one part of this proposal is
> "run everything under LayoutTests/imported/w3c/ from a web server".

We currently run all web-platform-tests using WPT server. In theory,
there's nothing preventing us from running these tests using file://
so long as tests don't rely on the networking aspect. However, we
don't currently rewrite URLs to testharness.js and
testharness-report.js both of which refers to the top-level directory
at the moment.

IMO, the entirety of the effort to make these imported tests runnable
in file scheme should be about re-enabling these URL rewrites that got
broken sometime ago.

>
> I think it would be cleaner to have a separate directory of tests intended
> for import, separate from imported tests.

You mean intended for exports?
> It could be right next to imported/w3c/web-platform-tests. I think mixing tests that are imported from
> upstream and tests intended for eventual upstreaming is more confusing than
> helpful.

Okay. I've actually advocated for this approach for the same reason.

One annoying thing in this approach is that if we wanted to add a test
to, say, html/infrastructure/conformance-requirements/extensibility/
then we either have to mirror the same directory structure in this
upstream directory, or add some sort of WebKit-only annotation that
this test needs to be added there for the upstream process to be
automated.

>
> One question I have is whether web platform tests can run under a regular
> HTTP server (maybe with appropriate configuration) or do we need something
> special? Is the WPT server more than just a web server with specific
> configuration settings?

No. Most of web-platform-tests that do require HTTP access have an
accompanying Python file which use WPT server's functionality as far
as I know.

In fact, I'd even argue that we should try to use the WPT server for
writing tests for Web APIs and security checks since those tests would
be extremely valuable for other browser vendors as well.

- R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Ryosuke Niwa-2
In reply to this post by Geoffrey Garen
Forgot to CC webkit-dev.

- R. Niwa


On Tue, May 9, 2017 at 12:43 PM, Ryosuke Niwa <[hidden email]> wrote:

> On Tue, May 9, 2017 at 8:11 AM, Geoffrey Garen <[hidden email]> wrote:
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>>
>> OK, I think this makes sense.
>>
>> But I still think the very best kind of test is a flat file with 10-20 lines
>> of code in it. Particularly for debugging JavaScript issues, large wrapper
>> frameworks get in the way.
>
> Sure. We certainly don't want to be using testharness.js for JSC
> issues for example. Because they can be contributed as a pure JS test
> to test262.
>
> However, if we're testing WebIDL semantics, for example, then there is
> a tremendous value in us writing a test case that can be shared with
> other browser vendors in web-platform-tests. And for those, the ship
> has sailed to use testharness.js because convincing all other browser
> vendors to use js-test.js or any other JS testing framework would be
> hard at this point, and they want some automated way to knowing
> whether a test had passed or not, a simple human readable output of
> PASS or FAIL in the document would not work for their needs.
>
>> - Tests would be more easily upstreamable to web-platform-tests, which are
>> run by all major browser engines. This would help a lot in terms of
>> interoperability. As previously discussed, Gecko and Blink already do
>> automated export of tests to web-platform-tests. I believe we should do in
>> the same direction and contribute more tests back.
>>
>> I wonder why these other projects do automated export instead of
>> incorporating testharness.js directly.
>
> Both Gecko and Blink use testharness.js in their own tests, and export
> them back to web-platform-tests. The automated part is the exportation
> of files from their own repositories to web-platform-tests.
>
> - R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Ryosuke Niwa-2
In reply to this post by Maciej Stachowiak
Forgot to CC webkit-dev.
- R. Niwa


On Tue, May 9, 2017 at 12:36 PM, Ryosuke Niwa <[hidden email]> wrote:

> On Tue, May 9, 2017 at 1:12 AM, Maciej Stachowiak <[hidden email]> wrote:
>>
>>
>> On May 8, 2017, at 11:15 PM, Ryosuke Niwa <[hidden email]> wrote:
>>
>> On Mon, May 8, 2017 at 11:01 PM, Brady Eidson <[hidden email]> wrote:
>>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa <[hidden email]> wrote:
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:
>>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>>
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>> I just object to making it the "recommended way" of writing tests.
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>>
>> Yes.
>>
>> If not, why?
>>
>> N/A
>>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>> "It's okay to write your test however you'd like. If you were considering
>> using js-test, maybe you should consider using testharness instead."
>> Is that's what's being proposed?
>>
>>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa <[hidden email]> wrote:
>>
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <[hidden email]> wrote:
>>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>>
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>>
>>
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>>
>> I just object to making it the "recommended way" of writing tests.
>>
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>>
>>
>> Yes.
>>
>> If not, why?
>>
>>
>> N/A
>>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>>
>> "It's okay to write your test however you'd like. If you were considering
>> using js-test, maybe you should consider using testharness instead."
>>
>> Is that's what's being proposed?
>>
>>
>> Besides other issues mentioned, testharness tends to result in more verbose
>> tests compared to js-test, at least for simple cases.
>>
>>
>> The thing I specifically asked Youenn to ask is, whether we should
>> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
>> we want to write a test using testharness.js which requires some sort
>> of network code.
>>
>> Since people have had some opinions about directory structures in the past.
>>
>>
>> It seems like we need a few different directories, here are my opinions on
>> them:
>>
>> (1) Imported web platform tests that don't need a server
>>     Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
>> (2) Imported web platform tests that do need a server
>>     Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe
>
> I don't think it's a good idea to split web-platform-tests into two
> separate directories like that because I'd imagine there are resource
> dependencies between tests.  Note that this whole process of detecting
> & separating tests that can be ran locally is the work we'd have to
> do, and upstream won't maintain it.  Given that, we should be making
> the minimal amount of differences to the upstream directory structure
> as well.
>
> Granted, this would make it harder to know which tests require a
> server and which one doesn't. Perhaps this is bad enough that we need
> two directories after all but perhaps there is some other way to solve
> this problem like adding suffix to test names. (Although some ref
> tests in web-platform-tests use other tests as expectation but ref
> tests don't tend to require servers anyway; I know this is a terrible
> idea but we don't really have a control over what upstream does).
>
>> under http/tests/ per point (4)
>> (3) Custom testharness.js tests that don't need a server
>>     Probably these should just go in their normal topic-specific directories
>> and should not need a special directory
>> (4) Custom testharness.js tests that do need a server
>>     Can these just be a subdirectory of http/tests/? We have websocket and
>> ssl/tls tests in there too. Would be nice to not need a separate directory
>> for networking tests that to use a particular test framework.
>
> I guess we could put them under http/tests/wpt/ then?  One weird thing
> is that by default, http/tests/wpt/ would map to the top-level
> directory during testing: i.e. /http/tests/wpt/ would be localhost:X/
> where X is WPT server's port.
>
> - R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|

Re: Another WPT bite

Mike Pennisi
In reply to this post by youenn fablet
> One question I have is whether web platform tests can run under a regular
> HTTP server (maybe with appropriate configuration) or do we need something
> special? Is the WPT server more than just a web server with specific
> configuration settings?

While many tests can probably be run in this way, many others are dependent
on the functionality of the Web Platform Test Server. It's written in Python
and supports (among other things) arbitrary HTTP request handling via
standalone Python scripts. Its capabilities are documented here:

https://wptserve.readthedocs.io/en/latest/

Like with `testharness.js`, this additional infrastructure is a case of
"simplicity versus power."

> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.

This is a good point. I'm currently migrating service worker tests from
Chromium to WPT, and I'm frequently coming across tests (or sometimes
specific assertions) that aren't appropriate for WPT.

> I'm also concerned that with 4 vendors upstreaming their WPT tests, the
> WPT repo will just become a morass of partially overlapping tests that
> takes 4x longer to run than a curated repo.

This gets at another consideration that hasn't been mentioned yet:
participating in WPT like this expands the responsibilities of authors and
reviewers.  Contributors may have to spend extra time cataloging their new
tests (or finding the correct test to extend). Likewise, reviewers will need
to develop some familiarity with certain sub-directories of WPT.

This can be kind of a drag, but only if you aren't committed to the
long-term benefits of sharing tests. From that perspective, meaningful
organization is just another part of the job, and the value accrues with
every new test.

Which is all to say: buy-in is important. If the team decides to participate in
WPT, folks ought to be excited about it!
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
1234