Quantcast

Where do we put WPT tests to be exported

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Where do we put WPT tests to be exported

Ryosuke Niwa-2
Hi all,

Youenn is working on a patch to automatically create a GitHub PR to
export tests from a WebKit patch which includes changes to
web-platform-tests [1].

That raises a question as to where we should put new tests or modified
tests intended to be exported to web-platform-tests from WebKit.

I think the most obvious option is to use
LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
thread about adopting testharness.js (titled Another WPT bite), Maciej
briefly expressed the preference for creating a new directory:
https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html

Do other people have strong opinions about this?

- R. Niwa

[1] https://bugs.webkit.org/show_bug.cgi?id=169462
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet
I see two main cases:
- Writer of the patch is making sure to upstream WPT test changes at WebKit landing time. It is ok to make the changes directly in LayoutTests/imported/w3c/web-platform-tests/
- Writer plans to upstream WPT test changes at some point but wants more time. It is better to develop the tests in LayoutTests/http/wpt and then migrate them later on.

I would start with an experimental phase with some of us making direct changes in LayoutTests/imported/w3c/web-platform-tests/.
When we are happy with the tools and think the risk for issues is low enough (or when the bots can handle most of it for us), hacking LayoutTests/imported/w3c/web-platform-tests/ could be the default.

Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa <[hidden email]> a écrit :
Hi all,

Youenn is working on a patch to automatically create a GitHub PR to
export tests from a WebKit patch which includes changes to
web-platform-tests [1].

That raises a question as to where we should put new tests or modified
tests intended to be exported to web-platform-tests from WebKit.

I think the most obvious option is to use
LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
thread about adopting testharness.js (titled Another WPT bite), Maciej
briefly expressed the preference for creating a new directory:
https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html

Do other people have strong opinions about this?

- R. Niwa

[1] https://bugs.webkit.org/show_bug.cgi?id=169462
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Maciej Stachowiak


On May 15, 2017, at 9:08 PM, youenn fablet <[hidden email]> wrote:

I see two main cases:
- Writer of the patch is making sure to upstream WPT test changes at WebKit landing time. It is ok to make the changes directly in LayoutTests/imported/w3c/web-platform-tests/
- Writer plans to upstream WPT test changes at some point but wants more time. It is better to develop the tests in LayoutTests/http/wpt and then migrate them later on.

My proposal was different from either of these, it was to have a directory specifically for tests meant to be upstreamed (LayoutTests/http/wpt should contain only tests not meant

I think adding new tests directly to LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most stuff in the imported/ directory is an exact copy of an upstream test suite, so if you run only a specific directory of tests, you know you are running an official conformance suite. With this proposal, it might also contain random tests that will hopefully be upstreamed but maybe not, or might be changed before the PR lands upstream, or might get renamed, or whatever. There's no guarantee that updating from the official version will ever fully resolve the delta. 

I think it would be more elegant to have a parallel directory (LayoutTests/for-export/w3c/web-platform-tests). Then when something is actually upstreamed and then pulled back down, we could delete the staged version. Directly modifying our local copy seems like it could easily lead to long-term divergences slipping through the cracks.

Of course, people could always go run the official copy from w3c-test.org . But we usually leave imported conformance test suites unmodified except the minimum necessary to make them run.

I would start with an experimental phase with some of us making direct changes in LayoutTests/imported/w3c/web-platform-tests/.
When we are happy with the tools and think the risk for issues is low enough (or when the bots can handle most of it for us), hacking LayoutTests/imported/w3c/web-platform-tests/ could be the default.

I think the problem with this plan is not tools risk, it's the fact that a directory of imported tests can no longer be trusted to actually just be imported tests. So a smaller number of people doing it to start would not address my concern. It might be that other people don't care about this. My opinion counts no more than anyone else's, and I'd be interested in hearing from others.

Regards,
Maciej


Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa <[hidden email]> a écrit :
Hi all,

Youenn is working on a patch to automatically create a GitHub PR to
export tests from a WebKit patch which includes changes to
web-platform-tests [1].

That raises a question as to where we should put new tests or modified
tests intended to be exported to web-platform-tests from WebKit.

I think the most obvious option is to use
LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
thread about adopting testharness.js (titled Another WPT bite), Maciej
briefly expressed the preference for creating a new directory:
https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html

Do other people have strong opinions about this?

- R. Niwa

[1] https://bugs.webkit.org/show_bug.cgi?id=169462
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Michael Catanzaro
In reply to this post by Ryosuke Niwa-2
On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa <[hidden email]> wrote:
> Hi all,
>
> Youenn is working on a patch to automatically create a GitHub PR to
> export tests from a WebKit patch which includes changes to
> web-platform-tests [1].
>
> That raises a question as to where we should put new tests or modified
> tests intended to be exported to web-platform-tests from WebKit.

Whatever you wind up doing, it would be helpful to add a README
somewhere that explains the workflow, so the process doesn't turn into
folklore only known to the developers who work on it regularly and
those who read this mailing list thread.

Thanks,

Michael

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet
In reply to this post by Maciej Stachowiak


Le lun. 15 mai 2017 à 23:15, Maciej Stachowiak <[hidden email]> a écrit :

On May 15, 2017, at 9:08 PM, youenn fablet <[hidden email]> wrote:

I see two main cases:
- Writer of the patch is making sure to upstream WPT test changes at WebKit landing time. It is ok to make the changes directly in LayoutTests/imported/w3c/web-platform-tests/
- Writer plans to upstream WPT test changes at some point but wants more time. It is better to develop the tests in LayoutTests/http/wpt and then migrate them later on.

My proposal was different from either of these, it was to have a directory specifically for tests meant to be upstreamed (LayoutTests/http/wpt should contain only tests not meant

The possibility to have LayoutTests/http/wpt/to-be-imported was mentionned previously.
I think it makes sense to run to-be-imported tests behind WPT as this is the way they will end up being run when upstreamed.
 

I think adding new tests directly to LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most stuff in the imported/ directory is an exact copy of an upstream test suite, so if you run only a specific directory of tests, you know you are running an official conformance suite. With this proposal, it might also contain random tests that will hopefully be upstreamed but maybe not, or might be changed before the PR lands upstream, or might get renamed, or whatever. There's no guarantee that updating from the official version will ever fully resolve the delta. 

LayoutTests/imported/w3c/web-platform-tests is mostly about regression testing.
For conformance testing, it is probably more accurate to either use w3c-test.org (probably not reliable enough though) or make a clone of W3C WPT, do the set up as specified in their README (etc/hosts...) and use wptrunner.
 

I think it would be more elegant to have a parallel directory (LayoutTests/for-export/w3c/web-platform-tests). Then when something is actually upstreamed and then pulled back down, we could delete the staged version. Directly modifying our local copy seems like it could easily lead to long-term divergences slipping through the cracks.

A parallel directory is fine when starting a test suite.
Often, changes are limited to modifying a test, which might be best done in-place.
Often, adding a test is easier by adding it in an already existing file.

Tooling should protect us from diverging.
Authoring in LayoutTests/for-export/w3c/web-platform-tests might not allow you to use WPT tools and common resources.
At upstream time, reviewers will probably tell to use that and that existing resource.

There is agreement that a imported/w3c/web-platform-test layout test change that lands in WebKit can be merged upstreamed.
I am not sure what would be the process with LayoutTests/for-export/w3c/web-platform-tests tests.
 

Of course, people could always go run the official copy from w3c-test.org . But we usually leave imported conformance test suites unmodified except the minimum necessary to make them run.

I would start with an experimental phase with some of us making direct changes in LayoutTests/imported/w3c/web-platform-tests/.
When we are happy with the tools and think the risk for issues is low enough (or when the bots can handle most of it for us), hacking LayoutTests/imported/w3c/web-platform-tests/ could be the default.

I think the problem with this plan is not tools risk, it's the fact that a directory of imported tests can no longer be trusted to actually just be imported tests. So a smaller number of people doing it to start would not address my concern. It might be that other people don't care about this. My opinion counts no more than anyone else's, and I'd be interested in hearing from others.

I am not sure to understand precisely what the consequences of "imported tests can no longer be trusted to actually just be imported tests" are. Ideally, a WPT test change landing in WebKit should be merged at the same time on W3C WPT.

There was a suggestion that LayoutTests/imported/w3c/web-platform-tests be moved to a shorter path like LayoutTests/web-platform-tests. That would also make it clear that this folder is not only about one-way-sync.
 

Regards,
Maciej


Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa <[hidden email]> a écrit :
Hi all,

Youenn is working on a patch to automatically create a GitHub PR to
export tests from a WebKit patch which includes changes to
web-platform-tests [1].

That raises a question as to where we should put new tests or modified
tests intended to be exported to web-platform-tests from WebKit.

I think the most obvious option is to use
LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
thread about adopting testharness.js (titled Another WPT bite), Maciej
briefly expressed the preference for creating a new directory:
https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html

Do other people have strong opinions about this?

- R. Niwa

[1] https://bugs.webkit.org/show_bug.cgi?id=169462
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet
In reply to this post by Michael Catanzaro
That is a fair point!
For importing tests, effort was done to ease the use of the import-w3c-tests script.
Documentation should be in "Tools/Scripts/import-w3c-tests --help"
I will be happy to beef it up based on suggestions.
    y

Le mar. 16 mai 2017 à 07:55, Michael Catanzaro <[hidden email]> a écrit :
On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa <[hidden email]> wrote:
> Hi all,
>
> Youenn is working on a patch to automatically create a GitHub PR to
> export tests from a WebKit patch which includes changes to
> web-platform-tests [1].
>
> That raises a question as to where we should put new tests or modified
> tests intended to be exported to web-platform-tests from WebKit.

Whatever you wind up doing, it would be helpful to add a README
somewhere that explains the workflow, so the process doesn't turn into
folklore only known to the developers who work on it regularly and
those who read this mailing list thread.

Thanks,

Michael

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Konstantin Tokarev
In reply to this post by Michael Catanzaro


16.05.2017, 17:54, "Michael Catanzaro" <[hidden email]>:

> On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa <[hidden email]> wrote:
>>  Hi all,
>>
>>  Youenn is working on a patch to automatically create a GitHub PR to
>>  export tests from a WebKit patch which includes changes to
>>  web-platform-tests [1].
>>
>>  That raises a question as to where we should put new tests or modified
>>  tests intended to be exported to web-platform-tests from WebKit.
>
> Whatever you wind up doing, it would be helpful to add a README
> somewhere that explains the workflow, so the process doesn't turn into
> folklore only known to the developers who work on it regularly and
> those who read this mailing list thread.

I think it would also be useful to add wiki page to "Layout Tests" section

>
> Thanks,
>
> Michael
>
> _______________________________________________
> webkit-dev mailing list
> [hidden email]
> https://lists.webkit.org/mailman/listinfo/webkit-dev

--
Regards,
Konstantin
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Ryosuke Niwa-2
On Tue, May 16, 2017 at 8:42 AM, Konstantin Tokarev <[hidden email]> wrote:

>
>
> 16.05.2017, 17:54, "Michael Catanzaro" <[hidden email]>:
>> On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa <[hidden email]> wrote:
>>>  Hi all,
>>>
>>>  Youenn is working on a patch to automatically create a GitHub PR to
>>>  export tests from a WebKit patch which includes changes to
>>>  web-platform-tests [1].
>>>
>>>  That raises a question as to where we should put new tests or modified
>>>  tests intended to be exported to web-platform-tests from WebKit.
>>
>> Whatever you wind up doing, it would be helpful to add a README
>> somewhere that explains the workflow, so the process doesn't turn into
>> folklore only known to the developers who work on it regularly and
>> those who read this mailing list thread.
>
> I think it would also be useful to add wiki page to "Layout Tests" section

With all likelihood, we'd just update webkit.org or the wiki. We don't
typically write a README for things like this.

- R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet
In reply to this post by youenn fablet


Le jeu. 18 mai 2017 à 05:09, Philip Jägenstedt <[hidden email]> a écrit :

FWIW, I think it makes sense to treat wpt as both regression tests and conformance tests. They're regression tests because they can in fact catch regressions. And they're conformance tests because they're supposed to be derived from the specs, and that should be true of local modifications as well if they're intended to be upstreamed.


When a spec gets updated, the WPT tests will ideally be updated at the same time.
The updated tests will no longer ensure non-regression until the browser implements the new behavior.
The same applies to deprecated features for which WPT tests might be removed for instance.
I don't know what WPT can do there. This issue is probably limited in practice.


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Anne van Kesteren
On Fri, May 19, 2017 at 5:11 PM, youenn fablet <[hidden email]> wrote:
> When a spec gets updated, the WPT tests will ideally be updated at the same
> time.
> The updated tests will no longer ensure non-regression until the browser
> implements the new behavior.

Ideally if importing test updates results in failures those end up
being tracked as new bugs. I realize that's not how it works today,
but that's how it should work I think.


> The same applies to deprecated features for which WPT tests might be removed
> for instance.

In this case it's expected that a "historical" test is added that
tests the feature isn't supported, so there's still coverage.


> I don't know what WPT can do there. This issue is probably limited in
> practice.

Modifying tests doesn't happen often, but it definitely does happen
and it would be nice if infrastructure accounted for it. (I've often
added additional assertions or additional tests to an existing
resource.)


--
https://annevankesteren.nl/
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet


Le ven. 19 mai 2017 à 09:44, Anne van Kesteren <[hidden email]> a écrit :
On Fri, May 19, 2017 at 5:11 PM, youenn fablet <[hidden email]> wrote:
> When a spec gets updated, the WPT tests will ideally be updated at the same
> time.
> The updated tests will no longer ensure non-regression until the browser
> implements the new behavior.

Ideally if importing test updates results in failures those end up
being tracked as new bugs. I realize that's not how it works today,
but that's how it should work I think.

Right, and that is very good as long as there is effort to match the implementation with the spec.
Until the bug is fixed, regressions related to the no-longer compliant behavior might not be caught by WPT.
 


> The same applies to deprecated features for which WPT tests might be removed
> for instance.

In this case it's expected that a "historical" test is added that
tests the feature isn't supported, so there's still coverage.

Sure, it is good to test conformance, but there is no longer regression testing on the deprecated feature behavior itself.
Ideally, this would require tighter syncing between the browsers. Might have positive and/or negative consequences.

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Ryosuke Niwa-2
On Fri, May 19, 2017 at 10:17 AM, youenn fablet <[hidden email]> wrote:

>
>
> Le ven. 19 mai 2017 à 09:44, Anne van Kesteren <[hidden email]> a écrit :
>>
>> On Fri, May 19, 2017 at 5:11 PM, youenn fablet <[hidden email]> wrote:
>> > When a spec gets updated, the WPT tests will ideally be updated at the
>> > same
>> > time.
>> > The updated tests will no longer ensure non-regression until the browser
>> > implements the new behavior.
>>
>> Ideally if importing test updates results in failures those end up
>> being tracked as new bugs. I realize that's not how it works today,
>> but that's how it should work I think.
>
>
> Right, and that is very good as long as there is effort to match the
> implementation with the spec.
> Until the bug is fixed, regressions related to the no-longer compliant
> behavior might not be caught by WPT.
>
>>
>>
>>
>> > The same applies to deprecated features for which WPT tests might be
>> > removed
>> > for instance.
>>
>> In this case it's expected that a "historical" test is added that
>> tests the feature isn't supported, so there's still coverage.
>
>
> Sure, it is good to test conformance, but there is no longer regression
> testing on the deprecated feature behavior itself.
> Ideally, this would require tighter syncing between the browsers. Might have
> positive and/or negative consequences.

I don't think we should be changing anything about our process to
deprecate a feature based on changes to web platform tests or its
limitations.

If web platform tests end up deleting tests that we decide to continue
to support, we should keep them elsewhere in WebKit repository instead
of deleting those features.

I would most vocally and strongly oppose to making any feature or
policy decisions in WebKit solely based on changes to or processes in
web platform tests.

- R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet

> Sure, it is good to test conformance, but there is no longer regression
> testing on the deprecated feature behavior itself.
> Ideally, this would require tighter syncing between the browsers. Might have
> positive and/or negative consequences.

I don't think we should be changing anything about our process to
deprecate a feature based on changes to web platform tests or its
limitations.

Agreed.
 

If web platform tests end up deleting tests that we decide to continue
to support, we should keep them elsewhere in WebKit repository instead
of deleting those features.

Exactly, we should be ready for those cases.
When doing a full resync, it is just impossible to check whether each modification is about feature removal.


I would most vocally and strongly oppose to making any feature or
policy decisions in WebKit solely based on changes to or processes in
web platform tests.

- R. Niwa

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

Maciej Stachowiak
In reply to this post by youenn fablet


On May 18, 2017, at 2:08 PM, Philip Jägenstedt <[hidden email]> wrote:

On Tue, May 16, 2017 at 5:38 PM youenn fablet <[hidden email]> wrote:

There was a suggestion that LayoutTests/imported/w3c/web-platform-tests be moved to a shorter path like LayoutTests/web-platform-tests. That would also make it clear that this folder is not only about one-way-sync.

In Blink we renamed it to LayoutTests/external/wpt/ to avoid the "read only" implications of the old name.

If we go forward with two-way sync, I think this is a good name.

Regards,
Maciej



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Where do we put WPT tests to be exported

youenn fablet

Le dim. 21 mai 2017 à 00:40, Maciej Stachowiak <[hidden email]> a écrit :
On May 18, 2017, at 2:08 PM, Philip Jägenstedt <[hidden email]> wrote:

On Tue, May 16, 2017 at 5:38 PM youenn fablet <[hidden email]> wrote:

There was a suggestion that LayoutTests/imported/w3c/web-platform-tests be moved to a shorter path like LayoutTests/web-platform-tests. That would also make it clear that this folder is not only about one-way-sync.

In Blink we renamed it to LayoutTests/external/wpt/ to avoid the "read only" implications of the old name.

If we go forward with two-way sync, I think this is a good name.

Regards,
Maciej



_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Loading...