Python

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Python

Michael Catanzaro
Hi,

As you're probably already aware, in Arch Linux /usr/bin/python has
been a symlink to /usr/bin/python3 for a long time now. In practice
this means that Arch users are not going to be able to use basically
any of our Python scripts, since our scripts use the shebang
#!/usr/bin/env python but expect it to be python2. This has not really
been a problem until now because none of Igalia's developers use Arch,
and occasional contributions rarely need to use the Python scripts. But
our scripts have been broken this whole time, since the Python
maintainers have decided the only reasonable way to avoid this problem
is for scripts to explicitly specify either python2 or python3, and we
have not been doing that. To be clear: the problem is that
#!/usr/bin/env python is *python3* on some systems, but *python2* on
others. WebKit scripts incorrectly assume it is always python2. I say
"incorrectly" because the Python folks have ratified some bizarre
standard for how to handle this, PEP 394 [1], which states: "in
preparation for an eventual change in the default version of Python,
Python 2 only scripts should either be updated to be source compatible
with Python 3 or else to use python2 in the shebang line." Lovely.

Now Fedora is planning to switch /usr/bin/python to link to
/usr/bin/python3 instead of /usr/bin/python2 [2]. Not for another three
years, but many of us (hi! :) use Fedora, so we're going to have to fix
our scripts by then. This could be really easy, or it could be really
hard.

The really easy solution is to just replace #!/usr/bin/env python with
#!/usr/bin/env python2 everywhere. This will work on all Linux distros
newer than Red Hat Enterprise Linux 6 or thereabouts, which we
definitely do not care about anymore. It doesn't work in very old
distros because they did not provide /usr/bin/python2, only
/usr/bin/python and very specific links like /usr/bin/python2.6. That
doesn't matter because we don't care about those old systems anymore,
but the problem is that last I checked, macOS did not provide a python2
binary either. I am hoping that has changed in the past few years. Has
it? If so, I can upload a patch to do this simple find/replace and we
can all move on. If this is the case, yay! Stop reading now! Let me
know. We should do that!

But if not, we need to decide what to do among bad options:

 * We could replace all our Python scripts that have shebangs with .in
files, and perform configure replacements at CMake/XCode configure time
to generate the scripts with the right python executable path. This
would mean the scripts would be generated into the WebKit build
directory instead of the source directory, so it will break those of us
who have added Tools/Scripts to our PATH. It's not clear how they would
find the right resource paths in the source directory, especially if
using a build directory other than WebKit/WebKitBuild. I'm not even
sure if XCode can do configure replacements. This would be a mess.
 * We could require Mac developers to install a python2 symlink
somewhere in PATH. Not really desirable, but possibly tolerable?
 * We could port all our Python scripts to python3 and use
#!/usr/bin/env python3, which will surely be unambiguous. But my
understanding is that macOS does not have python3, so it will need to
be installed locally. I guess that would be tolerable? Anyway, the main
disadvantage here is we'd have to make all our scripts work with
python3. If WebKit was a Linux-only project, we should definitely do
that anyway, since major Linux distros no longer ship python2 by
default: it's something we install extra now. But if macOS is sticking
with python2, it would probably be undesirable to port our scripts to
python3.
 * We could make all our Python scripts compatible with both python2
and python3, and maintain them this way forever. I really hope we can
avoid this option, since we'd be breaking these scripts all the time as
we would all surely not test one configuration or the other.

Any other ideas?

Michael

[1] https://www.python.org/dev/peps/pep-0394/
[2] https://lwn.net/SubscriberLink/729366/1d3cd2dbaea7070b/

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Konstantin Tokarev


07.08.2017, 16:47, "Michael Catanzaro" <[hidden email]>:

> Hi,
>
> As you're probably already aware, in Arch Linux /usr/bin/python has
> been a symlink to /usr/bin/python3 for a long time now. In practice
> this means that Arch users are not going to be able to use basically
> any of our Python scripts, since our scripts use the shebang
> #!/usr/bin/env python but expect it to be python2. This has not really
> been a problem until now because none of Igalia's developers use Arch,
> and occasional contributions rarely need to use the Python scripts. But
> our scripts have been broken this whole time, since the Python
> maintainers have decided the only reasonable way to avoid this problem
> is for scripts to explicitly specify either python2 or python3, and we
> have not been doing that. To be clear: the problem is that
> #!/usr/bin/env python is *python3* on some systems, but *python2* on
> others. WebKit scripts incorrectly assume it is always python2. I say
> "incorrectly" because the Python folks have ratified some bizarre
> standard for how to handle this, PEP 394 [1], which states: "in
> preparation for an eventual change in the default version of Python,
> Python 2 only scripts should either be updated to be source compatible
> with Python 3 or else to use python2 in the shebang line." Lovely.
>
> Now Fedora is planning to switch /usr/bin/python to link to
> /usr/bin/python3 instead of /usr/bin/python2 [2]. Not for another three
> years, but many of us (hi! :) use Fedora, so we're going to have to fix
> our scripts by then. This could be really easy, or it could be really
> hard.
>
> The really easy solution is to just replace #!/usr/bin/env python with
> #!/usr/bin/env python2 everywhere. This will work on all Linux distros
> newer than Red Hat Enterprise Linux 6 or thereabouts, which we
> definitely do not care about anymore. It doesn't work in very old
> distros because they did not provide /usr/bin/python2, only
> /usr/bin/python and very specific links like /usr/bin/python2.6. That
> doesn't matter because we don't care about those old systems anymore,
> but the problem is that last I checked, macOS did not provide a python2
> binary either. I am hoping that has changed in the past few years. Has
> it? If so, I can upload a patch to do this simple find/replace and we
> can all move on. If this is the case, yay! Stop reading now! Let me
> know. We should do that!
>
> But if not, we need to decide what to do among bad options:
>
>  * We could replace all our Python scripts that have shebangs with .in
> files, and perform configure replacements at CMake/XCode configure time
> to generate the scripts with the right python executable path. This
> would mean the scripts would be generated into the WebKit build
> directory instead of the source directory, so it will break those of us
> who have added Tools/Scripts to our PATH. It's not clear how they would
> find the right resource paths in the source directory, especially if
> using a build directory other than WebKit/WebKitBuild. I'm not even
> sure if XCode can do configure replacements. This would be a mess.
>  * We could require Mac developers to install a python2 symlink
> somewhere in PATH. Not really desirable, but possibly tolerable?
>  * We could port all our Python scripts to python3 and use
> #!/usr/bin/env python3, which will surely be unambiguous. But my
> understanding is that macOS does not have python3, so it will need to
> be installed locally. I guess that would be tolerable? Anyway, the main
> disadvantage here is we'd have to make all our scripts work with
> python3. If WebKit was a Linux-only project, we should definitely do
> that anyway, since major Linux distros no longer ship python2 by
> default: it's something we install extra now. But if macOS is sticking
> with python2, it would probably be undesirable to port our scripts to
> python3.
>  * We could make all our Python scripts compatible with both python2
> and python3, and maintain them this way forever. I really hope we can
> avoid this option, since we'd be breaking these scripts all the time as
> we would all surely not test one configuration or the other.

I think the latter option is quite feasible. The most widespread reason of
incompatibility with python3 is use of print operator instead of print()
function. I have a local patch that fixes these things, and it can be enforced by
style checker to keep new code compatible.

There are other issues, which are affecting just a few scripts (unfortunately they
are used in a build process). Those can be fixed once, and then we can rely on
python3 users to report incompatibilities. Anyway, these issues affect only code
that is already "legacy" in terms of python 2.7

>
> Any other ideas?
>
> Michael
>
> [1] https://www.python.org/dev/peps/pep-0394/
> [2] https://lwn.net/SubscriberLink/729366/1d3cd2dbaea7070b/
>
> _______________________________________________
> webkit-dev mailing list
> [hidden email]
> https://lists.webkit.org/mailman/listinfo/webkit-dev

--
Regards,
Konstantin
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Michael Catanzaro
In reply to this post by Michael Catanzaro
On Mon, Aug 7, 2017 at 8:47 AM, Michael Catanzaro
<[hidden email]> wrote:
>  This has not really been a problem until now because none of
> Igalia's developers use Arch, and occasional contributions rarely
> need to use the Python scripts.

I stand corrected: we have one developer who has been using an
elaborate local hack [1] to get WebKit's python scripts to work. Wow....

[1]
https://wiki.archlinux.org/index.php/python#Dealing_with_version_problem_in_build_scripts

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Andy Estes
In reply to this post by Michael Catanzaro


On Aug 7, 2017, at 6:47 AM, Michael Catanzaro <[hidden email]> wrote:

last I checked, macOS did not provide a python2 binary either. I am hoping that has changed in the past few years. Has it?

Nope.

Andy

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Andy Estes


On Aug 7, 2017, at 8:41 AM, Andy Estes <[hidden email]> wrote:



On Aug 7, 2017, at 6:47 AM, Michael Catanzaro <[hidden email]> wrote:

last I checked, macOS did not provide a python2 binary either. I am hoping that has changed in the past few years. Has it?

Nope.

macOS does have /usr/bin/python2.7, though.


Andy
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Carlos Alberto Lopez Perez
On 07/08/17 17:44, Andy Estes wrote:
>>> last I checked, macOS did not provide a python2 binary either. I am hoping that has changed in the past few years. Has it?
>> Nope.
> macOS does have /usr/bin/python2.7, though.
>

That's a good thing.

I believe all Linux distros we support have this, right?
And all the scripts actually assume python2.7 (AFAIK).

Would it work for everyone having as shebang :

#!/usr/bin/env python2.7

?


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

signature.asc (914 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Michael Catanzaro
On Mon, Aug 7, 2017 at 10:56 AM, Carlos Alberto Lopez Perez
<[hidden email]> wrote:

> That's a good thing.
>
> I believe all Linux distros we support have this, right?
> And all the scripts actually assume python2.7 (AFAIK).
>
> Would it work for everyone having as shebang :
>
> #!/usr/bin/env python2.7
>
> ?

Um... that's really terrible, but I guess it should work, yes. Perhaps
that's the path of least resistance for us.

Michael

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Olmstead, Don-2
In reply to this post by Michael Catanzaro
There are automated tools that will update to Python 3 while keeping compatibility with Python 2.

There is future, http://python-future.org/automatic_conversion.html,  and modernizer, https://python-modernize.readthedocs.io/en/latest 

This came up in https://bugs.webkit.org/show_bug.cgi?id=170439 and it seemed like 2/3 compatibility would be the ideal solution.

-----Original Message-----
From: webkit-dev [mailto:[hidden email]] On Behalf Of Michael Catanzaro
Sent: Monday, August 7, 2017 8:13 AM
To: [hidden email]
Subject: Re: [webkit-dev] Python

On Mon, Aug 7, 2017 at 8:47 AM, Michael Catanzaro <[hidden email]> wrote:
>  This has not really been a problem until now because none of Igalia's
> developers use Arch, and occasional contributions rarely need to use
> the Python scripts.

I stand corrected: we have one developer who has been using an elaborate local hack [1] to get WebKit's python scripts to work. Wow....

[1]
https://wiki.archlinux.org/index.php/python#Dealing_with_version_problem_in_build_scripts

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Keith Miller
In reply to this post by Michael Catanzaro
Out of curiosity, is this just about building or is part of the concern with development scripts?

AFAICT, we invoke all our python scripts by invoking $(PYTHON), which can be whatever path you need. I didn’t look too closely though...

Cheers,
Keith

> On Aug 7, 2017, at 6:47 AM, Michael Catanzaro <[hidden email]> wrote:
>
> Hi,
>
> As you're probably already aware, in Arch Linux /usr/bin/python has been a symlink to /usr/bin/python3 for a long time now. In practice this means that Arch users are not going to be able to use basically any of our Python scripts, since our scripts use the shebang #!/usr/bin/env python but expect it to be python2. This has not really been a problem until now because none of Igalia's developers use Arch, and occasional contributions rarely need to use the Python scripts. But our scripts have been broken this whole time, since the Python maintainers have decided the only reasonable way to avoid this problem is for scripts to explicitly specify either python2 or python3, and we have not been doing that. To be clear: the problem is that #!/usr/bin/env python is *python3* on some systems, but *python2* on others. WebKit scripts incorrectly assume it is always python2. I say "incorrectly" because the Python folks have ratified some bizarre standard for how to handle this, PEP 394 [1], which states: "in preparation for an eventual change in the default version of Python, Python 2 only scripts should either be updated to be source compatible with Python 3 or else to use python2 in the shebang line." Lovely.
>
> Now Fedora is planning to switch /usr/bin/python to link to /usr/bin/python3 instead of /usr/bin/python2 [2]. Not for another three years, but many of us (hi! :) use Fedora, so we're going to have to fix our scripts by then. This could be really easy, or it could be really hard.
>
> The really easy solution is to just replace #!/usr/bin/env python with #!/usr/bin/env python2 everywhere. This will work on all Linux distros newer than Red Hat Enterprise Linux 6 or thereabouts, which we definitely do not care about anymore. It doesn't work in very old distros because they did not provide /usr/bin/python2, only /usr/bin/python and very specific links like /usr/bin/python2.6. That doesn't matter because we don't care about those old systems anymore, but the problem is that last I checked, macOS did not provide a python2 binary either. I am hoping that has changed in the past few years. Has it? If so, I can upload a patch to do this simple find/replace and we can all move on. If this is the case, yay! Stop reading now! Let me know. We should do that!
>
> But if not, we need to decide what to do among bad options:
>
> * We could replace all our Python scripts that have shebangs with .in files, and perform configure replacements at CMake/XCode configure time to generate the scripts with the right python executable path. This would mean the scripts would be generated into the WebKit build directory instead of the source directory, so it will break those of us who have added Tools/Scripts to our PATH. It's not clear how they would find the right resource paths in the source directory, especially if using a build directory other than WebKit/WebKitBuild. I'm not even sure if XCode can do configure replacements. This would be a mess.
> * We could require Mac developers to install a python2 symlink somewhere in PATH. Not really desirable, but possibly tolerable?
> * We could port all our Python scripts to python3 and use #!/usr/bin/env python3, which will surely be unambiguous. But my understanding is that macOS does not have python3, so it will need to be installed locally. I guess that would be tolerable? Anyway, the main disadvantage here is we'd have to make all our scripts work with python3. If WebKit was a Linux-only project, we should definitely do that anyway, since major Linux distros no longer ship python2 by default: it's something we install extra now. But if macOS is sticking with python2, it would probably be undesirable to port our scripts to python3.
> * We could make all our Python scripts compatible with both python2 and python3, and maintain them this way forever. I really hope we can avoid this option, since we'd be breaking these scripts all the time as we would all surely not test one configuration or the other.
>
> Any other ideas?
>
> Michael
>
> [1] https://www.python.org/dev/peps/pep-0394/
> [2] https://lwn.net/SubscriberLink/729366/1d3cd2dbaea7070b/
>
> _______________________________________________
> webkit-dev mailing list
> [hidden email]
> https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Michael Catanzaro
On Mon, Aug 7, 2017 at 1:17 PM, Keith Miller <[hidden email]>
wrote:
> Out of curiosity, is this just about building or is part of the
> concern with development scripts?

Both.


_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Adrian Perez-10
In reply to this post by Carlos Alberto Lopez Perez
Hi all,

What I have been doing is having a Python 2.7 virtualenv [1], which gets
automatically enabled (or disabled) when changing directory under (or out of)
the WebKit source tree by means of zsh-autoenv [2]. This way the “python”
binary is the one for Python 2.7 inside the virtualenv, and I do not need to
remember about doing any setup manually when navigating directories in the
shell.

On Mon, 7 Aug 2017 17:56:51 +0200, Carlos Alberto Lopez Perez <[hidden email]> wrote:
> On 07/08/17 17:44, Andy Estes wrote:
> >>> last I checked, macOS did not provide a python2 binary either. I am hoping that has changed in the past few years. Has it?
> >> Nope.
> > macOS does have /usr/bin/python2.7, though.
>
> That's a good thing.

This would work for people using a virtualenv as well:

  WebKit % which python2.7
  /home/aperez/.virtualenvs/webkit/bin/python2.7
  WebKit %

> I believe all Linux distros we support have this, right?
> And all the scripts actually assume python2.7 (AFAIK).

Arch Linux has a “python2.7” binary system-wide, too. The default installation
done when building Python always creates:

  ${prefix}/bin/python2.7
  ${prefix}/bin/python2 -> python2.7
  ${prefix}/bin/python -> python2.7

So I would expect most systems which ship Python to have a “python2.7” binary.

Cheers,


---
[1] https://virtualenv.pypa.io/en/stable/
[2] https://github.com/Tarrasch/zsh-autoenv

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev

attachment0 (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Michael Catanzaro
In reply to this post by Carlos Alberto Lopez Perez
On Mon, Aug 7, 2017 at 10:56 AM, Carlos Alberto Lopez Perez
<[hidden email]> wrote:

> That's a good thing.
>
> I believe all Linux distros we support have this, right?
> And all the scripts actually assume python2.7 (AFAIK).
>
> Would it work for everyone having as shebang :
>
> #!/usr/bin/env python2.7
>
> ?

So this is by far the easiest solution of those that have been
proposed. I think we should try this as a stopgap until macOS is fixed
to provide a python2 binary.

Michael

_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Ryosuke Niwa-2
On Fri, Aug 11, 2017 at 8:06 AM, Michael Catanzaro
<[hidden email]> wrote:

> On Mon, Aug 7, 2017 at 10:56 AM, Carlos Alberto Lopez Perez
> <[hidden email]> wrote:
>>
>> That's a good thing.
>>
>> I believe all Linux distros we support have this, right?
>> And all the scripts actually assume python2.7 (AFAIK).
>>
>> Would it work for everyone having as shebang :
>>
>> #!/usr/bin/env python2.7
>>
>> ?
>
>
> So this is by far the easiest solution of those that have been proposed. I
> think we should try this as a stopgap until macOS is fixed to provide a
> python2 binary.

What's wrong with using that as the permanent solution until we make
it Python3 compatible?

I don't think it's wise to assume /usr/bin/python2 will be added on
Mac anytime soon.

- R. Niwa
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python

Alicia Boya García
On 08/12/2017 11:40 PM, Ryosuke Niwa wrote:

> On Fri, Aug 11, 2017 at 8:06 AM, Michael Catanzaro
> <[hidden email]> wrote:
>> On Mon, Aug 7, 2017 at 10:56 AM, Carlos Alberto Lopez Perez
>> <[hidden email]> wrote:
>>>
>>> That's a good thing.
>>>
>>> I believe all Linux distros we support have this, right?
>>> And all the scripts actually assume python2.7 (AFAIK).
>>>
>>> Would it work for everyone having as shebang :
>>>
>>> #!/usr/bin/env python2.7
>>>
>>> ?
>>
>>
>> So this is by far the easiest solution of those that have been proposed. I
>> think we should try this as a stopgap until macOS is fixed to provide a
>> python2 binary.
>
> What's wrong with using that as the permanent solution until we make
> it Python3 compatible?
>
> I don't think it's wise to assume /usr/bin/python2 will be added on
> Mac anytime soon.
>
> - R. Niwa
> _______________________________________________
> webkit-dev mailing list
> [hidden email]
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>

+1, python2.7 is well supported among different systems.

It would be desirable to move to Python 3 eventually, but that's an
endeavor that should not be taken lightly, as many things can break in
subtle ways.

--
Alicia
_______________________________________________
webkit-dev mailing list
[hidden email]
https://lists.webkit.org/mailman/listinfo/webkit-dev
Loading...