need help with reinplace

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

need help with reinplace

Helmut K. C. Tessarek
I've been trying for hours now, but I'm not sure where the problem is.
Either there's an issue with escaping or there's a bug somewhere. I know
regex, but this is ridiculous. It's not working and I can't figure out why.

I want to replace the following line in configure.ac
AC_INIT([tmux], next-2.9)
with
AC_INIT([tmux], next-2.9 (${version}))

The

reinplace -E "s|^AC_INIT\(\[tmux\],
(next-\[0-9\]+\.\[0-9\]+)\)$|AC_INIT(\[tmux\], \\1 ${version})|g"
${worksrcpath}/configure.ac

With that I get an error message: invalid command name "0-9"

Weird, but ok. Then I tried to escape the [] character class brackets
and got the following:
Warning: reinplace s|^AC_INIT([tmux],
(next-[0-9]+.[0-9]+))$|AC_INIT([tmux], \1 20181031-b1ad075)|g didn't
change anything in ....

So it seems, it parses the grouping brackets as literal brackets and I'm
really at a loss here.

What is going on?

With sed a working version would look like this:
sed -E "s/AC_INIT\(\[tmux\], (next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \\1
(${version}))/g"

Anyhow, can someone please let me know what I am doing wrong and how to
fix this?

Cheers,
  K. C.

--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: need help with reinplace

Joshua Root-8
On 2018-11-3 10:23 , Helmut K. C. Tessarek wrote:

> I've been trying for hours now, but I'm not sure where the problem is.
> Either there's an issue with escaping or there's a bug somewhere. I know
> regex, but this is ridiculous. It's not working and I can't figure out why.
>
> I want to replace the following line in configure.ac
> AC_INIT([tmux], next-2.9)
> with
> AC_INIT([tmux], next-2.9 (${version}))
>
> The
>
> reinplace -E "s|^AC_INIT\(\[tmux\],
> (next-\[0-9\]+\.\[0-9\]+)\)$|AC_INIT(\[tmux\], \\1 ${version})|g"
> ${worksrcpath}/configure.ac
>
> With that I get an error message: invalid command name "0-9"
>
> Weird, but ok. Then I tried to escape the [] character class brackets
> and got the following:
> Warning: reinplace s|^AC_INIT([tmux],
> (next-[0-9]+.[0-9]+))$|AC_INIT([tmux], \1 20181031-b1ad075)|g didn't
> change anything in ....
>
> So it seems, it parses the grouping brackets as literal brackets and I'm
> really at a loss here.
>
> What is going on?
>
> With sed a working version would look like this:
> sed -E "s/AC_INIT\(\[tmux\], (next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \\1
> (${version}))/g"
>
> Anyhow, can someone please let me know what I am doing wrong and how to
> fix this?

This is one of those annoying situations in which more than one layer
needs escaping. Both the Tcl parser and sed assign a special meaning to
brackets and substitute backslash-escaped characters, so they need to be
protected from both. You could use braces to stop the Tcl parser from
substituting anything, but you do want to substitute ${version}.

This works:

set reinplace_cmd {s/AC_INIT\(\[tmux\],
(next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \1 (}
append reinplace_cmd ${version} {))/g}
reinplace -E $reinplace_cmd ${worksrcpath}/configure.ac

And so does this:

set reinplace_cmd [subst -nobackslashes -nocommands
{s/AC_INIT\(\[tmux\], (next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \1
(${version}))/g}]
reinplace -E $reinplace_cmd ${worksrcpath}/configure.ac

- Josh
Reply | Threaded
Open this post in threaded view
|

Re: need help with reinplace

Helmut K. C. Tessarek
Thanks a lot for the explanation. It would be great, if this were
somewhere in the documentation.

Or at least a reference to what `subst -nobackslashes -nocommands` does
and how to use it to build useful regexes. ;-)

On 2018-11-02 20:10, Joshua Root wrote:

> set reinplace_cmd [subst -nobackslashes -nocommands
> {s/AC_INIT\(\[tmux\], (next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \1
> (${version}))/g}]
> reinplace -E $reinplace_cmd ${worksrcpath}/configure.ac

I ended up using this one, since it had the entire regex on one line.

Thanks again!

Cheers,
  K. C.

--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: need help with reinplace

Ryan Schmidt-24


On Nov 2, 2018, at 19:40, Helmut K. C. Tessarek wrote:

> Or at least a reference to what `subst -nobackslashes -nocommands` does
> and how to use it to build useful regexes. ;-)

That's in the Tcl documentation:

https://www.tcl.tk/man/tcl8.5/TclCmd/subst.htm



Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] Re: need help with reinplace

Helmut K. C. Tessarek
Hi Ryan,

On 2018-11-03 01:45, Ryan Schmidt wrote:
> That's in the Tcl documentation:

I think you are missing my point. I don't really know Tcl, thus I would
have never thought about using `set reinplace_cmd` and or any other
workaround you Tcl experts know.

So why not add a paragraph like this to reinplace:

Since the Tcl way of how regexes are used/interpreted (or whatnot) in
reinplace truly sucks (and is fucked up), here's a way to use a regex as
it used in any other language.

set reinplace_cmd [subst -nobackslashes -nocommands
{s/bad/awesome/g}]
reinplace -E $reinplace_cmd FILE

The text in () brackets is optional.
Of course you can remove the sarcasm, but I think it has a nice touch.
Anyway, you get the idea.

Boom, all of a sudden people who read the documentation know how to use
a regex with Portfiles. Isn't that wonderful?

Cheers,
 K. C.

--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] Re: need help with reinplace

Joshua Root-8
On 2018-11-4 08:10 , Helmut K. C. Tessarek wrote:
> set reinplace_cmd [subst -nobackslashes -nocommands
> {s/bad/awesome/g}]
> reinplace -E $reinplace_cmd FILE

This example misses the point because the command does not contain any
characters that have a special meaning in Tcl. The only reason you
needed to use subst is because you had a variable you wanted substituted
*and* had other special characters that you didn't want substituted.

This works fine: reinplace -E "s/bad/awesome/g" FILE

So does this: reinplace -E "s/bad/${good}/g" FILE

And this: reinplace -E {s/\[bad\]/good/g} FILE

It's only when you have those literal brackets *and* you want ${good} to
be treated as a Tcl variable that you need a different approach. And you
have exactly this problem running sed from a shell, just with a
different set of characters that need protecting. Single quotes in the
shell are very similar to Tcl's braces, stopping parameters from being
expanded.

Note that you could also go another layer deep into backslash hell to
solve this problem, but I really try to avoid recommending that.

- Josh


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] Re: need help with reinplace

Helmut K. C. Tessarek

On 2018-11-03 21:48, Joshua Root wrote:
> This example misses the point because the command does not contain any
> characters that have a special meaning in Tcl. The only reason you
> needed to use subst is because you had a variable you wanted substituted
> *and* had other special characters that you didn't want substituted.

You are right, although the s/good/bad/g was just meant as a placeholder
for a regex.
And let's be honest, a "real" regex usually entails character classes,
back references, and all the stuff which makes us use regexes in the
first place.

I know that escaping and quoting can always be a mess.

It makes sense to use tcl variables in the regex, thus I believe the
above example could be of help for not tcl experts (like me).
We could replace the {s/bad/awesome/g} with {YOUR_REGEX_HERE}.
The fact that YOUR_REGEX_HERE can be any regex that is understood by all
regex engines makes this example worthwhile. You don't have to worry
about escaping, quoting, or whatnot. Just add a regex and it works.

> This works fine: reinplace -E "s/bad/awesome/g" FILE
>
> So does this: reinplace -E "s/bad/${good}/g" FILE
>
> And this: reinplace -E {s/\[bad\]/good/g} FILE

Right, but all these examples also work without the -E option.
In this case it's not a regex, but a regular substitution.

> It's only when you have those literal brackets *and* you want ${good} to
> be treated as a Tcl variable that you need a different approach. And you
> have exactly this problem running sed from a shell, just with a
> different set of characters that need protecting. Single quotes in the
> shell are very similar to Tcl's braces, stopping parameters from being
> expanded.

This works perfectly on the commandline without additional escaping:

cat configure.ac |sed -E "s/AC_INIT\(\[tmux\],
(next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \1 (${version}))/g"

(with export version=bla)

But I get your point. There are cases where additional escaping is
required. Still, IMO this additional escaping makes sense, while in the
reinplace case it does not. I was trying for hours to correctly escape
in reinplace.

> Note that you could also go another layer deep into backslash hell to
> solve this problem, but I really try to avoid recommending that.

Right, I think I've tried unsuccessfully for hours, before I gave up and
posted my failure in this list. ;-)

Cheers,
  K. C.

--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] Re: need help with reinplace

Joshua Root-8
On 2018-11-4 16:26 , Helmut K. C. Tessarek wrote:
> This works perfectly on the commandline without additional escaping:
>
> cat configure.ac |sed -E "s/AC_INIT\(\[tmux\],
> (next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \1 (${version}))/g"
>
> (with export version=bla)

Yes, because the brackets here don't have to be protected from the
shell. Imagine if the string you wanted to replace contained backticks
or dollar signs instead of brackets.

> But I get your point. There are cases where additional escaping is
> required. Still, IMO this additional escaping makes sense, while in the
> reinplace case it does not. I was trying for hours to correctly escape
> in reinplace.

What about it doesn't make sense to you? We need to understand this in
order to write documentation that explains it well.

- Josh


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] need help with reinplace

Ryan Schmidt-24
In reply to this post by Helmut K. C. Tessarek


On Nov 3, 2018, at 16:10, Helmut K. C. Tessarek wrote:

> Hi Ryan,
>
> On 2018-11-03 01:45, Ryan Schmidt wrote:
>> That's in the Tcl documentation:
>
> I think you are missing my point. I don't really know Tcl, thus I would
> have never thought about using `set reinplace_cmd` and or any other
> workaround you Tcl experts know.

In fact, the solution Joshua suggested had not occurred to me either, and I haven't come across it being used in any portfiles. I usually just deal with the extra backslashes when needed. If your regex has gotten so complicated that this becomes a problem, it may be time to rethink whether that regex was the best way to solve the problem. If the regex is so hard to understand that you're having trouble figuring out what needs to be escaped, it will also be hard for someone else to understand later, which will make it difficult when inevitably sometime in the future the port is updated to a new version and the message appears that the reinplace didn't change anything, because someone will then have to understand and either update or remove that regex.

For instance, you said:

> I want to replace the following line in configure.ac
> AC_INIT([tmux], next-2.9)
> with
> AC_INIT([tmux], next-2.9 (${version}))

The way I would approach this is to write a patch file which inserts "@VERSION@" into the file where you want the version to appear. Then you can simply reinplace "s|@VERSION@|${version}|g" to get the real version into the file. That's very simple to understand and it's a pattern we use in many other portfiles to insert the prefix or other portfile variables.


> So why not add a paragraph like this to reinplace:
>
> Since the Tcl way of how regexes are used/interpreted (or whatnot) in
> reinplace truly sucks (and is fucked up), here's a way to use a regex as
> it used in any other language.
>
> set reinplace_cmd [subst -nobackslashes -nocommands
> {s/bad/awesome/g}]
> reinplace -E $reinplace_cmd FILE
>
> The text in () brackets is optional.
> Of course you can remove the sarcasm, but I think it has a nice touch.
> Anyway, you get the idea.
>
> Boom, all of a sudden people who read the documentation know how to use
> a regex with Portfiles. Isn't that wonderful?

Certainly, and we do accept pull requests for the documentation on GitHub. :) In my opinion the guide needs a complete rewrite from scratch, but until someone finds the time and energy to do that minor improvements are of course welcomed.

Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] Re: [SPAM] need help with reinplace

Helmut K. C. Tessarek
On 2018-11-04 02:49, Ryan Schmidt wrote:
> The way I would approach this is to write a patch file which inserts
> "@VERSION@" into the file where you want the version to appear. Then
> you can simply reinplace "s|@VERSION@|${version}|g" to get the real
> version into the file. That's very simple to understand and it's a
> pattern we use in many other portfiles to insert the prefix or other
> portfile variables.

Yes, I thought about that. However, this means I have to create a new
patch file every time there's an update to that line.

Call me lazy (or more efficient), but I rather use one simple regex.
And yes, I definitely would not call the regex I wrote complex or
complicated. You shoud see some of the regexes that span 2 lines - those
are complicated and unreadable.

Anyhow, thanks for your input.

Cheers,
  K. C.

--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] Re: need help with reinplace

Helmut K. C. Tessarek
In reply to this post by Joshua Root-8
On 2018-11-04 01:34, Joshua Root wrote:
> What about it doesn't make sense to you? We need to understand this in
> order to write documentation that explains it well.

I thought I made this clear in my original post.

I want to use this regex in inreplace:

s/AC_INIT\(\[tmux\],
(next-[0-9]+\.[0-9]+)\)/AC_INIT([tmux], \1 (${version}))/g

No matter how I tried to escape it, it didn't work. But I don't have to
now, since you gave me this awesome workaround.

Problem solved. All good. End of story. I just meant that others could
benefit from this workaround as well. Thus my suggestion to add it to
the documentation. That's all.

Cheers,
  K. C.

--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: need help with reinplace

Ryan Schmidt-24
In reply to this post by Helmut K. C. Tessarek


On Nov 4, 2018, at 16:27, Helmut K. C. Tessarek wrote:

> On 2018-11-04 02:49, Ryan Schmidt wrote:
>> The way I would approach this is to write a patch file which inserts
>> "@VERSION@" into the file where you want the version to appear. Then
>> you can simply reinplace "s|@VERSION@|${version}|g" to get the real
>> version into the file. That's very simple to understand and it's a
>> pattern we use in many other portfiles to insert the prefix or other
>> portfile variables.
>
> Yes, I thought about that. However, this means I have to create a new
> patch file every time there's an update to that line.

That's correct, and most of the time I think that's a good thing. Time and again we see the problem that a port contains a reinplace, the port is then abandoned and the next person to take up the port doesn't take the time to fully understand all of the portfile, and then the port is updated to a new version of the software in which upstream had changed the line(s) the reinplace was supposed to affect in such a way that the regex no longer matches. This used to be a silent failure, which depending on what the reinplace was supposed to do could cause a build failure (which would be noticed) or the build might still succeed but might be missing a vital change that causes the software not to work correctly; this might not get noticed before being committed. We finally got a warning added to MacPorts so that if a reinplace doesn't do anything you're notified, but since it's just a warning it's often ignored. A failing patchfile, on the other hand, is a hard error. The developer *must* resolve it before they can reasonably commit the port. That's good. It forces them to figure out what the patchfile was changing, figure out how to apply that same fix to the new upstream source or else determine that the patch is no longer needed and remove it.

-Ryan

Reply | Threaded
Open this post in threaded view
|

Re: need help with reinplace

Helmut K. C. Tessarek


On 2018-11-04 22:06, Ryan Schmidt wrote:
> We finally got a warning added to MacPorts so that if a reinplace doesn't do anything you're notified,

On that note...

Warning: reinplace s|-arch [a-z0-9_]+||g didn't change anything in
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_net_curl/curl/work/curl-7.62.0/libcurl.pc
Warning: reinplace s/ '(host_alias|--host)=[^']+'//g didn't change
anything in
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_net_curl/curl/work/curl-7.62.0/curl-config


--
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: need help with reinplace

Ryan Schmidt-24


On Nov 6, 2018, at 15:06, Helmut K. C. Tessarek wrote:

>
> On 2018-11-04 22:06, Ryan Schmidt wrote:
>> We finally got a warning added to MacPorts so that if a reinplace doesn't do anything you're notified,
>
> On that note...
>
> Warning: reinplace s|-arch [a-z0-9_]+||g didn't change anything in
> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_net_curl/curl/work/curl-7.62.0/libcurl.pc
> Warning: reinplace s/ '(host_alias|--host)=[^']+'//g didn't change
> anything in
> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_net_curl/curl/work/curl-7.62.0/curl-config

Yup.

https://trac.macports.org/ticket/53700