portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ken Cunningham
I noticed this recently "fixing" some ports that don't use a configure step.

During the run of portconfigure.tcl, various things (sdkroot) might be tested, and the appropriate values appended to the ENV variables.

But these things don't seem to come out in the configure.variables, like I would have expected them to.

Is this desired behaviour?

Or should portconfigure flesh out the configure.variables at the same time as it's appending to the ENV variables (like I would have expected)?

(Please tell me I'm missing some well-known step that everyone else knows about but I don't :> )

Ken
Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ryan Schmidt-24


On Nov 6, 2018, at 14:48, Ken Cunningham wrote:

> I noticed this recently "fixing" some ports that don't use a configure step.
>
> During the run of portconfigure.tcl, various things (sdkroot) might be tested, and the appropriate values appended to the ENV variables.
>
> But these things don't seem to come out in the configure.variables, like I would have expected them to.
>
> Is this desired behaviour?
>
> Or should portconfigure flesh out the configure.variables at the same time as it's appending to the ENV variables (like I would have expected)?
>
> (Please tell me I'm missing some well-known step that everyone else knows about but I don't :> )

Can you give a specific example?

Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ken Cunningham


On Nov 6, 2018, at 1:54 PM, Ryan Schmidt <[hidden email]> wrote:



On Nov 6, 2018, at 14:48, Ken Cunningham wrote:

I noticed this recently "fixing" some ports that don't use a configure step.

During the run of portconfigure.tcl, various things (sdkroot) might be tested, and the appropriate values appended to the ENV variables.

But these things don't seem to come out in the configure.variables, like I would have expected them to.

Is this desired behaviour?

Or should portconfigure flesh out the configure.variables at the same time as it's appending to the ENV variables (like I would have expected)?

(Please tell me I'm missing some well-known step that everyone else knows about but I don't :> )

Can you give a specific example?



OK. Let’s say to you want to build lz4 against an SDK. I defined the SDK to 10.13. To make the Portfile show the configure environment, I had to give it a sham configure phase by doing this tiny change:

#use_configure       no
configure.cmd       /usr/bin/true



When you go to build the port, sudo port -d build lz4, you see the configure environment variables are set up right, wth the SDK specified in several places, as it should be:

DEBUG: Preferred compilers: clang macports-clang-5.0 macports-clang-4.0
DEBUG: Using compiler 'Xcode Clang'
DEBUG: Executing org.macports.configure (lz4)
DEBUG: Environment: 
CC='/usr/bin/clang'
CC_PRINT_OPTIONS='YES'
CC_PRINT_OPTIONS_FILE='/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/.CC_PRINT_OPTIONS'
CFLAGS='-pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
CPATH='/opt/universal/include'
CPPFLAGS='-I/opt/universal/include -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk'
CXX='/usr/bin/clang++'
CXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
F77FLAGS='-m64'
F90FLAGS='-pipe -Os -m64'
FCFLAGS='-pipe -Os -m64'
FFLAGS='-pipe -Os'
INSTALL='/usr/bin/install -c'
LDFLAGS='-L/opt/universal/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
LIBRARY_PATH='/opt/universal/lib'
MACOSX_DEPLOYMENT_TARGET='10.13'
OBJC='/usr/bin/clang'
OBJCFLAGS='-pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
OBJCXX='/usr/bin/clang++'
OBJCXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
Executing:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/true --prefix=/opt/universal 
DEBUG: system:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/true --prefix=/opt/universal 
DEBUG: Privilege de-escalation not attempted as not running as root.
DEBUG: build phase started at Wed Nov  7 08:51:33 PST 2018



However, when the port is actually being built, the build args, which should be setup correctly by this:

    build.args-append       CFLAGS="${configure.cflags} [get_canonical_archflags cc]" \
                            CXXFLAGS="${configure.cxxflags} [get_canonical_archflags cxx]"


don’t actually show up on the build line with all the ${configure.cxxflags} as above. Instead, you just get the “default” values from the top of portconfigure.tcl:

Executing:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/make -j16 -w all CC=/usr/bin/clang CXX=/usr/bin/clang++ PREFIX=/opt/universal CFLAGS="-Os -arch x86_64" CXXFLAGS="-Os -stdlib=libc++ -arch x86_64" 
DEBUG: system:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/make -j16 -w all CC=/usr/bin/clang CXX=/usr/bin/clang++ PREFIX=/opt/universal CFLAGS="-Os -arch x86_64" CXXFLAGS="-Os -stdlib=libc++ -arch x86_64" 



The lz4 portfile has manually added the -stdlib=llibc++ in the Portfile, but it should not need to do that as it is already being done by portconfigure.tcl — but the modifications made by portconfigure.tcl to the CXXFLAGS are not making it into ${configure.cxxflags}, at least at the point where the build arguments are being set up.




I hope this is a sufficiently clean example.

Best, Ken






Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ryan Schmidt-24
On Nov 7, 2018, at 11:00, Ken Cunningham wrote:

> On Nov 6, 2018, at 1:54 PM, Ryan Schmidt wrote:
>
>> On Nov 6, 2018, at 14:48, Ken Cunningham wrote:
>>
>>> I noticed this recently "fixing" some ports that don't use a configure step.
>>>
>>> During the run of portconfigure.tcl, various things (sdkroot) might be tested, and the appropriate values appended to the ENV variables.
>>>
>>> But these things don't seem to come out in the configure.variables, like I would have expected them to.
>>>
>>> Is this desired behaviour?
>>>
>>> Or should portconfigure flesh out the configure.variables at the same time as it's appending to the ENV variables (like I would have expected)?
>>>
>>> (Please tell me I'm missing some well-known step that everyone else knows about but I don't :> )
>>
>> Can you give a specific example?
>>
>
>
> OK. Let’s say to you want to build lz4 against an SDK. I defined the SDK to 10.13. To make the Portfile show the configure environment, I had to give it a sham configure phase by doing this tiny change:
>
> #use_configure       no
> configure.cmd       /usr/bin/true
>
>
>
> When you go to build the port, sudo port -d build lz4, you see the configure environment variables are set up right, wth the SDK specified in several places, as it should be:
>
> DEBUG: Preferred compilers: clang macports-clang-5.0 macports-clang-4.0
> DEBUG: Using compiler 'Xcode Clang'
> DEBUG: Executing org.macports.configure (lz4)
> DEBUG: Environment:
> CC='/usr/bin/clang'
> CC_PRINT_OPTIONS='YES'
> CC_PRINT_OPTIONS_FILE='/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/.CC_PRINT_OPTIONS'
> CFLAGS='-pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> CPATH='/opt/universal/include'
> CPPFLAGS='-I/opt/universal/include -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk'
> CXX='/usr/bin/clang++'
> CXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> F77FLAGS='-m64'
> F90FLAGS='-pipe -Os -m64'
> FCFLAGS='-pipe -Os -m64'
> FFLAGS='-pipe -Os'
> INSTALL='/usr/bin/install -c'
> LDFLAGS='-L/opt/universal/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> LIBRARY_PATH='/opt/universal/lib'
> MACOSX_DEPLOYMENT_TARGET='10.13'
> OBJC='/usr/bin/clang'
> OBJCFLAGS='-pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> OBJCXX='/usr/bin/clang++'
> OBJCXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> Executing:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/true --prefix=/opt/universal
> DEBUG: system:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/true --prefix=/opt/universal
> DEBUG: Privilege de-escalation not attempted as not running as root.
> DEBUG: build phase started at Wed Nov  7 08:51:33 PST 2018
>
>
>
> However, when the port is actually being built, the build args, which should be setup correctly by this:
>
>     build.args-append       CFLAGS="${configure.cflags} [get_canonical_archflags cc]" \
>                             CXXFLAGS="${configure.cxxflags} [get_canonical_archflags cxx]"
>
>
> don’t actually show up on the build line with all the ${configure.cxxflags} as above. Instead, you just get the “default” values from the top of portconfigure.tcl:
>
> Executing:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/make -j16 -w all CC=/usr/bin/clang CXX=/usr/bin/clang++ PREFIX=/opt/universal CFLAGS="-Os -arch x86_64" CXXFLAGS="-Os -stdlib=libc++ -arch x86_64"
> DEBUG: system:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/make -j16 -w all CC=/usr/bin/clang CXX=/usr/bin/clang++ PREFIX=/opt/universal CFLAGS="-Os -arch x86_64" CXXFLAGS="-Os -stdlib=libc++ -arch x86_64"
>
>
>
> The lz4 portfile has manually added the -stdlib=llibc++ in the Portfile, but it should not need to do that as it is already being done by portconfigure.tcl — but the modifications made by portconfigure.tcl to the CXXFLAGS are not making it into ${configure.cxxflags}, at least at the point where the build arguments are being set up.
>
>
>
>
> I hope this is a sufficiently clean example.

Yes. And the reason why I asked for an example is that the answer would be different for different options!

For the -stdlib=libc++ flag, portconfigure.tcl does in fact add that into configure.cxxflags for you when needed (i.e. when the compiler is clang++) and goes to some lengths (using traces) to ensure that it cannot be removed, except via the approved mechanism of changing configure.cxx_stdlib. There's no need for the lz4 port to add this manually so I've removed that code. If a port needs the flag in configure.ldflags, then the port needs to add it there manually, but that's not the case for lz4.

For SDK flags, you're right, they're not added to the configure.*flags variables; instead, they're added directly to the environment by portconfigure::configure_main when they're needed.

The reason for this probably has to do with the fact that originally the decision about whether or not to use SDK flags was based on whether or not we were building universal. To build universal on Tiger PowerPC, you need to use the 10.4u SDK, otherwise you don't.

The way that the default universal variant gets added is as follows: After the entire portfile has been processed, MacPorts base defines an empty default universal variant if ${universal_variant} is "yes". The default value of universal_variant is ${use_configure}, and the default value of use_configure is "yes".

The addition of -arch flags is handled the same way, for the same reason -- we need to add different -arch flags depending on whether we're universal or not, and we won't know for sure if the port will have a universal variant until after the entire portfile has been processed.

This way of defining the default universal variant certainly causes some problems which have been pointed out before. For example, if you want to add universal support to a port that uses "use_configure no", you have to manually add the -arch flags using [get_canonical_archflags], but you have to make sure the universal variant exists before then. You might think you could just "universal_variant yes" prior to calling it, but that doesn't work because the default universal variant won't get created until after the portfile is processed. Instead you have to manually create an empty universal variant ("variant universal {}"). Alternately, you could restrict yourself to using [get_canonical_archflags] in a phase (for example in a pre-build block), since the phases run after the portfile is processed, but that makes the Portfile code uglier and seems to make it impossible to copy values between phases, such as assign destroot.args the value of build.args, which I've done in many portfiles and is very convenient to be able to do.

How could we improve this? What if MacPorts defined the empty default universal variant before processing the portfile (on systems where we can build universal)? We would then need a way of deleting the variant in the event that "universal_variant no" (or "use_configure no") is encountered in the portfile. I don't think MacPorts base currently has a way to delete a variant. But maybe that wouldn't be too hard to add.

If we could do that, then we could modify the way the -arch flags and SDK flags are added. We could handle them similar to how configure.cxx_stdlib is handled. That should do what you want.

But there are a *lot* of ports out there already that handle this manually. There's a good chance that by fixing this in base, we'll end up breaking ports or portgroups. Which could then be fixed. But we should be very careful.

I see for example that the cmake portgroup specifies the SDK root using -DCMAKE_OSX_SYSROOT and the architectures in -DCMAKE_OSX_ARCHITECTURES. It doesn't want them mixed in with the CXXFLAGS. We should consider that when making changes. For example, we may want to have separate variables like configure.archflags, configure.sdk_ldflags, configure.sdk_cflags, which we put in configure.cflags, configure.ldflags, etc. by default but which the cmake portgroup could clear out.


Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ken Cunningham

On 2018-11-07, at 1:27 PM, Ryan Schmidt wrote:

Thanks for the detailed explanation. I do understand what you mean about +universal. Just to point out that this port is not actually building universal here, it's just building normally.

What I was expecting would be that portconfigure.tcl would do something like this at the end of the process

foreach item in CXXFLAGS
    configure.cxxflags-append item
end

so then ${configure.cxxflags} would exactly match CXXFLAGS in the environment.

Then a whole lot of Portfile shenanigans could go away, I think.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ryan Schmidt-24


On Nov 7, 2018, at 17:28, Ken Cunningham wrote:

> Thanks for the detailed explanation. I do understand what you mean about +universal. Just to point out that this port is not actually building universal here, it's just building normally.

I'm sure. But as I explained, SDK flags can vary based on whether one is building universal or not. That's why the SDK flags is not run until we know for sure whether or not the portfile has a universal variant.


> What I was expecting would be that portconfigure.tcl would do something like this at the end of the process
>
> foreach item in CXXFLAGS
>    configure.cxxflags-append item
> end
>
> so then ${configure.cxxflags} would exactly match CXXFLAGS in the environment.
>
> Then a whole lot of Portfile shenanigans could go away, I think.

By "the end of the process" you mean at the end of portconfigure::configure_main? If so, there are two problems with that suggestion.

First, the value of e.g. ${configure.cxxflags} that you see in the global part of the portfile would be different from the value you would see in a phase. So the following two methods would see different results:


build.args-append CXXFLAGS="${configure.cxxflags}"


pre-build {
    build.args-append CXXFLAGS="${configure.cxxflags}"
}


That might be confusing.

And since the flags would hypothetically be added at the end of the configure phase, what would happen if the user runs "sudo port configure" separately from "sudo port build"? The answer is that the revised cxxflags would not be seen in the build phase or later. However they would be seen if "sudo port build" (or destroot or install) is run without "sudo port configure" having been run separately before. That would definitely be confusing.

Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ken Cunningham
OK. I guess the answer to my question then really is that it is apparently purposelful and desired behavoir that

configure.cxxflags is not set up the same as CXXFLAGS

and  likewise for all the others as well. That concept is a new one for me, but I'll try to wrap my head arond this.

Thanks,

Ken


> On Nov 8, 2018, at 04:38, Ryan Schmidt <[hidden email]> wrote:
>
>
>
>> On Nov 7, 2018, at 17:28, Ken Cunningham wrote:
>>
>> Thanks for the detailed explanation. I do understand what you mean about +universal. Just to point out that this port is not actually building universal here, it's just building normally.
>
> I'm sure. But as I explained, SDK flags can vary based on whether one is building universal or not. That's why the SDK flags is not run until we know for sure whether or not the portfile has a universal variant.
>
>
>> What I was expecting would be that portconfigure.tcl would do something like this at the end of the process
>>
>> foreach item in CXXFLAGS
>>   configure.cxxflags-append item
>> end
>>
>> so then ${configure.cxxflags} would exactly match CXXFLAGS in the environment.
>>
>> Then a whole lot of Portfile shenanigans could go away, I think.
>
> By "the end of the process" you mean at the end of portconfigure::configure_main? If so, there are two problems with that suggestion.
>
> First, the value of e.g. ${configure.cxxflags} that you see in the global part of the portfile would be different from the value you would see in a phase. So the following two methods would see different results:
>
>
> build.args-append CXXFLAGS="${configure.cxxflags}"
>
>
> pre-build {
>    build.args-append CXXFLAGS="${configure.cxxflags}"
> }
>
>
> That might be confusing.
>
> And since the flags would hypothetically be added at the end of the configure phase, what would happen if the user runs "sudo port configure" separately from "sudo port build"? The answer is that the revised cxxflags would not be seen in the build phase or later. However they would be seen if "sudo port build" (or destroot or install) is run without "sudo port configure" having been run separately before. That would definitely be confusing.
>
Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ken Cunningham

On 2018-11-08, at 8:16 AM, Ken Cunningham wrote:

> OK. I guess the answer to my question then really is that it is apparently purposelful and desired behavoir that
>
> configure.cxxflags is not set up the same as CXXFLAGS
>
> and  likewise for all the others as well. That concept is a new one for me, but I'll try to wrap my head arond this.
>
> Thanks,
>
> Ken

I will say, though -- look at all the shenanigans in

graphics/freeimage/Portfile

pretty much all of which would not be needed if portconfigure did poplulate the configure.NNflags

it has almost completely reduplicated the portconfigure.tcl  logic.

That's more or less what every non-autotools non-cmake port would have to do to get full compliance with isysroot, syslibroot, etc etc etc.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

Ryan Schmidt-24
In reply to this post by Ken Cunningham


On Nov 8, 2018, at 10:16, Ken Cunningham wrote:

> OK. I guess the answer to my question then really is that it is apparently purposelful and desired behavoir that
>
> configure.cxxflags is not set up the same as CXXFLAGS
>
> and  likewise for all the others as well. That concept is a new one for me, but I'll try to wrap my head arond this.

It's only purposeful and desired to the extent that it's what the developers who implemented it came up with for the problem they were trying to solve, given the constraints and difficulties I explained earlier. Certainly the method has its drawbacks, as you've demonstrated. We can certainly explore, as I started to in my previous messages, changing it.