Re: Is it time for a libc_fixes library yet?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Jan Stary
On Jul 03 10:07:18, [hidden email] wrote:
> So the last 10 or so tickets in trac seem like they are all for basically the same issue
> - a few missing symbols from libc prior to 10.7.
>
> It is easy enough, but time consuming, to patch each individual source file that is missing the definition (there might be several, also, so you might have to do it multiple times in different files).

The real bug is in the upstream which does not care about portability.
Most often, it is one of the auto* tools doing

        GNU version string => good
        anything else      => bad

without ever checking for the function/feature itself.
http://marc.info/?l=openbsd-ports&m=126805847322995&w=2


> With this library of these missing functions <https://github.com/kencu/snowleopardfixes>,
> all of them from the Apple open source website IIRC, all you need to do is the following
> in the Portfile:
>
> if {${os.platform} eq "darwin" && ${os.major} < 11}  {
> depends_lib-append          port:snowleopardfixes
> configure.ldflags-append   -lsnowleopardfixes
> }
> It could be better named, perhaps libcfixes, as it applies to 10.4 to 10.6, not just SnowLeopard.

Call it kenutils or libcompat (or something); it's not specific to any version of MacOSX
(or any other system, for that matter) and it does not "fix libc".

More importantly, it masks the real problem: upstream will never get fixed.
Tell upstream to provide a compat_*.c for systems that don't have the function.
It's not hard to do - here is how the portable mandoc does it:
http://mdocml.bsd.lv/cgi-bin/cvsweb/

Chances are upstream will not care that their software does not compile
on older MacOSes. But we should still tell them, over and over.
I understand your library is meant precisely to avoid that ...


> Addendum 2
>
> As we have said before (last year) this library could be automatically linked in by base. That would cause trouble with the ports that have already patched in a def, tho.

Please don't.


On Jul 03 22:09:27, [hidden email] wrote:
> Taking a quick look at this library, all implementations seem to be
> covered by GPL-2+.
>
> That makes this library not suitable for many ports, as the result after
> linking would no longer be distributable. It would be better to use
> implementations under a non-copyleft license such BSD, MIT, ISC, etc.

http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/



On Jul 03 16:05:04, [hidden email] wrote:
> I just noticed that the getdelim code manipulates the FILE* directly, so
> that's not allowed in your lib. But it's essentially just filling a buffer
> with data, processing it, and checking feof()/ferror() along the way.
> Nothing complicated you can't reproduce with std api's.

http://mdocml.bsd.lv/cgi-bin/cvsweb/compat_getline.c?rev=1.1&content-type=text/x-cvsweb-markup


On Jul 03 22:51:37, [hidden email] wrote:

> On 3 July 2017 at 22:09, Rainer Müller wrote:
> > On 2017-07-03 19:07, Ken Cunningham wrote:
> >> So the last 10 or so tickets in trac seem like they are all for
> >> basically the same issue - a few missing symbols from libc prior to 10.7.
> >
> > Well, I would take this as a reason to just drop support for such an old
> > OS release... I know, I cannot convince you to do that yet. ;-)
>
> MacPorts is basically the only thing that keeps those OSes useful :)
> I'm grateful for enthusiasts like Ken who make effort to keep the MP
> working there in the best possible way.
> And mostly to Jeremy who made sure that clang works.

Absolutely. I only have a 10.5.8 and a 10.6.8
and MacPorts makes these a lot more useful.

While I appreciate all the effort that goes into keeping software usable on these systems,
I believe this kind of fix should really be pushed upstream.

On Jul 04 03:38:31, [hidden email] wrote:
> Your library offers multiple functions... getline, strndup, etc.
> What happens if a software package provides, for example, a compatibility implementation of getline but not strndup? Can your library be used in this case or will that also "cause trouble", as you put it above?

Ken's library does not check whether the given function
is provided but the underlying system at all (right?).
The cleanest way I have seen is how mandoc does it:

http://mdocml.bsd.lv/cgi-bin/cvsweb/configure
http://mdocml.bsd.lv/cgi-bin/cvsweb/configure.local.example

Yes, that's a hand-written ./configure script; orders of magnitude
smaller than the auto-* produced atrocities.

> I see the port and a portgroup have been added to MacPorts. I would like to request that each port that adopts this port/portgroup also add a comment explaining upstream's stance on the issue. For example, a link to the upstream bug report.

+1

> Perfect would be if I just wrote it into the clang code directly.
> I am almost to the point where I could do that, actually. I know where it would go, I think.

I don't get it. Do you mean to mess with the compiler so that
it implements itself the C functions that the system does not provide?

> You are 100% correct that upstream should do this, and this is far better. I should say over the past year I can't think of a single instance where I saw that upstream actually did add one of these old functions, though. Maybe you know of some cases.

If the upstream is Linux, they mostly don't care as long as glibc has the function.
But they should be told that the software is unportable, even as we try to port it.

> autotools is an ugly thing to learn beyond the defaults.
> And I still don't see how to do this kind of thing by default with cmake.

autotools and cmake should have nothing to do with this.

        Jan


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Kenneth F. Cunningham
>>
>> Perfect would be if I just wrote it into the clang code directly.
>> I am almost to the point where I could do that, actually. I know where it would go, I think.
>
> I don't get it. Do you mean to mess with the compiler so that
> it implements itself the C functions that the system does not provide?


Yep :> That is just exactly what I mean.... clang does stuff like this already to help you out. Adding in various libraries, rearranging header paths, sticking in compiler and linker flags, all to suit to build system and make things work.

funny thing is - in a perfect world - if it was done that way, you'd never even know it happened. Your code would just compile without errors.

But, to be honest, if that was going to happen it probably would have been done by now.
>

> autotools and cmake should have nothing to do with this.
>
> Jan

What i meant was that if it was perceived to be a trivial project for developers to add in a few tests for strnlen, etc, they probably wouldn't be as reluctant as they seem to be to doing it. I think the system is intimidating for most -- it sure is for me.

K


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Jan Stary
On Jul 04 13:47:27, [hidden email] wrote:
> >>
> >> Perfect would be if I just wrote it into the clang code directly.
> >> I am almost to the point where I could do that, actually. I know where it would go, I think.
> >
> > I don't get it. Do you mean to mess with the compiler so that
> > it implements itself the C functions that the system does not provide?
>
>
> Yep :> That is just exactly what I mean.... clang does stuff like this already to help you out. Adding in various libraries, rearranging header paths, sticking in compiler and linker flags, all to suit to build system and make things work.

Does clang currently provide implementations of functions that do not exist,
and link against them if the code calls them? Would you please shre an example?

If so, it seems to be quite a deviation from what a compiler is supposed to do.
If I call a compiler, I expect it to compile the given code, not something that

        1. the system does not provide
        2. the standard library such as libc does not provide
        3. the given code does not contain

Jan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Kenneth F. Cunningham

On 2017-07-05, at 2:31 AM, Jan Stary wrote:

> On Jul 04 13:47:27, [hidden email] wrote:
>>>>
>>>> Perfect would be if I just wrote it into the clang code directly.
>>>> I am almost to the point where I could do that, actually. I know where it would go, I think.
>>>
>>> I don't get it. Do you mean to mess with the compiler so that
>>> it implements itself the C functions that the system does not provide?
>>
>>
>> Yep :> That is just exactly what I mean.... clang does stuff like this already to help you out. Adding in various libraries, rearranging header paths, sticking in compiler and linker flags, all to suit to build system and make things work.
>
> Does clang currently provide implementations of functions that do not exist,
> and link against them if the code calls them? Would you please shre an example?
>
> If so, it seems to be quite a deviation from what a compiler is supposed to do.
> If I call a compiler, I expect it to compile the given code, not something that
>
> 1. the system does not provide
> 2. the standard library such as libc does not provide
> 3. the given code does not contain
>
> Jan
>


So, just spitballing here as it appears this idea is not too popular already, but to answer your question.

IClang does in fact provide some of it's own replacement functions that are either highly optimized, or don't exist on some targets (like the blocks runtime implementation provided to linux). These features of clang are in the compiler runtime library <https://compiler-rt.llvm.org/>.

It appears fairly easy to extend this, but that would not likely be the way I'd choose to do it with my skill level. Instead of that, Clang quite easily can call in support libraries (ffi, omp, all those other ones you see listed as deps).

Very easy would be to have clang list snowleopardfixes (or whatever it might be called to make everyone happy, but I guess that name is sticking for the moment and probably forever now) as a library dependency either on macports or as a subproject of clang itself, and then in the clang compile line handling code do something like this:

check the deployment target
if < 11 {
add in the snowleopardfixes include directory ahead of the system include path
add in a link to -lsnowleopardfixes
}

Then a call to #include string.h would find the snowleopardfixes version of a string.h first, that would call the official system one and then add in the extra couple of definitions for getline, etc. And there would be another one for memmem and I guess a third for the ws* stuff.

Probably could do this without much trouble at all, I think. It would be better, really - provide proper headers and function definitions, fix older systems, no effect on newer ones.

[Most likely could even supply the blocks runtime to 10.4 and 10.5 while we're at it :> , to get ld64-137+ running on PPC -- but I digress.]

But I'm not actually suggesting heading down this road at the moment. Lets see if the snowleopardfixes library makes things better by itself.

Ken


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Jan Stary
On Jul 05 08:28:40, [hidden email] wrote:

>
> On 2017-07-05, at 2:31 AM, Jan Stary wrote:
>
> > On Jul 04 13:47:27, [hidden email] wrote:
> >>>>
> >>>> Perfect would be if I just wrote it into the clang code directly.
> >>>> I am almost to the point where I could do that, actually. I know where it would go, I think.
> >>>
> >>> I don't get it. Do you mean to mess with the compiler so that
> >>> it implements itself the C functions that the system does not provide?
> >>
> >>
> >> Yep :> That is just exactly what I mean.... clang does stuff like this already to help you out. Adding in various libraries, rearranging header paths, sticking in compiler and linker flags, all to suit to build system and make things work.
> >
> > Does clang currently provide implementations of functions that do not exist,
> > and link against them if the code calls them? Would you please shre an example?
> >
> > If so, it seems to be quite a deviation from what a compiler is supposed to do.
> > If I call a compiler, I expect it to compile the given code, not something that
> >
> > 1. the system does not provide
> > 2. the standard library such as libc does not provide
> > 3. the given code does not contain
> >
> > Jan
> >
>
>
> So, just spitballing here as it appears this idea is not too popular already, but to answer your question.
>
> IClang does in fact provide some of it's own replacement functions that are either highly optimized, or don't exist on some targets (like the blocks runtime implementation provided to linux). > These features of clang are in the compiler runtime library <https://compiler-rt.llvm.org/>.

The features described there seem to be quite different
then e.g. implementing a missing strnlen(), which is what your lib does.

The compiler runtime is described as "optional" at
http://clang.llvm.org/get_started.html
so I assume a default build of clang does not use it.
Do the clangs provided by the MP ports use it by default?

> It appears fairly easy to extend this, but that would not likely be the way I'd choose to do it with my skill level. Instead of that, Clang quite easily can call in support libraries (ffi, omp, all those other ones you see listed as deps).

Do any of those reimplement or otherwise substitute
standard libc functions such as strnlen() or getline()?

> Very easy would be to have clang list snowleopardfixes (or whatever it might be called to make everyone happy, but I guess that name is sticking for the moment and probably forever now)

Traditionally, I believe, things like this are called "libcompat".

> as a library dependency either on macports or as a subproject of clang itself,

Didi you talk to the clang upstream about this?
Do they already have a thing like that for some other systems?
I just don't see something as general as compiler suite
having a "snowleopard fixes" subproject.

> and then in the clang compile line handling code do something like this:
> check the deployment target
> if < 11 {
> add in the snowleopardfixes include directory ahead of the system include path
> add in a link to -lsnowleopardfixes
> }

Perhaps it is a point of view, but I don't think it's
the compiler's job to implement a getline(3) itself.

> Then a call to #include string.h would find the snowleopardfixes version of a string.h first, that would call the official system one and then add in the extra couple of definitions for getline, etc. And there would be another one for memmem and I guess a third for the ws* stuff.

Please don't. It is the source's job to have a compat_getline()
if it is to be ported to systems that don't have getline(),
or the porter's job to provide that as a patch.

This is a UNIX system. Tweaking the compiler and doing clever hacks
around standard string.h seems like the opposite of "keep it simple".

        Jan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Kenneth F. Cunningham
This is a UNIX system. Tweaking the compiler and doing clever hacks
around standard string.h seems like the opposite of "keep it simple”.


OK. 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Kenneth F. Cunningham
In reply to this post by Jan Stary
The compiler runtime is described as "optional" at
http://clang.llvm.org/get_started.html
so I assume a default build of clang does not use it.
Do the clangs provided by the MP ports use it by default?


All versions of clang on all version of macOS use the clang_rt.

You can see a very interesting view of what clang does in the background for you by compiling a simple “hello.c” application with some verbose flags to the compiler and linker like this:

clang -v -Wl,-v hello.c

You might be surprised by all the things done in the background to make it work for you. 

K
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Dave Horsfall
On Wed, 5 Jul 2017, Ken Cunningham wrote:

> clang -v -Wl,-v hello.c
>
> You might be surprised by all the things done in the background to make
> it work for you. 

It's even more interesting with "-Wall"; apart from all the excruciating
detail it also reports:

    ignoring nonexistent directory "/usr/local/include"

I guess I haven't had to install anything there yet.

--
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it time for a libc_fixes library yet?

Richard L. Hamilton-3
In reply to this post by Jan Stary

> On Jul 5, 2017, at 05:31, Jan Stary <[hidden email]> wrote:
>
> On Jul 04 13:47:27, [hidden email] wrote:
>>>>
>>>> Perfect would be if I just wrote it into the clang code directly.
>>>> I am almost to the point where I could do that, actually. I know where it would go, I think.
>>>
>>> I don't get it. Do you mean to mess with the compiler so that
>>> it implements itself the C functions that the system does not provide?
>>
>>
>> Yep :> That is just exactly what I mean.... clang does stuff like this already to help you out. Adding in various libraries, rearranging header paths, sticking in compiler and linker flags, all to suit to build system and make things work.
>
> Does clang currently provide implementations of functions that do not exist,
> and link against them if the code calls them? Would you please shre an example?
>
> If so, it seems to be quite a deviation from what a compiler is supposed to do.
> If I call a compiler, I expect it to compile the given code, not something that
>
> 1. the system does not provide
> 2. the standard library such as libc does not provide
> 3. the given code does not contain
>
> Jan
>
Depends on where it appears.  If it's added to a .o file, I'd be worried; but if it only appears when creating a full executable, then it's just another flavor of added runtime support, perhaps to bridge the difference between what the compilation environment is attempting to support in terms of source, and what the system offers in terms of needed system calls or library functions.  And in the case of providing a builtin inline implementation of  a well-known function if nothing is taking its address (so there's no worry of possible later use via a pointer-to-function), then even a .o file might have something added to it whose provenance may not be entirely obvious.


signature.asc (859 bytes) Download Attachment
Loading...