Quantcast

build progress bar

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

build progress bar

René J.V. Bertin
Hi,

I've been tinkering a bit with a build progress bar feature that uses the information certain build systems provide.
CMake for instance generates Makefiles that print regular "[xyz%] " tags in their regular output which can be processed with a simple scan in ui_message (macports.tcl). One should be able to use `macports::ui_options(progress_generic)` from there, but that feature would have to be initialised. I've failed to do that from portbuild.tcl as well as from `command_exec` in portutil.tcl .

What have I been missing to access that variable from those places?


On a more general note I was thinking of supporting this kind of progress reporting via a callback hook which can then be set by (say) the CMake portgroup. I'd love to have progress feedback when not doing a verbose build but not enough to try and hack something generic together that works across most build systems even if they don't provide progress (percentage or "X of of Y") info.
IIRC we discussed a feature like this a while back; has anyone been giving it any thought?

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Ryan Schmidt-24

> On Feb 18, 2017, at 12:45, René J.V. Bertin <[hidden email]> wrote:
>
> Hi,
>
> I've been tinkering a bit with a build progress bar feature that uses the information certain build systems provide.
> CMake for instance generates Makefiles that print regular "[xyz%] " tags in their regular output which can be processed with a simple scan in ui_message (macports.tcl). One should be able to use `macports::ui_options(progress_generic)` from there, but that feature would have to be initialised. I've failed to do that from portbuild.tcl as well as from `command_exec` in portutil.tcl .
>
> What have I been missing to access that variable from those places?
>
>
> On a more general note I was thinking of supporting this kind of progress reporting via a callback hook which can then be set by (say) the CMake portgroup. I'd love to have progress feedback when not doing a verbose build but not enough to try and hack something generic together that works across most build systems even if they don't provide progress (percentage or "X of of Y") info.
> IIRC we discussed a feature like this a while back; has anyone been giving it any thought?
>
> R.

Thanks for working on this.

The problem is that it only applies to cmake-generated makefiles and certain other one-off makefiles. But maybe a progress bar for some builds is better than no build progress bar ever.

Providing a way for portfiles and portgroups to indicate what, if any, regexp matches progress information in the build output is probably a good way to go. There is an additional complication that some types of builds may need special handling. For example, a port building with cmake and muniversal will go from 0% to 100% for the first architecture, then go from 0% to 100% for the second architecture; this detail should be hidden from the user behind a single combined progress bar.

The ticket for this issue is https://trac.macports.org/ticket/15939 . Clemens had some thoughts there for how it might be done for other ports whose build systems don't provide this information, but it would involve maintaining an online database.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Saturday February 18 2017 17:27:45 Ryan Schmidt wrote:

> The problem is that it only applies to cmake-generated makefiles and certain other one-off makefiles. But maybe a progress bar for some builds is better than no build progress bar ever.

Yeah. It might work for any project that uses ninja, too.

> For example, a port building with cmake and muniversal will go from 0% to 100% for the first architecture, then go from 0% to 100% for the second architecture; this detail should be hidden from the user behind a single combined progress bar.

Or the muniversal portgroup could indicate what architecture it's building for. I've already hacked my copy to do that because it also provides a periodic wake-up call to ask myself if I really need that port installed +universal. That's a different issue of course.

> The ticket for this issue is https://trac.macports.org/ticket/15939 . Clemens had some thoughts there for how it might be done for other ports whose build systems don't provide this information, but it would involve maintaining an online database.

Ah yes. In my memory someone suggested using the number of files, but the number of output lines would be a reasonable indicator if you know how many there are.

If we go for an approach that relies on ports (or portgroups) providing the required information it should also be possible to provide the number of expected lines that way, removing the need for an additional online database. Something like

{{{
build.progress.expected_lines 1000
}}}

should be all that's needed to activate a progressbar mechanism based on this generic principle.

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Ryan Schmidt-24

On Feb 18, 2017, at 17:55, René J.V. Bertin wrote:

> On Saturday February 18 2017 17:27:45 Ryan Schmidt wrote:
>
>> The problem is that it only applies to cmake-generated makefiles and certain other one-off makefiles. But maybe a progress bar for some builds is better than no build progress bar ever.
>
> Yeah. It might work for any project that uses ninja, too.
>
>> For example, a port building with cmake and muniversal will go from 0% to 100% for the first architecture, then go from 0% to 100% for the second architecture; this detail should be hidden from the user behind a single combined progress bar.
>
> Or the muniversal portgroup could indicate what architecture it's building for. I've already hacked my copy to do that because it also provides a periodic wake-up call to ask myself if I really need that port installed +universal. That's a different issue of course.

The muniversal portgroup used to say "building for i386", "building for x86_64", but this was removed, perhaps because it was thought to be confusing that some ports were printing this information and others weren't.


>> The ticket for this issue is https://trac.macports.org/ticket/15939 . Clemens had some thoughts there for how it might be done for other ports whose build systems don't provide this information, but it would involve maintaining an online database.
>
> Ah yes. In my memory someone suggested using the number of files, but the number of output lines would be a reasonable indicator if you know how many there are.
>
> If we go for an approach that relies on ports (or portgroups) providing the required information it should also be possible to provide the number of expected lines that way, removing the need for an additional online database. Something like
>
> {{{
> build.progress.expected_lines 1000
> }}}
>
> should be all that's needed to activate a progressbar mechanism based on this generic principle.

Nobody's going to remember to update that number when the update a port to a new version number.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Saturday February 18 2017 17:57:04 Ryan Schmidt wrote:

> The muniversal portgroup used to say "building for i386", "building for x86_64", but this was removed, perhaps because it was thought to be confusing that some ports were printing this information and others weren't.

I never saw that, but that's exactly what it prints for me now.

> Nobody's going to remember to update that number when the update a port to a new version number.

Maybe, but you can say the same thing about updating the database.

Either way that's all a bit premature, let's first see how we can get a basic version to work. I've got the scan logic working but am still looking for inspiration how to move it to the cmake PortGroup and inject it into base from there.
But above all I need to understand how to get at the port client's progressbar functions from macports.tcl .

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Ryan Schmidt-24
On Feb 18, 2017, at 19:46, René J.V. Bertin <[hidden email]> wrote:
>
>> On Saturday February 18 2017 17:57:04 Ryan Schmidt wrote:
>>
>> The muniversal portgroup used to say "building for i386", "building for x86_64", but this was removed, perhaps because it was thought to be confusing that some ports were printing this information and others weren't.
>
> I never saw that, but that's exactly what it prints for me now.

I... remembered that being removed. But it hasn't been. Huh.


>> Nobody's going to remember to update that number when the update a port to a new version number.
>
> Maybe, but you can say the same thing about updating the database.

The idea was that the buildbot would update the database automatically after each build.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Saturday February 18 2017 21:59:25 Ryan Schmidt wrote:

>> I never saw that, but that's exactly what it prints for me now.
>
>I... remembered that being removed. But it hasn't been. Huh.

Huh indeed. I remember hacking it in myself ...!

>The idea was that the buildbot would update the database automatically after each build.

I just closed the ticket window but I distinctly remember reading something about users opting in to provide the information. Maybe that applied only to ports not handled by the bots.
The one advantage of letting bots define this information is that you'd get OS version specific counts which is probably necessary anyway. But I don't really like the idea that each `port build` would query a remote server; I'd much prefer for the database to be installed locally and updated through `port selfupdate`.

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Ryan Schmidt-24

> On Feb 19, 2017, at 04:10, René J.V. Bertin <[hidden email]> wrote:
>
> On Saturday February 18 2017 21:59:25 Ryan Schmidt wrote:
>
>>> I never saw that, but that's exactly what it prints for me now.
>>
>> I... remembered that being removed. But it hasn't been. Huh.
>
> Huh indeed. I remember hacking it in myself ...!
>
>> The idea was that the buildbot would update the database automatically after each build.
>
> I just closed the ticket window but I distinctly remember reading something about users opting in to provide the information. Maybe that applied only to ports not handled by the bots.
> The one advantage of letting bots define this information is that you'd get OS version specific counts which is probably necessary anyway. But I don't really like the idea that each `port build` would query a remote server; I'd much prefer for the database to be installed locally and updated through `port selfupdate`.

Yes, selfupdate is another reasonable mechanism for distributing that information. It could even be just another field in the portindex.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Clemens Lang-2
In reply to this post by René J.V. Bertin
Hi,

On Sun, Feb 19, 2017 at 11:10:50AM +0100, René J.V. Bertin wrote:
> > The idea was that the buildbot would update the database
> > automatically after each build.

Not only, because using only the buildbot would not give us any
information about variants, which could significantly change the number
of printed lines.

> I just closed the ticket window but I distinctly remember reading
> something about users opting in to provide the information. Maybe that
> applied only to ports not handled by the bots.

No, my idea originally was to do that for all ports, even though most
people probably would not build ports already built by our buildbots
(but then the line count wouldn't matter for them anyway, unless the
port was not distributable).

> The one advantage of letting bots define this information is that
> you'd get OS version specific counts which is probably necessary
> anyway.

We could always include the OS version in a request, e.g. using a hash
of OS version + separator + port name + variants.

> But I don't really like the idea that each `port build` would query a
> remote server; I'd much prefer for the database to be installed
> locally and updated through `port selfupdate`.

Syncing such a database via selfupdate may leave you in a
chicken-and-egg situation where at the time the portindex is generated
and you learn about the new port, line number info is not available most
of the time. In a crowd-sourced (or buildbot-based) approach, I would
expect line number information to show up a couple of hours after a port
has appeared in portindex, by which time quite a few people may already
have installed it.

I agree that privacy is a concern, which is why this should be opt-in.
Maybe we could make the requests opaque enough so that it's not clear
which port is being installed, or just update a database of all ports
before an attempted installation.

--
Clemens
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Monday February 20 2017 21:41:48 Clemens Lang wrote:

Hi,

> On Sun, Feb 19, 2017 at 11:10:50AM +0100, René J.V. Bertin wrote:
> > > The idea was that the buildbot would update the database
> > > automatically after each build.

I don't think I was the one who wrote that :)


> expect line number information to show up a couple of hours after a port
> has appeared in portindex, by which time quite a few people may already
> have installed it.

Speaking of "quite a few people" who provide line number info: how do you consider handling if their information doesn't match, for instance because not everyone is using exactly the same compiler and compiler options?

And the other thing that was mentioned, incremental builds. Do you also see a clever solution to the challenge those pose?

> I agree that privacy is a concern, which is why this should be opt-in.

Evidently, but it's not the opt-in sending of information that I find a bit frown-worthy. You either decide to participate or you don't. It's the fact that you'd need to query a remote server each time you start a build if you want to have progress information.

Independent of that there's also the fact that ultimately the progress reporting mechanism will hook into the existing code at a level where it gets executed for every line that's being logged, so there's a bit of a performance concern.

Let me ask about my blocker once more: how do I get at macports::ui_options(progress_generic) from macports.tcl?

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Clemens Lang-2
On Mon, Feb 20, 2017 at 10:17:05PM +0100, René J.V. Bertin wrote:
> I don't think I was the one who wrote that :)

Hence the double quotation markers as opposed to single ones where I was
quoting you.

> Speaking of "quite a few people" who provide line number info: how do
> you consider handling if their information doesn't match, for instance
> because not everyone is using exactly the same compiler and compiler
> options?

Average and/or median would probably be close enough in most cases. We
could also add compiler versions to the mix.

> And the other thing that was mentioned, incremental builds. Do you
> also see a clever solution to the challenge those pose?

No, unfortunately not. I don't think line counts work for incremental
builds at all, but fortunately they wouldn't be a very common use-case.

> Evidently, but it's not the opt-in sending of information that I find
> a bit frown-worthy. You either decide to participate or you don't.
> It's the fact that you'd need to query a remote server each time you
> start a build if you want to have progress information.

This could be solved by updating a database once before all builds.

> Independent of that there's also the fact that ultimately the progress
> reporting mechanism will hook into the existing code at a level where
> it gets executed for every line that's being logged, so there's a bit
> of a performance concern.

We already run C code for every line of output produced by a port's
build system. Of course calling back into Tcl for each line isn't
possible, but keeping track of the last call and invoking a Tcl callback
every second, for example, is entirely possible.

> Let me ask about my blocker once more: how do I get at
> macports::ui_options(progress_generic) from macports.tcl?

See the rev-upgrade code, which is in macports.tcl and draws a progress
bar.

--
Clemens
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Tuesday February 21 2017 19:52:27 Clemens Lang wrote:

>> And the other thing that was mentioned, incremental builds. Do you
>> also see a clever solution to the challenge those pose?
>
>No, unfortunately not. I don't think line counts work for incremental
>builds at all, but fortunately they wouldn't be a very common use-case.

I'm pretty sure they won't, but I think that incremental builds might be more common than you think among the users who are esp. interested in progress reporting.

The progress info provided by Ninja and CMake+Make works in those cases.

>This could be solved by updating a database once before all builds.

That would work, but only for batch builds.

>We already run C code for every line of output produced by a port's
>build system. Of course calling back into Tcl for each line isn't
>possible, but keeping track of the last call and invoking a Tcl callback
>every second, for example, is entirely possible.

That's what I thought too initially, but as far as I've found the Pextlib/C version of ui_info and family do exactly that: they call the Tcl versions. That's why my current barebones prototype that just prints "XX% done" works. Am I overlooking something? It does make sense, in the end.

>> Let me ask about my blocker once more: how do I get at
>> macports::ui_options(progress_generic) from macports.tcl?
>
>See the rev-upgrade code, which is in macports.tcl and draws a progress
>bar.

My bad, macports.tcl has access to macports::ui_options. The blocker is that I need to call the start and finish methods from another file, and I haven't managed to access macports::ui_options from there.

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Clemens Lang-2
Hi,

----- On 21 Feb, 2017, at 20:29, René J.V. Bertin [hidden email] wrote:

> I'm pretty sure they won't, but I think that incremental builds might be more
> common than you think among the users who are esp. interested in progress
> reporting.

There are two cases here, reporting that there is some progress (where lines
can still be used) and drawing a progress bar (where line count does not work
in incremental builds). Regardless, incremental builds are uncommon for our
users aside from "power users".


> The progress info provided by Ninja and CMake+Make works in those cases.

Sure, I'm not saying we shouldn't use the progress info provided by CMake or
Ninja, I'm saying we could also attempt to provide progress information where
we don't have Ninja or CMake.

> > We already run C code for every line of output produced by a port's
> > build system. Of course calling back into Tcl for each line isn't
> > possible, but keeping track of the last call and invoking a Tcl callback
> > every second, for example, is entirely possible.
>
> That's what I thought too initially, but as far as I've found the Pextlib/C
> version of ui_info and family do exactly that: they call the Tcl versions.

Well, the magic happens in

  https://github.com/macports/macports-base/blob/master/src/pextlib1.0/system.c#L303

so it seems we do call to Tcl once for each line generated by a build system.
That's probably buffered, though, so those calls happens every few KB of
output.

This is also the place where progress reporting could be added.


> That's why my current barebones prototype that just prints "XX% done" works. Am
> I overlooking something? It does make sense, in the end.

Yes. Probably still not a good idea to put progress guessing into the
implementation of ui_info, though.


> My bad, macports.tcl has access to macports::ui_options. The blocker is that I
> need to call the start and finish methods from another file, and I haven't
> managed to access macports::ui_options from there.

Which file? Why do you need to call start and finish from different files? That
sounds like a problem waiting to happen if you do the progress setting from
macports.tcl but the start and finish from somewhere else.

--
Clemens Lang
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Wednesday February 22 2017 14:57:17 Clemens Lang wrote:

Hi,


> in incremental builds). Regardless, incremental builds are uncommon for our
> users aside from "power users".

True, my point is that those power users are probably more likely to appreciate progress reporting. The average Joe or Jane user probably launches an upgrade or install and goes on to do something else.
In fact, I myself would probably be more interested in a progress bar during an incremental build because more likely to sit around and wait for it to finish. I tend to plan expensive builds I know will take too long to wait for, and in that case checking the logfile from time to time is usually good enough.


> Well, the magic happens in
>
>   https://github.com/macports/macports-base/blob/master/src/pextlib1.0/system.c#L303
>
> so it seems we do call to Tcl once for each line generated by a build system.

Yep, so I wasn't mistaken in my analysis. Actually, the system function hands off the output string to ui_info(), which ultimately leads to a C version of ui_message being called:
https://github.com/macports/macports-base/blob/master/src/pextlib1.0/Pextlib.c#L113


> That's probably buffered, though, so those calls happens every few KB of
> output.

That's the impression I have from doing `tail -f` on log files.

> This is also the place where progress reporting could be added.
...
> Yes. Probably still not a good idea to put progress guessing into the
> implementation of ui_info, though.

proc ui_message is where I've put it because that's the function that gets called from Pextlib.

Putting it there instead of in C code means you could also do (line-number based) progress reporting for long operations that are written in pure Tcl.

> Which file? Why do you need to call start and finish from different files? That
> sounds like a problem waiting to happen if you do the progress setting from
> macports.tcl but the start and finish from somewhere else.

Yes, but take the case of the build phase. proc ui_message has no way of knowing when that phase starts unless it starts tracking changes in macports::current_phase . And that's probably something we'd want to avoid to prevent unnecessary overhead.
The logical places to inform the progress reporter that it needs to rewind (or is done) are in portbuild.tcl or possibly more generically in portutil.tcl (proc command_exec; start just above "Call this command", finish just before returning).

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Clemens Lang-2
Hi,

On Wed, Feb 22, 2017 at 03:47:32PM +0100, René J.V. Bertin wrote:
> > Yes. Probably still not a good idea to put progress guessing into
> > the implementation of ui_info, though.
>
> proc ui_message is where I've put it because that's the function that
> gets called from Pextlib.

That function, however, also gets called from a myriad of places
throughout other MacPorts code that is not related to build system
output at all. Better change the Pextlib behavior to call a different
function for output generated by the build system instead and implement
your progress handling there, before passing the payload on to ui_info
for logging purposes.

> Putting it there instead of in C code means you could also do
> (line-number based) progress reporting for long operations that are
> written in pure Tcl.

I doubt there are many of those. The biggest things we do in Tcl are
some fs-traverse calls. Any non-Portfile code (i.e. in MacPorts base)
can usually better draw a progress bar using other metrics (e.g.
activation could trivially have a progress bar by counting the files it
activates).

> > Which file? Why do you need to call start and finish from different
> > files? That sounds like a problem waiting to happen if you do the
> > progress setting from macports.tcl but the start and finish from
> > somewhere else.
>
> Yes, but take the case of the build phase. proc ui_message has no way
> of knowing when that phase starts unless it starts tracking changes in
> macports::current_phase . And that's probably something we'd want to
> avoid to prevent unnecessary overhead.

You could explicitly re-set a counter at the beginning of a phase and
then use that in between. But then again, that might also be happening
from different files, I guess.

> The logical places to inform the progress reporter that it needs to
> rewind (or is done) are in portbuild.tcl or possibly more generically
> in portutil.tcl (proc command_exec; start just above "Call this
> command", finish just before returning).

Yes, I agree.


--
Clemens
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Wednesday March 08 2017 22:29:29 Clemens Lang wrote:
>Hi,
Hi,

>That function, however, also gets called from a myriad of places
>throughout other MacPorts code that is not related to build system
>output at all. Better change the Pextlib behavior to call a different
>function for output generated by the build system instead and implement
>your progress handling there, before passing the payload on to ui_info
>for logging purposes.

I don't disagree, but how would we determine from there that a build is going to be done (or destroot, I'd add that step too)? Or would we simply support progress logging for everything invoked through Pextlib's system call (SystemCmd, IIRC)?

That would solve the question of where to initialise and terminate the progress reporter but wouldn't it add an extra layer of complexity to calling the progressbar "class" defined in Tcl in the port client? It wouldn't be very nice if the system command all of a sudden starts failing when used in a context where that progressbar class isn't accessible.

If ui_message contains a few additional lines like below, wouldn't the overhead drown in the noise of what's already going on if there's nothing to report?

{{{
    if {ui_may_have_progress_info} {
        if {![ui_isset ports_verbose]} {
            # check _reporting_progress_, initialise progress reporting if not set
            # call a procedure that checks for and processes progress info
        }
    } elseif {_reporting_progress_} {
        # finalise
        _reporting_progress = no
    }
}}}

I think that should give not more than an acceptable overhead (and a small enough addition to proc ui_message) at least for a first approach, allowing us to get a better idea what kind of progress reporting is feasible and useful. In this kind of situation optimisation through refactoring into compiled code is usually something one does in a 2nd step, when justified, no?


>You could explicitly re-set a counter at the beginning of a phase and
>then use that in between. But then again, that might also be happening
>from different files, I guess.
>
>> The logical places to inform the progress reporter that it needs to
>> rewind (or is done) are in portbuild.tcl or possibly more generically
>> in portutil.tcl (proc command_exec; start just above "Call this
>> command", finish just before returning).
>
>Yes, I agree.

I think the variation and code snippet above address the problem. I'd have to try to see if this can really work but I think it should, and it ought to allow to put all the code that does the actual progress reporting stuff in macports.tcl . Files that start or terminate operations that might provide progress info only have to set or `unset ui_may_have_progress_info` and maybe provide the equivalent of a function pointer to a procedure that provides the progress information itself.

R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

René J.V. Bertin
On Thursday March 09 2017 09:45:44 René J.V. Bertin wrote:

Hi,

I'm trying to implement the approach below, adding some code to proc ui_message in macports.tcl

{{{
            # progress reporting
            if {[may_have_progress_info]} {
                if {!${macports::portverbose}} {
                    ui_progress_display_from_message $string
                }
            } elseif {${_reporting_progress_}} {
                #puts "end progress reporting"
                if {[info exists $macports::ui_options(progress_generic)]} {
                    $macports::ui_options(progress_generic) finish
                }
                set _reporting_progress_ no
            }
}}}

but for the hell of me I cannot toggle the "may report progress" state from proc command_exec in portutil.tcl . I've tried global variables and now a proc (with accompanying setter) `may_have_progress_info` defined in macports_util.tcl:

{{{
# progress reporting
proc set_may_have_progress_info {val} {
    namespace upvar ::macports_util command_may_report_progress var
    set var [string is true -strict $val]
    puts "newvar=${var}"
    puts "newval=${macports_util::command_may_report_progress}"
    return ${var}
}
if {![info exists macports_util::command_may_report_progress]} {
    set macports_util::command_may_report_progress 0
}

proc may_have_progress_info {} {
    puts "val=${macports_util::command_may_report_progress}"
    return ${macports_util::command_may_report_progress}
}
}}}

It's as if the code in portutil.tcl and macports.tcl execute in completely different memory spaces; `macports_util::command_may_report_progress` will appear to be true from within portutil.tcl but will remain false from within macports.tcl .

IOW, I'm getting nowhere fast and the Tcl documentation I've consulted so far hasn't been of any help to understand what I'm doing wrong :-/

What am I missing?

Thanks,
R.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: build progress bar

Clemens Lang-2
On Thu, Mar 09, 2017 at 09:45:44AM +0100, René J.V. Bertin wrote:

> I don't disagree, but how would we determine from there that a build
> is going to be done (or destroot, I'd add that step too)? Or would we
> simply support progress logging for everything invoked through
> Pextlib's system call (SystemCmd, IIRC)?
>
> That would solve the question of where to initialise and terminate the
> progress reporter but wouldn't it add an extra layer of complexity to
> calling the progressbar "class" defined in Tcl in the port client? It
> wouldn't be very nice if the system command all of a sudden starts
> failing when used in a context where that progressbar class isn't
> accessible.

I'd say such an approach should come with a way to disable it or pass in
a reference/function to invoke for new progress output to avoid this
situation.

> If ui_message contains a few additional lines like below, wouldn't the
> overhead drown in the noise of what's already going on if there's
> nothing to report?
>
> ...
>
> I think that should give not more than an acceptable overhead (and a
> small enough addition to proc ui_message) at least for a first
> approach, allowing us to get a better idea what kind of progress
> reporting is feasible and useful. In this kind of situation
> optimisation through refactoring into compiled code is usually
> something one does in a 2nd step, when justified, no?

I must say I'm not particularly good at discussing code in snippets on a
mailing list, so please excuse me for not commenting on this.


On Sat, Mar 11, 2017 at 08:16:27PM +0100, René J.V. Bertin wrote:

> I'm trying to implement the approach below, adding some code to proc
> ui_message in macports.tcl
>
> {{{
>             # progress reporting
>             if {[may_have_progress_info]} {
>                 if {!${macports::portverbose}} {
>                     ui_progress_display_from_message $string
>                 }
>             } elseif {${_reporting_progress_}} {
>                 #puts "end progress reporting"
>                 if {[info exists $macports::ui_options(progress_generic)]} {
>                     $macports::ui_options(progress_generic) finish
>                 }
>                 set _reporting_progress_ no
>             }
> }}}
>
> but for the hell of me I cannot toggle the "may report progress" state
> from proc command_exec in portutil.tcl.

Have you considered that code in macports.tcl runs in a different
interpreter than code in the port1.0 folder? macports.tcl creates a
sub-interpreter for running Portfiles.

Take a look at how the progress reporting works for the download
progress bar to get an idea how to pass things through this boundary.


> It's as if the code in portutil.tcl and macports.tcl execute in
> completely different memory spaces;
> `macports_util::command_may_report_progress` will appear to be true
> from within portutil.tcl but will remain false from within
> macports.tcl.

That's the effect of the sub-interpreter and actually desired. Portfiles
shouldn't be able to modify the global macports state in uncontrolled
ways.

--
Clemens
Loading...