Re: purpose of DP? (was Re: Problems installing several Aqua ports)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: purpose of DP? (was Re: Problems installing several Aqua ports)

Bryan Blackburn-2
On Mar 22, 2006, at 10:41 AM, Jordan K. Hubbard wrote:

>
> On Mar 22, 2006, at 7:13 AM, Daniel J. Luke wrote:
>
>> Do we have hardware available to work on this? I don't (currently)  
>> have a spare machine that I can dedicate to just building ports,  
>> but it might be interesting to work on getting automated builds  
>> running again.
>
> If we have the technology, a machine can be found.  The problem is  
> that "automated builds" aren't.
>

So I've been putting together some scripts for this (my personal  
rewrites of buildall.sh so I know better what it's doing and to make  
it somewhat more modular) and have come up with a question:  when  
building, should previously-built items stored in DP's packages be  
allowed to satisfy later dependencies, or should things always be  
built from scratch?

In other words, suppose we have three ports: A, B, and C; C depends  
on B depends on A for simplicity.  In the automated build, A gets  
built successfully and DP makes a package.  When building B, do we  
first remove all built packages, causing A to be built again or  
should we use the successful build of A?  In theory, always building  
is closer to what many people will do (install port C instead of A  
then B then C manually), but I'm not sure there's really any true  
difference when building A as a dep of B than building A as the  
intended target.

Obviously, every port will be uninstalled prior to installing each  
new port to make sure dependencies are specified correctly, but I  
don't see any benefit to doing a rebuild every time around.

Bryan


> - Jordan
>



_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: purpose of DP? (was Re: Problems installing several Aqua ports)

James Berry-3
Hey Bryan,

Nice work!

On Mar 30, 2006, at 3:56 PM, Bryan Blackburn wrote:

> On Mar 22, 2006, at 10:41 AM, Jordan K. Hubbard wrote:
>>
>> On Mar 22, 2006, at 7:13 AM, Daniel J. Luke wrote:
>>
>>> Do we have hardware available to work on this? I don't  
>>> (currently) have a spare machine that I can dedicate to just  
>>> building ports, but it might be interesting to work on getting  
>>> automated builds running again.
>>
>> If we have the technology, a machine can be found.  The problem is  
>> that "automated builds" aren't.
>>
>
> So I've been putting together some scripts for this (my personal  
> rewrites of buildall.sh so I know better what it's doing and to  
> make it somewhat more modular) and have come up with a question:  
> when building, should previously-built items stored in DP's  
> packages be allowed to satisfy later dependencies, or should things  
> always be built from scratch?
>
> In other words, suppose we have three ports: A, B, and C; C depends  
> on B depends on A for simplicity.  In the automated build, A gets  
> built successfully and DP makes a package.  When building B, do we  
> first remove all built packages, causing A to be built again or  
> should we use the successful build of A?  In theory, always  
> building is closer to what many people will do (install port C  
> instead of A then B then C manually), but I'm not sure there's  
> really any true difference when building A as a dep of B than  
> building A as the intended target.

If nothing that A depends on has changed, than A can be reused. I  
think the more interesting subject is whether C should be rebuilt if  
B or A changes.

> Obviously, every port will be uninstalled prior to installing each  
> new port to make sure dependencies are specified correctly, but I  
> don't see any benefit to doing a rebuild every time around.
I agree with  you. I'm going to be working soon on the dependency  
engine to try to make some of this simpler, and after that on a  
chrooted auto build environment. Maybe we can collaborate to some  
degree. I've been informally working on some of the requirements for  
this process.

James.
_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: purpose of DP? (was Re: Problems installing several Aqua ports)

Mark Duling
James Berry <[hidden email]> writes:
>I agree with  you. I'm going to be working soon on the dependency  
>engine to try to make some of this simpler, and after that on a  
>chrooted auto build environment. Maybe we can collaborate to some  
>degree. I've been informally working on some of the requirements for  
>this process.

While we've got people thinking on the topic of dependencies, can someone
tell me if these bugs are still considered valid?

http://bugzilla.opendarwin.org/show_bug.cgi?id=2568

http://bugzilla.opendarwin.org/show_bug.cgi?id=126

Mark  (The bug Nazi)

_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: purpose of DP? (was Re: Problems installing several Aqua ports)

Jordan K. Hubbard-3
In reply to this post by Bryan Blackburn-2

On Mar 30, 2006, at 3:56 PM, Bryan Blackburn wrote:

> So I've been putting together some scripts for this (my personal  
> rewrites of buildall.sh so I know better what it's doing and to  
> make it somewhat more modular) and have come up with a question:  
> when building, should previously-built items stored in DP's  
> packages be allowed to satisfy later dependencies, or should things  
> always be built from scratch?

I think satisfying deps with previously built packages, assuming that  
each and every package has been built with the same cleanroom scripts  
(so you know you can trust it), is the only way such a system can  
ultimately work.  Let's say DP were to decide to build its own X.org  
package for all ports which depended on X11; building all of X for  
each and every package which lists it as a dep would be cost  
prohibitive, to say the least.   That's just one example - there are  
many others!

> Obviously, every port will be uninstalled prior to installing each  
> new port to make sure dependencies are specified correctly, but I  
> don't see any benefit to doing a rebuild every time around.

I agree completely.

- Jordan

_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Automated building (was Re: purpose of DP?)

Bryan Blackburn-2
In reply to this post by James Berry-3
On Mar 30, 2006, at 5:05 PM, James Berry wrote:

> Hey Bryan,
>
> Nice work!
>
> On Mar 30, 2006, at 3:56 PM, Bryan Blackburn wrote:
>
>> On Mar 22, 2006, at 10:41 AM, Jordan K. Hubbard wrote:
>>>
>>> On Mar 22, 2006, at 7:13 AM, Daniel J. Luke wrote:
>>>
>>>> Do we have hardware available to work on this? I don't  
>>>> (currently) have a spare machine that I can dedicate to just  
>>>> building ports, but it might be interesting to work on getting  
>>>> automated builds running again.
>>>
>>> If we have the technology, a machine can be found.  The problem  
>>> is that "automated builds" aren't.
>>>
>>
>> So I've been putting together some scripts for this (my personal  
>> rewrites of buildall.sh so I know better what it's doing and to  
>> make it somewhat more modular) and have come up with a question:  
>> when building, should previously-built items stored in DP's  
>> packages be allowed to satisfy later dependencies, or should  
>> things always be built from scratch?
>>
>> In other words, suppose we have three ports: A, B, and C; C  
>> depends on B depends on A for simplicity.  In the automated build,  
>> A gets built successfully and DP makes a package.  When building  
>> B, do we first remove all built packages, causing A to be built  
>> again or should we use the successful build of A?  In theory,  
>> always building is closer to what many people will do (install  
>> port C instead of A then B then C manually), but I'm not sure  
>> there's really any true difference when building A as a dep of B  
>> than building A as the intended target.
>
> If nothing that A depends on has changed, than A can be reused. I  
> think the more interesting subject is whether C should be rebuilt  
> if B or A changes.
>

Right, and since this is building/installing DP itself in a chroot  
and building all ports within that, there's no way A can be changed  
by the time C comes up in the build (unless the script actually does  
something like selfupdate, which it doesn't).

I definitely haven't brought it to the level of detecting that, for  
example, port B has changed in cvs so let's update, kill off B's  
package and anything depending on it, etc...

>> Obviously, every port will be uninstalled prior to installing each  
>> new port to make sure dependencies are specified correctly, but I  
>> don't see any benefit to doing a rebuild every time around.
> I agree with  you. I'm going to be working soon on the dependency  
> engine to try to make some of this simpler, and after that on a  
> chrooted auto build environment. Maybe we can collaborate to some  
> degree. I've been informally working on some of the requirements  
> for this process.
>

This is actually the "chrooted auto build environment" so  
collaboration would definitely be in our best interest...However,  
it's mostly working at this point, I'm currently getting the final  
reporting stuff together to be useful now.

Bryan


> James.
>

_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automated building (was Re: purpose of DP?)

Bryan Blackburn-2
In reply to this post by Jordan K. Hubbard-3
On Mar 30, 2006, at 9:38 PM, Jordan K. Hubbard wrote:

>
> On Mar 30, 2006, at 3:56 PM, Bryan Blackburn wrote:
>
>> So I've been putting together some scripts for this (my personal  
>> rewrites of buildall.sh so I know better what it's doing and to  
>> make it somewhat more modular) and have come up with a question:  
>> when building, should previously-built items stored in DP's  
>> packages be allowed to satisfy later dependencies, or should  
>> things always be built from scratch?
>
> I think satisfying deps with previously built packages, assuming  
> that each and every package has been built with the same cleanroom  
> scripts (so you know you can trust it), is the only way such a  
> system can ultimately work.  Let's say DP were to decide to build  
> its own X.org package for all ports which depended on X11; building  
> all of X for each and every package which lists it as a dep would  
> be cost prohibitive, to say the least.   That's just one example -  
> there are many others!
>

Yeah, X stuff was definitely a big issue since, as your original  
buildall.sh script did, mine doesn't include any of Apple's X11  
install either (although it also copies stuff more discriminately  
instead of copying most of the world then deleting), not to mention  
g77/gfortran stuff...

It also (when not given an explicit port list) generates a list such  
that dependencies are built first to be an initial step to eventually  
detecting that, if a dependency for a port failed earlier, no point  
in trying again...

Bryan


>> Obviously, every port will be uninstalled prior to installing each  
>> new port to make sure dependencies are specified correctly, but I  
>> don't see any benefit to doing a rebuild every time around.
>
> I agree completely.
>
> - Jordan
>

_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automated building (was Re: purpose of DP?)

James Berry-3
In reply to this post by Bryan Blackburn-2

On Mar 31, 2006, at 12:44 AM, Bryan Blackburn wrote:

> On Mar 30, 2006, at 5:05 PM, James Berry wrote:
>> Hey Bryan,
>>
>> Nice work!
>> I agree with  you. I'm going to be working soon on the dependency  
>> engine to try to make some of this simpler, and after that on a  
>> chrooted auto build environment. Maybe we can collaborate to some  
>> degree. I've been informally working on some of the requirements  
>> for this process.
>>
>
> This is actually the "chrooted auto build environment" so  
> collaboration would definitely be in our best interest...However,  
> it's mostly working at this point, I'm currently getting the final  
> reporting stuff together to be useful now.


Bryan,

Great to hear that you've got this working. I'd love to see/hear more  
when you're ready to share. Collaboration is in everybody's best  
interest.

James.
_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: purpose of DP? (was Re: Problems installing several Aqua ports)

Chris-158
In reply to this post by Jordan K. Hubbard-3
Quoting "Jordan K. Hubbard" <[hidden email]>:

>
> On Mar 30, 2006, at 3:56 PM, Bryan Blackburn wrote:
>
>> So I've been putting together some scripts for this (my personal  
>> rewrites of buildall.sh so I know better what it's doing and to  
>> make it somewhat more modular) and have come up with a question:  
>> when building, should previously-built items stored in DP's  
>> packages be allowed to satisfy later dependencies, or should things  
>> always be built from scratch?
>
> I think satisfying deps with previously built packages, assuming that
>  each and every package has been built with the same cleanroom
> scripts  (so you know you can trust it), is the only way such a
> system can  ultimately work.  Let's say DP were to decide to build
> its own X.org  package for all ports which depended on X11; building
> all of X for  each and every package which lists it as a dep would be
> cost  prohibitive, to say the least.   That's just one example -
> there are  many others!

While I would tend to agree with this for the most part. I am wondering about
trapping apps that do not; or correctly; search and satisfy their
dependencies.
That is; gimp needs glib and sources it's availability on the system;
then proceeds to build; having found it's existance on the system. But
what if gimp needs glib; sources it on the system; but doesn't find it
and doesn't properly source the source for creating/ building it. Or finds
the sources, but doesn't initiate the building of glib correctly?
Basically, what if the app doesn't call the right build process for
deps correctly? If they're already built/ installed. It won't trap these
deficiencies in these apps. I only mention this. As I have noiticed ppl
on the list making requests for situations similar to these.

--Chris

>
>> Obviously, every port will be uninstalled prior to installing each  
>> new port to make sure dependencies are specified correctly, but I  
>> don't see any benefit to doing a rebuild every time around.
>
> I agree completely.
>
> - Jordan
>
> _______________________________________________
> Darwinports mailing list
> [hidden email]
> http://www.opendarwin.org/mailman/listinfo/darwinports
>



--
Microsoft:
Disc space -- the final frontier!

-----------------------------------------------------------------
FreeBSD 5.4-RELEASE-p12 (SMP - 900x2) Tue Mar 7 19:37:23 PST 2006
/////////////////////////////////////////////////////////////////

_______________________________________________
Darwinports mailing list
[hidden email]
http://www.opendarwin.org/mailman/listinfo/darwinports
Loading...