Quantcast

GSoC'17: Add migrate action to port command Project

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

GSoC'17: Add migrate action to port command Project

Umesh Singla
Hello,

As we discussed on IRC, I've been looking into the project "Add migrate action to port command" and hoping to submit a draft proposal very soon. I'll need your help in putting up a nice proposal for sure. 

I'm currently trying to acquaint myself with the MacPorts codebase and came across the Macports guide [0]. Thanks for keeping such guides for newbies. The project description points to a migration instruction guide [1], automating which is more or less the goal of the project, am I right? 

Since final submission deadline is close, I'll be grateful if someone can point me in the right direction for now and not waste time fiddling around.

Also, @Jackson, I'm not able to find any discussion about this project on mailing list threads. Can you link me to them? I think they'll help me in understanding the need and expected outcomes by the community from this project and might give a better insight into the project.

@pixilla, if we need to discuss, what'd be the best time according to your timezone?

Hoping for a quick reply,
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Jackson Isaac-2
Hi Umesh,


First of all, Welcome to The MacPorts Project. We are always happy to
see new developers around.

On Fri, Mar 24, 2017 at 4:30 AM, Umesh Singla <[hidden email]> wrote:
> Hello,
>
> As we discussed on IRC, I've been looking into the project "Add migrate
> action to port command" and hoping to submit a draft proposal very soon.
> I'll need your help in putting up a nice proposal for sure.
>

Submitting a draft early is a good idea so that we could guide you
well in time. We had submitted a template for proposal on the
summerofcode.withgoogle.com page. Let me know if you are unable to
access it.

> I'm currently trying to acquaint myself with the MacPorts codebase and came
> across the Macports guide [0]. Thanks for keeping such guides for newbies.
> The project description points to a migration instruction guide [1],
> automating which is more or less the goal of the project, am I right?
>
> Since final submission deadline is close, I'll be grateful if someone can
> point me in the right direction for now and not waste time fiddling around.
>
> Also, @Jackson, I'm not able to find any discussion about this project on
> mailing list threads. Can you link me to them? I think they'll help me in
> understanding the need and expected outcomes by the community from this
> project and might give a better insight into the project.
>

It was in macports-users (I thought it was posted on macports-dev,
took me some time to search :P)
https://lists.macports.org/pipermail/macports-users/2017-March/042837.html

> @pixilla, if we need to discuss, what'd be the best time according to your
> timezone?
>

Pixilla,

Could you please guide the student on migration tool project.

Let us know if you face any issue(s).

--
Jackson Isaac
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Bradley Giesbrecht-3
> On Mar 23, 2017, at 9:08 PM, Jackson Isaac <[hidden email]> wrote:
>
> Hi Umesh,
>
>
> First of all, Welcome to The MacPorts Project. We are always happy to
> see new developers around.

Yes, welcome.

> On Fri, Mar 24, 2017 at 4:30 AM, Umesh Singla <[hidden email]> wrote:
>> Hello,
>>
>> As we discussed on IRC, I've been looking into the project "Add migrate
>> action to port command" and hoping to submit a draft proposal very soon.
>> I'll need your help in putting up a nice proposal for sure.
>>
>
> Submitting a draft early is a good idea so that we could guide you
> well in time. We had submitted a template for proposal on the
> summerofcode.withgoogle.com page. Let me know if you are unable to
> access it.
>
>> I'm currently trying to acquaint myself with the MacPorts codebase and came
>> across the Macports guide [0]. Thanks for keeping such guides for newbies.
>> The project description points to a migration instruction guide [1],
>> automating which is more or less the goal of the project, am I right?
>>
>> Since final submission deadline is close, I'll be grateful if someone can
>> point me in the right direction for now and not waste time fiddling around.
>>
>> Also, @Jackson, I'm not able to find any discussion about this project on
>> mailing list threads. Can you link me to them? I think they'll help me in
>> understanding the need and expected outcomes by the community from this
>> project and might give a better insight into the project.
>>
>
> It was in macports-users (I thought it was posted on macports-dev,
> took me some time to search :P)
> https://lists.macports.org/pipermail/macports-users/2017-March/042837.html
>
>> @pixilla, if we need to discuss, what'd be the best time according to your
>> timezone?
>>
>
> Pixilla,
>
> Could you please guide the student on migration tool project.

Yes. I am short on time today. Umesh , what timezone are you and what hours are you generally available for online chat or IRC? I am UTC-8.
If we  can set some standard times to communicate that would be a good start.

> Let us know if you face any issue(s).

Jackson, will do, thank you for your leadership.


Brad

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Umesh Singla
Hi Brad,

I'm UTC +5:30. Around 10:00 to 15:00 (UTC -8) is a good time, I suppose. 

I have been going through discussion on the mailing list again and found that the automating the tasks given under Migration documentation is just a start for the port migration tool we need. Can you tell me what other cases it needs to cater? Do I need to include the solution proposals to them in my GSoC proposal at this time or we can work it out as we go?

As of now, I'm not able to divide the work into different phases, might need your help.

I don't know if we'll allow Tcl to run in the new environment. Otherwise, it seems to implement port with a bash script calling on sqlite3 on the macports registry/registry.d to get the relevant data. 

Also, will the project phasing out dependency on XCode tools affect this one anytime soon?

Thanks,
Umesh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Bradley Giesbrecht-3
> On Mar 28, 2017, at 11:55 AM, Umesh Singla <[hidden email]> wrote:
>
> Hi Brad,
>
> I'm UTC +5:30. Around 10:00 to 15:00 (UTC -8) is a good time, I suppose.

Ok, good, you make it easy for me :)

> I have been going through discussion on the mailing list again and found that the automating the tasks given under Migration documentation is just a start for the port migration tool we need. Can you tell me what other cases it needs to cater?

I believe at a minimum we should plan on two new actions, port “snapshot” and “migrate”. A snapshot will the installed ports, their variants and if the port was “requested”.

The migrate action will call the snapshot action to create and/or use snapshots to rebuild ports on the new platform.

> Do I need to include the solution proposals to them in my GSoC proposal at this time or we can work it out as we go?

I think it is a good idea to include snapshot as an early milestone.

> As of now, I'm not able to divide the work into different phases, might need your help.

Does the migrate and snapshot actions help you divide the work?
The snapshot action is basically an inventory of what is installed along with meta data like requested and variants. We will store snapshots in our sqlite database.

It is early on but I can imagine a schema like:

Tables
snapshots (
        id,
        datetime,
        note
)
snapshot_ports (
        id,
        snapshots_id,
        port_name,
        requested
)
snapshot_port_variants (
        id,
        snapshot_ports_id,
        variant_sign, // (we allow plus and minus variants like +variant1 -variant2)
        variant_name
)

> I don't know if we'll allow Tcl to run in the new environment. Otherwise, it seems to implement port with a bash script calling on sqlite3 on the macports registry/registry.d to get the relevant data.

I think we can require any version of port that includes the migrate action, I do not think we will need bash. Clemens, others, do you concur?

> Also, will the project phasing out dependency on XCode tools affect this one anytime soon?

Good question, I cannot give you a definitive answer at the moment but we should keep this in mind.


Brad
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Umesh Singla
Hi


I believe at a minimum we should plan on two new actions, port “snapshot” and “migrate”. A snapshot will the installed ports, their variants and if the port was “requested”.

The migrate action will call the snapshot action to create and/or use snapshots to rebuild ports on the new platform.

This looks quite a plan.
So, what snapshot action will do is to take a "snapshot" of each installed port and keep a list of these snapshots having a name, requested boolean and other required info (will try to figure it out soon) additionally. 

A variants table will help to specify the appropriate variants, so basically an indirect version of this: `sudo port install port_name +variant1 +variant2`.

Migrate action will take these snapshots and use the original guidelines in the migration docs to rebuild acc to the specifications we now have. Correct me if I'm wrong.
 
I think it is a good idea to include snapshot as an early milestone.

Does the migrate and snapshot actions help you divide the work?

Yes, thanks, this gives a lot of direction to my work.
 
The snapshot action is basically an inventory of what is installed along with meta data like requested and variants. We will store snapshots in our sqlite database. 

I hope I understood it well as above.
 

> I don't know if we'll allow Tcl to run in the new environment. Otherwise, it seems to implement port with a bash script calling on sqlite3 on the macports registry/registry.d to get the relevant data.

I think we can require any version of port that includes the migrate action, I do not think we will need bash. Clemens, others, do you concur?

That time, I was trying to say that for a more general scenario, it'll take to implement port with a bash front-end that invoked sqlite3 directly upon the MacPorts registry (located in ${prefix}/var/macports/registry/registry.d) (not anymore) and extracting the required data. Yes, the above seems an elegant solution but still migrate action needs more implementation detailing. Let me work on it.

> Also, will the project phasing out dependency on XCode tools affect this one anytime soon?
Good question, I cannot give you a definitive answer at the moment but we should keep this in mind.

will include this as stretch goal or something like "Plans after GSoC" :D

An additonal point of thought would be to resolve the conflicts arising if there are conflicting ports in the list and I guess, order of reinstalling ports could also be one such factor for conflict if not the default variants? 

Thanks,
Umesh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Mojca Miklavec-2
Hi,

I'm not too familiar with the project, but just some of my thoughts.

One of the biggest pains at the moment is the fact that we don't
record requested variants anywhere. We do record requested ports, but
not the variants.

It may easily happen that "port install foo" would install variant A
on one OS version and variant B on another. If you just record all the
variants, the user might end up with variant A installed on the new OS
even if he never requested it.

I don't know whether that fits anywhere within the scope of the
project, but I would find it super useful if one of the side products
of the project would be extension of the SQL table that would also
record +requested and -excluded variants.

It might be slightly tricky to properly cover all the corner cases.
One might run
    "sudo port install foo -x11"
where "foo" depends on "bar" ("bar" will not be requested and doesn't
even have option x11) and "bar" depends on "baz" ("baz" will also not
be requested, but will have the option x11 that perhaps(?) needs to be
disabled).

That is: at least having that info available would already be a huge
benefit. If "sudo port upgrade outdated" would take that info into
account ... so much better.

I don't want to distract you from the goal of the project, but if
talking about upgrades etc, taking a bit more care about the variants
seems relatively important to me.


***


I would like to see one more scenario covered if there is sufficient
time. When users change some setting in their macports configuration
(I have https://trac.macports.org/wiki/LibcxxOnOlderSystems in mind,
but there might be other cases), it would be super helpful if users
with a working MacPorts installation could run
    sudo port <something>
which would:
- remember which packages are installed
- deactivate all of them (and perhaps move the files to some temporary
location just in case they are needed again)
- reinstall the same packages using the new settings

Basically the same scenario as with regular migration, except that the
user would not upgrade the OS, only some of the basic macports
settings (OK, I'm not sure how exactly you planned to implement the
migration tool, I'm just speculating).


***


I cannot comment on Bash vs. Tcl, but I imagine that quite a bit
should be done in Tcl.

Some example of mixture of Bash and Tcl that come to my mind:
    https://github.com/macports/mpbb
For example the code that prints variants:
    https://github.com/macports/mpbb/blob/master/tools/canonical-variants.tcl

(Don't bother trying to understand what that does. Just that one can
write some Tcl script that uses MacPorts "API" even if you use a
completely different language from the outside.)


***


Just curious: how exactly does Xcode affect this project?

Mojca
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Umesh Singla
In reply to this post by Jackson Isaac-2
Hi

I've shared a draft proposal with MacPorts through the GSoC portal. Please review.

Thanks,
Umesh

On Mar 25, 2017 7:13 AM, "Jackson Isaac" <[hidden email]> wrote:
Hi,


On Sat, Mar 25, 2017 at 4:03 AM, Umesh Singla <[hidden email]> wrote:
> Hi Jackson
>
> Thanks for acknowledging my mail and also, digging mailing threads. That
> discussion actually clarified the need for the project in the first place
> and what other people community may be expecting from it.
> And about that template, is this the one you're talking about -
> https://trac.macports.org/wiki/SummerOfCodeApplicationTemplate?
>

Please check the draft here
https://summerofcode.withgoogle.com/organizations/4688060231974912/.
You can take some ideas from the link that you found too. Let me know
if you need any further help.

--
Jackson Isaac
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Clemens Lang-2
In reply to this post by Mojca Miklavec-2
Hey everybody,

On Tue, Mar 28, 2017 at 01:37:55PM -0700, Bradley Giesbrecht wrote:

> > On Mar 28, 2017, at 11:55 AM, Umesh Singla <[hidden email]> wrote:
>
> Does the migrate and snapshot actions help you divide the work? The
> snapshot action is basically an inventory of what is installed along
> with meta data like requested and variants. We will store snapshots in
> our sqlite database.
>
> It is early on but I can imagine a schema like:
>
> Tables
> snapshots (
> id,
> datetime,
> note
> )
> snapshot_ports (
> id,
> snapshots_id,
> port_name,
> requested
> )
> snapshot_port_variants (
> id,
> snapshot_ports_id,
> variant_sign, // (we allow plus and minus variants like +variant1 -variant2)
> variant_name
> )

This follows all the good rules of data normalization, but MacPorts
internally uses a different representation of variants. Not saying I
prefer that, just saying that we already have code to deal with it.

I would also not mind changing the format. It's a detail we can ignore
until the project starts, though.

I've also left this as a comment in the proposal.


> > I don't know if we'll allow Tcl to run in the new environment.
> > Otherwise, it seems to implement port with a bash script calling on
> > sqlite3 on the macports registry/registry.d to get the relevant
> > data.
>
> I think we can require any version of port that includes the migrate
> action, I do not think we will need bash. Clemens, others, do you
> concur?

I would suggest that the first action of a 'port snapshot'/'port
migration' action would be to recompile base, i.e. the equivalent of
'port -f selfupdate'. This would allow us to implement this completely
in Tcl/C and use the existing abstractions and helper functions that
MacPorts already has.

> > Also, will the project phasing out dependency on XCode tools affect
> > this one anytime soon?

I won't, but thanks for keeping it in mind.

I've also left some other comments in your proposal. Feel free to take
a look.


On Wed, Mar 29, 2017 at 01:38:23PM +0200, Mojca Miklavec wrote:
> One of the biggest pains at the moment is the fact that we don't
> record requested variants anywhere. We do record requested ports, but
> not the variants.

I agree that's a pain point we should change, but I don't think that
should be covered by this proposal, unless it turns out to be way too
small.


> I would like to see one more scenario covered if there is sufficient
> time. When users change some setting in their macports configuration
> (I have https://trac.macports.org/wiki/LibcxxOnOlderSystems in mind,
> but there might be other cases), it would be super helpful if users
> with a working MacPorts installation could run
>     sudo port <something>
> which would:
> - remember which packages are installed
> - deactivate all of them (and perhaps move the files to some temporary
> location just in case they are needed again)
> - reinstall the same packages using the new settings
>
> Basically the same scenario as with regular migration, except that the
> user would not upgrade the OS, only some of the basic macports
> settings (OK, I'm not sure how exactly you planned to implement the
> migration tool, I'm just speculating).

Good idea, that could just be a run of 'port snapshot' and 'port
migrate' without the base update part.


HTH,
--
Clemens
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Umesh Singla
Hi cal, ijackson

Thanks for reviewing my proposal and providing a great feedback. I've replied to most of your comments in the proposal. This surely helps. Please let me know if I have made some false assumption about anything, the timeline/ project plan needs to be modified a great bit or I haven't talked about some issue (like hardware, performance, ) which you were expecting etc. These things can affect my application, so I'm just asking for more guidance.

Thank you so much,
Umesh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GSoC'17: Add migrate action to port command Project

Bradley Giesbrecht-3
In reply to this post by Clemens Lang-2
> On Mar 30, 2017, at 12:13 PM, Clemens Lang <[hidden email]> wrote:
>
> Hey everybody,
>
> On Tue, Mar 28, 2017 at 01:37:55PM -0700, Bradley Giesbrecht wrote:
>>> On Mar 28, 2017, at 11:55 AM, Umesh Singla <[hidden email]> wrote:
>>
>> Does the migrate and snapshot actions help you divide the work? The
>> snapshot action is basically an inventory of what is installed along
>> with meta data like requested and variants. We will store snapshots in
>> our sqlite database.
>>
>> It is early on but I can imagine a schema like:
>>
>> Tables
>> snapshots (
>> id,
>> datetime,
>> note
>> )
>> snapshot_ports (
>> id,
>> snapshots_id,
>> port_name,
>> requested
>> )
>> snapshot_port_variants (
>> id,
>> snapshot_ports_id,
>> variant_sign, // (we allow plus and minus variants like +variant1 -variant2)
>> variant_name
>> )
>
> This follows all the good rules of data normalization, but MacPorts
> internally uses a different representation of variants. Not saying I
> prefer that, just saying that we already have code to deal with it.
>
> I would also not mind changing the format. It's a detail we can ignore
> until the project starts, though.
>
> I've also left this as a comment in the proposal.
>
>
>>> I don't know if we'll allow Tcl to run in the new environment.
>>> Otherwise, it seems to implement port with a bash script calling on
>>> sqlite3 on the macports registry/registry.d to get the relevant
>>> data.
>>
>> I think we can require any version of port that includes the migrate
>> action, I do not think we will need bash. Clemens, others, do you
>> concur?
>
> I would suggest that the first action of a 'port snapshot'/'port
> migration' action would be to recompile base, i.e. the equivalent of
> 'port -f selfupdate'. This would allow us to implement this completely
> in Tcl/C and use the existing abstractions and helper functions that
> MacPorts already has.
>
>>> Also, will the project phasing out dependency on XCode tools affect
>>> this one anytime soon?
>
> I won't, but thanks for keeping it in mind.
>
> I've also left some other comments in your proposal. Feel free to take
> a look.
>
>
> On Wed, Mar 29, 2017 at 01:38:23PM +0200, Mojca Miklavec wrote:
>> One of the biggest pains at the moment is the fact that we don't
>> record requested variants anywhere. We do record requested ports, but
>> not the variants.
>
> I agree that's a pain point we should change, but I don't think that
> should be covered by this proposal, unless it turns out to be way too
> small.
>
>
>> I would like to see one more scenario covered if there is sufficient
>> time. When users change some setting in their macports configuration
>> (I have https://trac.macports.org/wiki/LibcxxOnOlderSystems in mind,
>> but there might be other cases), it would be super helpful if users
>> with a working MacPorts installation could run
>>    sudo port <something>
>> which would:
>> - remember which packages are installed
>> - deactivate all of them (and perhaps move the files to some temporary
>> location just in case they are needed again)
>> - reinstall the same packages using the new settings
>>
>> Basically the same scenario as with regular migration, except that the
>> user would not upgrade the OS, only some of the basic macports
>> settings (OK, I'm not sure how exactly you planned to implement the
>> migration tool, I'm just speculating).
>
> Good idea, that could just be a run of 'port snapshot' and 'port
> migrate' without the base update part.

Or migrate could call a “restore” action:

port snapshot // create a snapshot
port restore // restore a snapshot, default to last snapshot
port migrate // port snapshot, port selfupdate and port restore

And restore can accept a snapshot id:
port restore snapshot=20170330154118



Brad
Loading...