Fwd: Urgent communication required

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: Urgent communication required

Umesh Singla
Forwarding this to macports-dev mailing list once again, as I was not subscribed to the mailing list with my @macports address.

Thanks

---------- Forwarded message ----------
From: Umesh Singla <[hidden email]>
Date: Fri, Jun 9, 2017 at 3:42 AM
Subject: Re: Urgent communication required
To: Bradley Giesbrecht <[hidden email]>
Cc: MacPorts Dev <[hidden email]>


Hi Bradley, 

I'm very sorry to not communicate for a long time. I did not plan any absence before and yes, I'll update you every detail from now on, better to be transparent about any other commitments and my work. I'm bad at communicating and I should not let it reflect here. Someone suggested pushing your work even if it is incomplete, it acts like a record or backup for your record, I can feel the importance of it now.


1. The agenda as per the proposal timeline [1] for the period from May 30 to Jun 10 is to finalize the different cases to be included in the scope of the project and their priorities, plan the design of the procedures to be written in port.tcl (now in snapshot.tcl and migrate.tcl in macports1.0) and begin with the implementation of the snapshot action. So, for now, we'll be focusing only on the specific snapshot action.

I have not followed Rainer's strategy for having `port snapshot --create` and `port snapshot --restore` as discussed in the previous thread, instead have 3 separate actions. Since we agreed on other use cases as well during the proposal submission time like only keeping a snapshot for now, listing the snapshots to choose from later etc. which will be cumbersome if we resort to `port snapshot <--create, list, migrate, restore>` and could be confusing at times. 


2. In the proposal, I had written "what would be desirable is if `port` was to recommend a migration upon detecting a new environment", so we can have it two ways - either check for the changes in the environment before running every command or only check for the changes when a user actually uses restore (or migrate) action. 

While the first one seems to be more realistic from user's perspective but running it every time is also not a good idea since OS/hardware changes are not frequent. I suggest running the detection for changes in architecture before a set of some commands like install, sync, selfupdate etc. The second option is actually on-top-of-head type solution which assumes the user to be aware of any arch changes by himself and thus, is actually not "recommending". Please suggest a way to proceed.

Also, is there any existing action which checks for changes in the environment which I can use as a reference. I checked selfupdate but it only checks how old port definitions are. Is it sufficed to check for universal_arch/build_arch like options in `macports.conf` file, comparing it with `uname -a` or `env` command outputs. How rigorous we need the detection to be?


3. For the flow, I am following the strategy of diagnose: Add the action in action_array list -> call macports::snapshot_main{} -> call main{} from namespace eval snapshot procedure with opts. If you have another specific flow in mind like you had a nice solution of going with 3 actions before and we finally sorted to that, please share. I'm thinking about adding the detection procedure as a middleware once we finalize the strategy to follow for it, as discussed in the previous point. 


4. Another thing that ran my mind while pondering that there are 2 options for sqlite database as well: make the tables in the very beginning (while initial installation) or while running the snapshot for the first time. I suggest to go with the first one because it's simple.
The major target is to finish the snapshot action before Jun 24.


5. I saw the copyright license on top of most of the files. Do developers have this, right from the beginning or after the initial work on the module gets finished?


6. One request, I asked this before too, can we please, please fix some time for IRC/Hangouts meeting at least a week for the beginning if it's comfortable with you or Clemens or other people involved? This will help me keep on track. I am a person who can only work under strict deadlines. I have failed under loose deadlines before.

Also, Brad, I have missed one of your emails in the past, too, even after marking them important. For some reason, it ended up in spam folder. Please check the screenshot if it is of some use.

I again apologize.

Regards,
Umesh



On Thu, Jun 8, 2017 at 7:57 PM, Bradley Giesbrecht <[hidden email]> wrote:
Umesh:


The GSoC 2017 coding period began May 30th, 2017. I have not received any communications from you and your project blog has not been updated since your first post.

If you have a planned absence you must communicate in advance. If you do not communicate effectively this project will not be successful and you will be in jeopardy of failing your evaluation.

I hope you respond favorably to this notice so we can all do are part to have a successful project thereby maintaining and enhancing our reputations.

Here is a good blog post [1] I encourage you to read.


[1] “The DOs and DON’Ts of Google Summer of Code: Student Edition” https://opensource.googleblog.com/2011/03/dos-and-donts-of-google-summer-of-code.html


Regards,
Bradley Giesbrecht (pixilla)








--
Umesh Singla

Screen Shot 2017-06-09 at 3.18.20 AM.png (235K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GSoC] migration

Rainer Müller-4
Hello Umesh,

good to hear from you! I was afraid we had already lost you...

Where you will push your work to? Either create a new branch (e.g.
gsoc17-migration) in macports/macports-base, or push to your own github
account. It would be good to know which place to watch for updates.

I will answer some of other questions inline below.

On 2017-06-09 00:38, Umesh Singla wrote:

> 2. In the proposal, I had written "what would be desirable is if `port`
> was to /recommend/ a migration upon detecting a new environment", so we
> can have it two ways - either check for the changes in the environment
> before running every command or only check for the changes when a user
> actually uses restore (or migrate) action.
>
> While the first one seems to be more realistic from user's perspective
> but running it every time is also not a good idea since OS/hardware
> changes are not frequent. I suggest running the detection for changes in
> architecture before a set of some commands like install, sync,
> selfupdate etc. The second option is actually on-top-of-head type
> solution which assumes the user to be aware of any arch changes by
> himself and thus, is actually not "recommending". Please suggest a way
> to proceed.
>
> Also, is there any existing action which checks for changes in the
> environment which I can use as a reference. I checked selfupdate but it
> only checks how old port definitions are. Is it sufficed to check for
> universal_arch/build_arch like options in `macports.conf` file,
> comparing it with `uname -a` or `env` command outputs. How rigorous we
> need the detection to be?

Such a check already exists, which is executed an every initialization
of the macports1.0 package. As this is just a simple compare of the OS
version, it is no problem to run this every time. Currently it prints a
link to the Migration wiki page:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/macports1.0/macports.tcl#L651-L656

> 5. I saw the copyright license on top of most of the files. Do
> developers have this, right from the beginning or after the initial work
> on the module gets finished?

The following goes with the usual IANAL disclaimer. In almost all
jurisdictions, you own the copyright from the moment you create
something. There is no need to explicitly claim copyright any more
(mostly after USA reformed their copyright law in 1989). The headers in
source code files have pure informational use.

In my opinion, it only makes sense to list authors who contributed a
significant amount of code to the file. Most of files in base should
already list "The MacPorts Project". Although not being a legal entity,
this is a kind of placeholder for all contributors holding the joint
copyright ownership.


While I could not answer all your questions right now (I have to leave
some work for Bradley ;-)), feel free to ask questions as much as you
like on the list or via IRC.

Happy hacking on your project!

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
Hi Rainer, 

I have already created a new branch gsoc17-migrate in macports-base after discussing with Jackson on IRC and just now, pushed some initial code to https://github.com/macports/macports-base/tree/gsoc17-migrate, to begin with. For the copyright part, I'll just ignore it, for now, it doesn't seem to be a big deal.

Thanks

On Fri, Jun 9, 2017 at 5:18 AM, Rainer Müller <[hidden email]> wrote:
Hello Umesh,

good to hear from you! I was afraid we had already lost you...

Where you will push your work to? Either create a new branch (e.g.
gsoc17-migration) in macports/macports-base, or push to your own github
account. It would be good to know which place to watch for updates.

I will answer some of other questions inline below.

On 2017-06-09 00:38, Umesh Singla wrote:
> 2. In the proposal, I had written "what would be desirable is if `port`
> was to /recommend/ a migration upon detecting a new environment", so we
> can have it two ways - either check for the changes in the environment
> before running every command or only check for the changes when a user
> actually uses restore (or migrate) action.
>
> While the first one seems to be more realistic from user's perspective
> but running it every time is also not a good idea since OS/hardware
> changes are not frequent. I suggest running the detection for changes in
> architecture before a set of some commands like install, sync,
> selfupdate etc. The second option is actually on-top-of-head type
> solution which assumes the user to be aware of any arch changes by
> himself and thus, is actually not "recommending". Please suggest a way
> to proceed.
>
> Also, is there any existing action which checks for changes in the
> environment which I can use as a reference. I checked selfupdate but it
> only checks how old port definitions are. Is it sufficed to check for
> universal_arch/build_arch like options in `macports.conf` file,
> comparing it with `uname -a` or `env` command outputs. How rigorous we
> need the detection to be?

Such a check already exists, which is executed an every initialization
of the macports1.0 package. As this is just a simple compare of the OS
version, it is no problem to run this every time. Currently it prints a
link to the Migration wiki page:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/macports1.0/macports.tcl#L651-L656

> 5. I saw the copyright license on top of most of the files. Do
> developers have this, right from the beginning or after the initial work
> on the module gets finished?

The following goes with the usual IANAL disclaimer. In almost all
jurisdictions, you own the copyright from the moment you create
something. There is no need to explicitly claim copyright any more
(mostly after USA reformed their copyright law in 1989). The headers in
source code files have pure informational use.

In my opinion, it only makes sense to list authors who contributed a
significant amount of code to the file. Most of files in base should
already list "The MacPorts Project". Although not being a legal entity,
this is a kind of placeholder for all contributors holding the joint
copyright ownership.


While I could not answer all your questions right now (I have to leave
some work for Bradley ;-)), feel free to ask questions as much as you
like on the list or via IRC.

Happy hacking on your project!

Rainer

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Bradley Giesbrecht-3
In reply to this post by Rainer Müller-4
> On Jun 8, 2017, at 4:48 PM, Rainer Müller <[hidden email]> wrote:
>
> While I could not answer all your questions right now (I have to leave
> some work for Bradley ;-))

All help is welcome, the more the merrier :)


Regards,
Bradley Giesbrecht (pixilla)

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Bradley Giesbrecht-3
In reply to this post by Rainer Müller-4
Hi Umesh,


Thanks to Rainer for his comments on 2 and 5. My comments and questions inline.


On Jun 8, 2017, at 3:12 PM, Umesh Singla <[hidden email]> wrote:
>
> 1. The agenda as per the proposal timeline [1] for the period from May 30 to Jun 10 is to finalize the different cases to be included in the scope of the project and their priorities, plan the design of the procedures to be written in port.tcl (now in snapshot.tcl and migrate.tcl in macports1.0) and begin with the implementation of the snapshot action. So, for now, we'll be focusing only on the specific snapshot action.
>
> I have not followed Rainer's strategy for having `port snapshot --create` and `port snapshot --restore` as discussed in the previous thread, instead have 3 separate actions. Since we agreed on other use cases as well during the proposal submission time like only keeping a snapshot for now, listing the snapshots to choose from later etc. which will be cumbersome if we resort to `port snapshot <--create, list, migrate, restore>` and could be confusing at times.

If I am following this correctly I think isolating the functionality into actions makes sense. If we want to refactor the interface in the future it will be easier to combine pieces then split into parts.

> 4. Another thing that ran my mind while pondering that there are 2 options for sqlite database as well: make the tables in the very beginning (while initial installation) or while running the snapshot for the first time. I suggest to go with the first one because it's simple.
> The major target is to finish the snapshot action before Jun 24.

Does “port selfupdate” constitute an “initial installation”?

Does port currently perform schema checks?

If port can detect if the schema needs updating then perhaps we can hook in and “do the right thing".

> 6. One request, I asked this before too, can we please, please fix some time for IRC/Hangouts meeting at least a week for the beginning if it's comfortable with you or Clemens or other people involved? This will help me keep on track. I am a person who can only work under strict deadlines. I have failed under loose deadlines before.

I suggest IRC #macports on Wednesdays at 2:00 PM UTC. If this time does not work for anyone who would like to be included suggest an alternative or additional day and time.


Regards,
Bradley Giesbrecht (pixilla)




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Rainer Müller-4
On 2017-06-11 19:07, Bradley Giesbrecht wrote:
>> 4. Another thing that ran my mind while pondering that there are 2 options for sqlite database as well: make the tables in the very beginning (while initial installation) or while running the snapshot for the first time. I suggest to go with the first one because it's simple.
>> The major target is to finish the snapshot action before Jun 24.
>
> Does “port selfupdate” constitute an “initial installation”?
>
> Does port currently perform schema checks?
>
> If port can detect if the schema needs updating then perhaps we can hook in and “do the right thing".

For testing it should be enough to just modify the registry.db locally
as you need it. Once you reach a stable schema, you will have to add
modifications to the registry schema at two places.

1) The initial database schema for new installations is defined here:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L122-L131

2) The metadata table in registry.db has a row with key='version', where
value holds the schema version. The code to update from one schema
version to the next is here:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L231-L240

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
In reply to this post by Rainer Müller-4
Hi

Please see inline.

> While the first one seems to be more realistic from user's perspective
> but running it every time is also not a good idea since OS/hardware
> changes are not frequent. I suggest running the detection for changes in
> architecture before a set of some commands like install, sync,
> selfupdate etc. The second option is actually on-top-of-head type
> solution which assumes the user to be aware of any arch changes by
> himself and thus, is actually not "recommending". Please suggest a way
> to proceed.
>
> Also, is there any existing action which checks for changes in the
> environment which I can use as a reference. I checked selfupdate but it
> only checks how old port definitions are. Is it sufficed to check for
> universal_arch/build_arch like options in `macports.conf` file,
> comparing it with `uname -a` or `env` command outputs. How rigorous we
> need the detection to be?

Such a check already exists, which is executed an every initialization
of the macports1.0 package. As this is just a simple compare of the OS


Okay, since there's already a OS comparison made, which I think can be directly used here. But to clarify, just the OS check? a check on x86_64/ppc changes is also needed.

You're pointing to mportinit procedure. A noob doubt here, does mportinit get invoked on every command because I guess not. I walked throught the Clemens video again but couldn't figure out. 

So, if it does, then recommending a migration is already there, then all we need to do is to point out the next action to the user. But If it doesn't run on every port command, I'll raise it in the meeting today.
 

version, it is no problem to run this every time. Currently it prints a
link to the Migration wiki page:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/macports1.0/macports.tcl#L651-L656


It can be changed to suggest user to run new migration commands now instead of a link.

--
Umesh Singla
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
In reply to this post by Bradley Giesbrecht-3
Hi

> I have not followed Rainer's strategy for having `port snapshot --create` and `port snapshot --restore` as discussed in the previous thread, instead have 3 separate actions...

If I am following this correctly I think isolating the functionality into actions makes sense. If we want to refactor the interface in the future it will be easier to combine pieces then split into parts.

Acknowledged. 

> 4. Another thing that ran my mind while pondering that there are 2 options for sqlite database as well: make the tables in the very beginning (while initial installation) or while running the snapshot for the first time. I suggest to go with the first one because it's simple.
> The major target is to finish the snapshot action before Jun 24.

Does “port selfupdate” constitute an “initial installation”?

The way I went through macports1.0 -> selfupdate.tcl file, I don't see any such thing.

Does port currently perform schema checks?

If there's sql.c -> update_db() function, then it should be called somewhere. I only see it at one place inside registry attach function which is not really relevant here. 
 
If port can detect if the schema needs updating then perhaps we can hook in and “do the right thing".

But all this is for the first time, right? I'll ponder over it more. 

Point to me if I missed something vital.
 
I suggest IRC #macports on Wednesdays at 2:00 PM UTC. If this time does not work for anyone who would like to be included suggest an alternative or additional day and time.
 
So, today!


--
Umesh Singla
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
In reply to this post by Rainer Müller-4
Hi

For testing it should be enough to just modify the registry.db locally
as you need it. Once you reach a stable schema, you will have to add
modifications to the registry schema at two places.

1) The initial database schema for new installations is defined here:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L122-L131

2) The metadata table in registry.db has a row with key='version', where
value holds the schema version. The code to update from one schema
version to the next is here:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L231-L240

Thanks Rainer for this. This is really useful. I had doubts on how often cregistry/sql.c file gets updated or whether this is the one. The name sql.c seems to be so OS core file. So, I can just add the new tables here? 

Also, new tables means adding them in update_db() too, just like it is the case with existing port or portgroup tables?

Forgive me for asking almost everything.


--
Umesh Singla
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Rainer Müller-4
In reply to this post by Umesh Singla-2
On 2017-06-14 05:47, Umesh Singla wrote:

>     > 4. Another thing that ran my mind while pondering that there are 2
>     options for sqlite database as well: make the tables in the very
>     beginning (while initial installation) or while running the snapshot
>     for the first time. I suggest to go with the first one because it's
>     simple.
>     > The major target is to finish the snapshot action before Jun 24.
>
>     Does “port selfupdate” constitute an “initial installation”?
>
>
> The way I went through macports1.0 -> selfupdate.tcl file, I don't see
> any such thing.
>
>     Does port currently perform schema checks?
>
>
> If there's sql.c -> update_db() function, then it should be called
> somewhere. I only see it at one place inside registry attach function
> which is not really relevant here.

It is called every time the registry is opened, therefore there is
nothing special in selfupdate. The registry will be upgraded when the
newly installed port command is called for the first time (which already
happens during selfupdate).

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Rainer Müller-4
In reply to this post by Umesh Singla-2
On 2017-06-14 05:48, Umesh Singla wrote:
> Thanks Rainer for this. This is really useful. I had doubts on how often
> cregistry/sql.c file gets updated or whether this is the one. The name
> sql.c seems to be so OS core file. So, I can just add the new tables here?
>
> Also, new tables means adding them in update_db() too, just like it is
> the case with existing port or portgroup tables?

Yes, add new tables to create_tables(). Also add them to update_db() so
older installations get them, too.

Rainer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Bradley Giesbrecht-3
> On Jun 14, 2017, at 3:31 AM, Rainer Müller <[hidden email]> wrote:
>
> On 2017-06-14 05:48, Umesh Singla wrote:
>> Thanks Rainer for this. This is really useful. I had doubts on how often
>> cregistry/sql.c file gets updated or whether this is the one. The name
>> sql.c seems to be so OS core file. So, I can just add the new tables here?
>>
>> Also, new tables means adding them in update_db() too, just like it is
>> the case with existing port or portgroup tables?
>
> Yes, add new tables to create_tables(). Also add them to update_db() so
> older installations get them, too.

Rainer, thank you for the help.


Brad
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Bradley Giesbrecht-3
In reply to this post by Umesh Singla-2
> On Jun 13, 2017, at 8:47 PM, Umesh Singla <[hidden email]> wrote:
>
> I suggest IRC #macports on Wednesdays at 2:00 PM UTC. If this time does not work for anyone who would like to be included suggest an alternative or additional day and time.
>  
> So, today!

I hung out by myself on IRC from 2:30-4:30 PM UTC :)


Brad
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Mojca Miklavec-2
In reply to this post by Umesh Singla-2
On 14 June 2017 at 05:47, Umesh Singla wrote:
>
> Okay, since there's already a OS comparison made, which I think can be
> directly used here. But to clarify, just the OS check? a check on x86_64/ppc
> changes is also needed.

And potentially cxx_stdlib?

Mojca
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
Hi Bradley,

I'm having a bit difficulty in implementing the body of snapshot procedure. Basically what I need to do now is (as I have written in my notes, so you can tell me if I'm on right track in my thoughts as well):

"get the list of installed ports, their-installed-variants and if-requested, 
then add one port at a time to the 2nd table, its variants to the 3rd table, 
after all this, add an entry in the snapshot table for the entire snapshot, 
that's it."

I can get this info from `port -v installed` and `port -v installed requested` command, that is, calling action_installed [1] which in turn, calls registry::installed [2] for each port from snapshot_main. Is this best way to go?

Further, can you point me to some port action where it deals with retrieving and populating tables as a hint? The action_target used for most of the port commands like install, clean, fetch doesn't really hint on how to deal with it. 

I hunted down till I reached macports::registry.format  -->  receipt_{flat/sqlite}.tcl files which have a bunch of functions using registry::entry which I think is the most "basic" operation and then, also receipt_sqlite::create_entry_l function. 

Do I need to write for inserting into tables? If yes, write direct SQL queries or Tcl procedures ( a better way, I think)?

Any help would be appreciated. 

Regards,
Umesh


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Joshua Root-8
On 2017-6-20 03:58 , Umesh Singla wrote:

> Hi Bradley,
>
> I'm having a bit difficulty in implementing the body of snapshot
> procedure. Basically what I need to do now is (as I have written in my
> notes, so you can tell me if I'm on right track in my thoughts as well):
>
> "get the list of installed ports, their-installed-variants and
> if-requested,
> then add one port at a time to the 2nd table, its variants to the 3rd
> table,
> after all this, add an entry in the snapshot table for the entire snapshot,
> that's it."

Taking a step back for a moment, why is an SQL database the best way to
store this data? What sorts of queries are you going to want to run on
it? Would a text (Tcl array) representation similar to the PortIndex be
a better fit?

If there's a good reason for doing it in SQL, great. If not, maybe
you're making life more difficult than it needs to be. :)

> I can get this info from `port -v installed` and `port -v installed
> requested` command, that is, calling /action_installed/ [1] which in
> turn, calls /registry::installed/ [2] for each port from
> /snapshot_main/. Is this best way to go?

Using registry::installed will work, but it's not the most efficient
since it provides the same interface as the old flat-file registry (See
below).

> Further, can you point me to some port action where it deals with
> retrieving and populating tables as a hint? The /action_target/ used for
> most of the port commands like /install/, /clean/, /fetch/ doesn't
> really hint on how to deal with it.

These don't deal with SQL directly, they use the registry API.

> I hunted down till I reached /macports::registry.format/  -->
> /receipt_{flat/sqlite}.tcl /files//which have a bunch of functions using
> /registry::entry/ which I think is the most "basic" operation and then,
> also /receipt_sqlite:://create_entry_l/function.

The way to get the list of installed ports is with 'registry::entry
imaged'. Have a look at how reclaim.tcl does it for example.

The create_entry_l procedure is there to help in converting the old
flat-file format to sqlite. It's probably not very relevant for you.

The registry::entry proc is indeed where much of the magic happens at
the Tcl level. It's implemented in src/registry2.0/entry.c and calls
down into the code in the src/cregistry directory to do the actual
sqlite calls.

> Do I need to write for inserting into tables? If yes, write direct SQL
> queries or Tcl procedures ( a better way, I think)?

If you do end up adding new tables to the db, you will need to add
support for them to the entire stack from cregistry up.

- Josh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
Hi Josh,

Taking a step back for a moment, why is an SQL database the best way to store this data? What sorts of queries are you going to want to run on it? Would a text (Tcl array) representation similar to the PortIndex be a better fit?


We need to store all the information about the existing state of the ports first, then uninstall all the ports and re-install on the updated OS after self-updating using the info we have in the database. So, we can't have a temporary kind of Tcl array representation.


I can get this info from `port -v installed` and `port -v installed requested` command, that is, calling /action_installed/ [1] which in turn, calls /registry::installed/ [2] for each port from /snapshot_main/. Is this best way to go?

Using registry::installed will work, but it's not the most efficient since it provides the same interface as the old flat-file registry (See below).

Further, can you point me to some port action where it deals with retrieving and populating tables as a hint? The /action_target/ used for most of the port commands like /install/, /clean/, /fetch/ doesn't really hint on how to deal with it.

These don't deal with SQL directly, they use the registry API.

I hunted down till I reached /macports::registry.format/  --> /receipt_{flat/sqlite}.tcl /files//which have a bunch of functions using /registry::entry/ which I think is the most "basic" operation and then, also /receipt_sqlite:://create_entry_l/function.

The way to get the list of installed ports is with 'registry::entry imaged'. Have a look at how reclaim.tcl does it for example.


Yes, thanks for this. @Brad, 'registry::entry imaged' or 'registry::entry installed', may be? 

I found 2 functions reg_entry_imaged() and reg_entry_installed() in registry, which look almost similar implementation wise. Can you tell me what is an imaged port? I mean, how will the two results be different?
 

The create_entry_l procedure is there to help in converting the old flat-file format to sqlite. It's probably not very relevant for you.
 
Yeah, ignoring this.
 
If you do end up adding new tables to the db, you will need to add support for them to the entire stack from cregistry up.


I think you're referring to the work in last two commits here


Thank you for the help,
Umesh

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Joshua Root-8
On 2017-6-22 02:09 , Umesh Singla wrote:

> Hi Josh,
>
>     Taking a step back for a moment, why is an SQL database the best way
>     to store this data? What sorts of queries are you going to want to
>     run on it? Would a text (Tcl array) representation similar to the
>     PortIndex be a better fit?
>
>
>
> We need to store all the information about the existing state of the
> ports first, then uninstall all the ports and re-install on the updated
> OS after self-updating using the info we have in the database. So, we
> can't have a temporary kind of Tcl array representation.

A Tcl array representation doesn't have to be temporary. It can easily
be written to a file.

If the only operations are going to be storing the list of ports and
then reinstalling all the ports in the list, that says to me that SQL is
much more complexity than you need.

>     The way to get the list of installed ports is with 'registry::entry
>     imaged'. Have a look at how reclaim.tcl does it for example.
>
>
>
> Yes, thanks for this. @Brad, 'registry::entry imaged' or
> 'registry::entry installed', may be?
>
> I found 2 functions reg_entry_imaged() and reg_entry_installed() in
> registry, which look almost similar implementation wise. Can you tell me
> what is an imaged port? I mean, how will the two results be different?

In registry2 parlance, "imaged" means installed and "installed" means
active. Confusing I know.

- Josh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Umesh Singla-2
Hi

    Taking a step back for a moment, why is an SQL database the best way
    to store this data? What sorts of queries are you going to want to
    run on it? Would a text (Tcl array) representation similar to the
    PortIndex be a better fit?


We need to store all the information about the existing state of the ports first, then uninstall all the ports and re-install on the updated OS after self-updating using the info we have in the database. So, we can't have a temporary kind of Tcl array representation.

A Tcl array representation doesn't have to be temporary. It can easily be written to a file.

If the only operations are going to be storing the list of ports and then reinstalling all the ports in the list, that says to me that SQL is much more complexity than you need.

I'll check with my mentor. I may not have been clear in explaining the need for SQL or Tcl array representation could be an easy way go.

In registry2 parlance, "imaged" means installed and "installed" means active. Confusing I know.

Thanks for the clarification.

- Umesh 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [GSoC] migration

Bradley Giesbrecht-3
> On Jun 21, 2017, at 7:25 PM, Umesh Singla <[hidden email]> wrote:
>
> Hi
>
>     Taking a step back for a moment, why is an SQL database the best way
>     to store this data? What sorts of queries are you going to want to
>     run on it? Would a text (Tcl array) representation similar to the
>     PortIndex be a better fit?
>
>
> We need to store all the information about the existing state of the ports first, then uninstall all the ports and re-install on the updated OS after self-updating using the info we have in the database. So, we can't have a temporary kind of Tcl array representation.
>
> A Tcl array representation doesn't have to be temporary. It can easily be written to a file.
>
> If the only operations are going to be storing the list of ports and then reinstalling all the ports in the list, that says to me that SQL is much more complexity than you need.
>
> I'll check with my mentor. I may not have been clear in explaining the need for SQL or Tcl array representation could be an easy way go.

I assumed we would allow multiple snapshots and be able to chose from a list of snapshots by date-sequence.

For the migration functionality wouldn’t we only install “requested” ports? Dependencies could be different with a platform change.

Also, if the installed variants for a given port are the default variants would we want to ignore variants?


Regards,
Bradley Giesbrecht (pixilla)

12
Loading...