registry with multiple portfiles

classic Classic list List threaded Threaded
18 messages Options
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

registry with multiple portfiles

db
Is it normal or expected to have multiple portfiles for the same port in the registry?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-11 21:09 , db wrote:
> Is it normal or expected to have multiple portfiles for the same port in the registry?

When multiple versions of the port are installed, yes.

- Josh
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 11 Jul 2017, at 14:50, Joshua Root <[hidden email]> wrote:
> On 2017-7-11 21:09 , db wrote:
>> Is it normal or expected to have multiple portfiles for the same port in the registry?
> When multiple versions of the port are installed, yes.

And when having just one?

$ tree /opt/local/var/macports/registry/portfiles/py-ipaddr-2.1.11_0
/opt/local/var/macports/registry/portfiles/py-ipaddr-2.1.11_0
├── 2e52e6dac0be45e76177be8807c75a8530ed272cc05aee96a0684980aa8548bf-1278
│   └── Portfile
└── 8504b2882d636f124067ef74c887debeb83592c287165ed32fff7ef05104667d-1244
    └── Portfile

2 directories, 2 files
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-11 23:24 , db wrote:

> On 11 Jul 2017, at 14:50, Joshua Root <[hidden email]> wrote:
>> On 2017-7-11 21:09 , db wrote:
>>> Is it normal or expected to have multiple portfiles for the same port in the registry?
>> When multiple versions of the port are installed, yes.
>
> And when having just one?
>
> $ tree /opt/local/var/macports/registry/portfiles/py-ipaddr-2.1.11_0
> /opt/local/var/macports/registry/portfiles/py-ipaddr-2.1.11_0
> ├── 2e52e6dac0be45e76177be8807c75a8530ed272cc05aee96a0684980aa8548bf-1278
> │   └── Portfile
> └── 8504b2882d636f124067ef74c887debeb83592c287165ed32fff7ef05104667d-1244
>      └── Portfile
>
> 2 directories, 2 files

Less expected. It can still happen if one version is installed with
multiple variant selections, and the portfile changed in between. How do
the files differ? Is one not referenced in the registry database?

- Josh
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 11 Jul 2017, at 16:16, Joshua Root <[hidden email]> wrote:
> How do the files differ? Is one not referenced in the registry database?

port cat shows the right file. I think I might explain this case — but not others. While writing the portfile for py-ipaddr in vim I saved it and stopped the process, then installed/uninstalled the port and made some corrections in the portfile, repeat. Does it sound plausible?

But other non-installed ports also show more than one file

$ tree /opt/local/var/macports/registry/portfiles/bison-3.0.4_*
/opt/local/var/macports/registry/portfiles/bison-3.0.4_1
├── e93220c1e552b45d87e79b3a8844d01e4654ddc219eb7f845e030296ab326516-2427
│   └── Portfile
└── f483d3ba6033e5e570bcbd8eed06d3ec2a7d8526c74c46d6c1e081651cb07f1c-3227
    └── Portfile
/opt/local/var/macports/registry/portfiles/bison-3.0.4_2
└── a7da10d6954a883bc3a799e1ec1ac3064475957d2228aba4b70f1d6b9b77a69a-3242
    └── Portfile

3 directories, 3 files
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-12 02:40 , db wrote:
> On 11 Jul 2017, at 16:16, Joshua Root <[hidden email]> wrote:
>> How do the files differ? Is one not referenced in the registry database?
>
> port cat shows the right file.

port cat always uses the copy in the ports tree. That might be the same
as one or neither of the copies in the registry.

> I think I might explain this case — but not others. While writing the portfile for py-ipaddr in vim I saved it and stopped the process, then installed/uninstalled the port and made some corrections in the portfile, repeat. Does it sound plausible?

Sure. I don't think we have code to roll back the copying of the
Portfile to the registry directory if we get interrupted by a signal
mid-install.

> But other non-installed ports also show more than one file
>
> $ tree /opt/local/var/macports/registry/portfiles/bison-3.0.4_*
> /opt/local/var/macports/registry/portfiles/bison-3.0.4_1
> ├── e93220c1e552b45d87e79b3a8844d01e4654ddc219eb7f845e030296ab326516-2427
> │   └── Portfile
> └── f483d3ba6033e5e570bcbd8eed06d3ec2a7d8526c74c46d6c1e081651cb07f1c-3227
>      └── Portfile
> /opt/local/var/macports/registry/portfiles/bison-3.0.4_2
> └── a7da10d6954a883bc3a799e1ec1ac3064475957d2228aba4b70f1d6b9b77a69a-3242
>      └── Portfile
>
> 3 directories, 3 files

Hard to say how right or wrong this is without also seeing what 'port
installed bison' reports, but in general these files are supposed to be
deleted when the last port that references them is uninstalled.

- Josh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Rainer Müller-4
In reply to this post by db
On 2017-07-11 18:40, db wrote:
> On 11 Jul 2017, at 16:16, Joshua Root <[hidden email]> wrote:
>> How do the files differ? Is one not referenced in the registry database?
>
> port cat shows the right file. I think I might explain this case — but not others. While writing the portfile for py-ipaddr in vim I saved it and stopped the process, then installed/uninstalled the port and made some corrections in the portfile, repeat. Does it sound plausible?

'port cat' only works with the ports tree, it will not use Portfiles
from the registry.

'port uninstall' is supposed to remove the corresponding Portfile. Maybe
you interrupted it at a point where the Portfile was already copied, but
the SQLite transaction was rolled back...

Rainer
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
In reply to this post by Joshua Root-8
On 11 Jul 2017, at 19:21, Joshua Root <[hidden email]> wrote:
>> I think I might explain this case — but not others. While writing the portfile for py-ipaddr in vim I saved it and stopped the process, then installed/uninstalled the port and made some corrections in the portfile, repeat. Does it sound plausible?
> Sure. I don't think we have code to roll back the copying of the Portfile to the registry directory if we get interrupted by a signal mid-install.

I didn't interrupt during port installation, but vim's editing process between installations.

I re-checked the terminal buffer and noticed this message

$ sudo port install py-textfsm
Portfile changed since last build; discarding previous state.

py-textfsm has 3 files in the registry.

> Hard to say how right or wrong this is without also seeing what 'port installed bison' reports, but in general these files are supposed to be deleted when the last port that references them is uninstalled.

$ port installed bison
None of the specified ports are installed.

In my logs I realised that it was installed once last year and twice the present, possibily as a run dep.
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
In reply to this post by db
On 11 Jul 2017, at 18:40, db <[hidden email]> wrote:
> While writing the portfile for py-ipaddr in vim I saved it and stopped the process, then installed/uninstalled the port and made some corrections in the portfile, repeat.

I tried this with another port while monitoring its registry folder and it creates a new portfile. Port install logs

Portfile changed since last build; discarding previous state.

Should I leave the old portfiles where they are? Is there any way to clean the registry portfile tree?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-12 07:24 , db wrote:
> On 11 Jul 2017, at 18:40, db <[hidden email]> wrote:
>> While writing the portfile for py-ipaddr in vim I saved it and stopped the process, then installed/uninstalled the port and made some corrections in the portfile, repeat.
>
> I tried this with another port while monitoring its registry folder and it creates a new portfile. Port install logs
>
> Portfile changed since last build; discarding previous state.

This message means that the hash of the Portfile in the ports tree does
not match the hash stored in the statefile in the work directory. The
"discarding previous state" means deleting the work directory. None of
this should have any effect on the registry, as the Portfile is only
added to the registry in the install phase, and should be removed in the
uninstall phase.

> Should I leave the old portfiles where they are? Is there any way to clean the registry portfile tree?

The attached script will find any Portfiles that are not referenced by
any installed port, and are thus safe to delete. It uses the same check
that is done during uninstall so I'm still not sure how they are getting
left behind.

- Josh

find_unreferenced_portfiles.tcl (341 bytes) Download Attachment
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 11 Jul 2017, at 23:35, Joshua Root <[hidden email]> wrote:
> The attached script will find any Portfiles that are not referenced by any installed port, and are thus safe to delete. It uses the same check that is done during uninstall so I'm still not sure how they are getting left behind.

Thanks for the script. It removed almost all the unreferenced portfiles and I cleaned up some empty directories, but still left 15 for ports that are not installed (diff'ed /opt/local/var/macports/registry/portfiles/ and /opt/local/var/macports/software/).

MPREGPF=/opt/local/var/macports/registry/portfiles/

./find_unreferenced_portfiles.tcl | while read LINE ; do sudo rm -r $MPREGPF$LINE ; done

ls $MPREGPF | \
while read LINE ; \
do [[ $(ls $MPREGPF$LINE | wc -l | xargs) == 0 ]] \
&& sudo rm -r $MPREGPF$LINE ; done

Is it safe to delete the rest? I haven't manually checked the reg db yet.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-13 11:01 , db wrote:

> On 11 Jul 2017, at 23:35, Joshua Root <[hidden email]> wrote:
>> The attached script will find any Portfiles that are not referenced by any installed port, and are thus safe to delete. It uses the same check that is done during uninstall so I'm still not sure how they are getting left behind.
>
> Thanks for the script. It removed almost all the unreferenced portfiles and I cleaned up some empty directories, but still left 15 for ports that are not installed (diff'ed /opt/local/var/macports/registry/portfiles/ and /opt/local/var/macports/software/).
>
> MPREGPF=/opt/local/var/macports/registry/portfiles/
>
> ./find_unreferenced_portfiles.tcl | while read LINE ; do sudo rm -r $MPREGPF$LINE ; done
>
> ls $MPREGPF | \
> while read LINE ; \
> do [[ $(ls $MPREGPF$LINE | wc -l | xargs) == 0 ]] \
> && sudo rm -r $MPREGPF$LINE ; done
>
> Is it safe to delete the rest? I haven't manually checked the reg db yet.

I don't know. Theoretically if the script doesn't print something then a
row in the ports table matched it.

Easy enough to do something like:

sqlite3 /opt/local/var/macports/registry/registry.db "select * from ports"

and grep for the hashes in question.

- Josh
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 13 Jul 2017, at 03:37, Joshua Root <[hidden email]> wrote:
> I don't know. Theoretically if the script doesn't print something then a row in the ports table matched it.

Here's an extract of my filter, reg portfiles dir + db row(s). It doesn't make sense as port dir it's different from the actual port in db.

/opt/local/var/macports/registry/portfiles/clang-3.7-3.7.1_5
465|llvm-3.7|791b9f5c2268d5a8a3cbcbea2c09baf5c1f2a3f8166556b7b0372dda28faee4e-21225|/opt/local/var/macports/software/llvm-3.7/llvm-3.7-3.7.1_4.darwin_12.x86_64.tbz2|1|3.7.1|4|||installed|1492270652|image|x86_64|0|darwin|12

/opt/local/var/macports/registry/portfiles/py-ipaddr-2.1.11_0
860|py27-ipaddr|5335e8a8dc79a72f33027a9c6ea0ece99cb8e3d0c6c68d7ed5ab71371a265dba-1532|/opt/local/var/macports/software/py27-ipaddr/py27-ipaddr-2.1.11_0.darwin_12.x86_64.tbz2|0|2.1.11|0|||installed|1499887736|image|x86_64|0|darwin|12

/opt/local/var/macports/registry/portfiles/py35-certifi-2017.04.17_0
501|py27-certifi|d20c4c50b90ce475a1991311671f3e6f5a752e543e07ab2c7edf037b9772acaa-1007|/opt/local/var/macports/software/py27-certifi/py27-certifi-2017.04.17_0.darwin_12.noarch.tbz2|0|2017.04.17|0|||installed|1492701114|image|noarch|0|darwin|12

/opt/local/var/macports/registry/portfiles/py36-virtualenv-15.1.0_0
295|py27-virtualenv|08371508a0c82f3c7bb2e91f12207900d7a145d2d08ce18a2eb6f548f2cfecb6-1707|/opt/local/var/macports/software/py27-virtualenv/py27-virtualenv-15.1.0_0.darwin_12.noarch.tbz2|0|15.1.0|0|||installed|1492256983|image|noarch|0|darwin|12 300|py-virtualenv|08371508a0c82f3c7bb2e91f12207900d7a145d2d08ce18a2eb6f548f2cfecb6-1707|/opt/local/var/macports/software/py-virtualenv/py-virtualenv-15.1.0_0.darwin_12.noarch.tbz2|0|15.1.0|0|||installed|1492257000|image|noarch|1|darwin|12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-13 20:28 , db wrote:

> On 13 Jul 2017, at 03:37, Joshua Root <[hidden email]> wrote:
>> I don't know. Theoretically if the script doesn't print something then a row in the ports table matched it.
>
> Here's an extract of my filter, reg portfiles dir + db row(s). It doesn't make sense as port dir it's different from the actual port in db.
>
> /opt/local/var/macports/registry/portfiles/clang-3.7-3.7.1_5
> 465|llvm-3.7|791b9f5c2268d5a8a3cbcbea2c09baf5c1f2a3f8166556b7b0372dda28faee4e-21225|/opt/local/var/macports/software/llvm-3.7/llvm-3.7-3.7.1_4.darwin_12.x86_64.tbz2|1|3.7.1|4|||installed|1492270652|image|x86_64|0|darwin|12
>
> /opt/local/var/macports/registry/portfiles/py-ipaddr-2.1.11_0
> 860|py27-ipaddr|5335e8a8dc79a72f33027a9c6ea0ece99cb8e3d0c6c68d7ed5ab71371a265dba-1532|/opt/local/var/macports/software/py27-ipaddr/py27-ipaddr-2.1.11_0.darwin_12.x86_64.tbz2|0|2.1.11|0|||installed|1499887736|image|x86_64|0|darwin|12
>
> /opt/local/var/macports/registry/portfiles/py35-certifi-2017.04.17_0
> 501|py27-certifi|d20c4c50b90ce475a1991311671f3e6f5a752e543e07ab2c7edf037b9772acaa-1007|/opt/local/var/macports/software/py27-certifi/py27-certifi-2017.04.17_0.darwin_12.noarch.tbz2|0|2017.04.17|0|||installed|1492701114|image|noarch|0|darwin|12
>
> /opt/local/var/macports/registry/portfiles/py36-virtualenv-15.1.0_0
> 295|py27-virtualenv|08371508a0c82f3c7bb2e91f12207900d7a145d2d08ce18a2eb6f548f2cfecb6-1707|/opt/local/var/macports/software/py27-virtualenv/py27-virtualenv-15.1.0_0.darwin_12.noarch.tbz2|0|15.1.0|0|||installed|1492256983|image|noarch|0|darwin|12 300|py-virtualenv|08371508a0c82f3c7bb2e91f12207900d7a145d2d08ce18a2eb6f548f2cfecb6-1707|/opt/local/var/macports/software/py-virtualenv/py-virtualenv-15.1.0_0.darwin_12.noarch.tbz2|0|15.1.0|0|||installed|1492257000|image|noarch|1|darwin|12

Aha, I understand the problem now. The check is for whether *any*
installed port has a portfile with the same hash and size, but we are
only deleting the directory for the particular subport we are
uninstalling. Other subports will of course have an identical portfile.

Fixed in master:
<https://github.com/macports/macports-base/commit/6c8c70b370753b9015400b27afdcd303b7a0cb63>

- Josh
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 13 Jul 2017, at 18:08, Joshua Root <[hidden email]> wrote:
> Fixed in master: <https://github.com/macports/macports-base/commit/6c8c70b370753b9015400b27afdcd303b7a0cb63>

I understand that there's no additional name field for subports in the registry and that they have the same hash. But I don't get how the additional checking you introduced would prevent this from happening (I cannot speak the details of tcl though). Neither I understand how only portfiles of these 15 out of ~130 ports that have subports/are subports were left in the registry. Only thing that widly comes to mind — I haven't modified their portfiles locally, but AFAIR of dependents on these while interrupting vim's editing process, but again that would have meant leaving orphaned dirs for other ports too.
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 13 Jul 2017, at 21:50, db <[hidden email]> wrote:
> On 13 Jul 2017, at 18:08, Joshua Root <[hidden email]> wrote:
>> Fixed in master: <https://github.com/macports/macports-base/commit/6c8c70b370753b9015400b27afdcd303b7a0cb63>
> I understand that there's no additional name field for subports in the registry and that they have the same hash. But I don't get how the additional checking you introduced would prevent this from happening (I cannot speak the details of tcl though). Neither I understand how only portfiles of these 15 out of ~130 ports that have subports/are subports were left in the registry. Only thing that widly comes to mind — I haven't modified their portfiles locally, but AFAIR of dependents on these while interrupting vim's editing process, but again that would have meant leaving orphaned dirs for other ports too.

Can I delete those rows safely?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

Joshua Root-8
On 2017-7-17 17:53 , db wrote:
> On 13 Jul 2017, at 21:50, db <[hidden email]> wrote:
>> On 13 Jul 2017, at 18:08, Joshua Root <[hidden email]> wrote:
>>> Fixed in master: <https://github.com/macports/macports-base/commit/6c8c70b370753b9015400b27afdcd303b7a0cb63>
>> I understand that there's no additional name field for subports in the registry and that they have the same hash. But I don't get how the additional checking you introduced would prevent this from happening (I cannot speak the details of tcl though). Neither I understand how only portfiles of these 15 out of ~130 ports that have subports/are subports were left in the registry. Only thing that widly comes to mind — I haven't modified their portfiles locally, but AFAIR of dependents on these while interrupting vim's editing process, but again that would have meant leaving orphaned dirs for other ports too.
>
> Can I delete those rows safely?

Don't delete anything from the database, its contents are correct (and
have been the whole time). You can delete the portfiles that are in a
directory named after a port (or version of a port) that is not
installed. Those should have been deleted on uninstall.

- Josh
db
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: registry with multiple portfiles

db
On 17 Jul 2017, at 10:38, Joshua Root <[hidden email]> wrote:
> Don't delete anything from the database, its contents are correct (and have been the whole time). You can delete the portfiles that are in a directory named after a port (or version of a port) that is not installed. Those should have been deleted on uninstall.

Yeah, sorry, the directories. Still I don't get how your commit should fix this from happening.
Loading...