registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Ken Cunningham
On one of my systems, 10.5 PPC, things are taking a long time when uninstalling a port - like 4-5 minutes of heavy tcl interpreter use.

The registry.db is pretty huge:

$ ls -la
total 1286264
drwxr-xr-x     7 root  admin        238 21 Jun 11:06 .
drwxr-xr-x@   15 root  admin        510 18 Dec  2018 ..
-rw-r--r--     1 root  admin          0 21 Jun 10:59 .registry.lock
-rw-rw-r--     1 root  admin          0  1 Oct  2015 .turd_MacPorts
drwxr-xr-x  2954 root  admin     100436 21 Jun 10:59 portfiles
drwxr-xr-x   185 root  admin       6290 13 Jun 19:51 portgroups
-rw-r--r--     1 root  admin  658564096 21 Jun 11:06 registry.db


As you can see, this install has been going along for 5 years, so I guess some bitrot could occur. I’m sure I could blow out the installation and start fresh, but that’s not terribly trivial on 10.5 PPC with no functional buildbot.

Is there something else I can do to put that registry.db on a diet?

Thanks,

Ken
Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Jeffrey Walton
On Sun, Jun 21, 2020 at 2:18 PM Ken Cunningham
<[hidden email]> wrote:

>
> On one of my systems, 10.5 PPC, things are taking a long time when uninstalling a port - like 4-5 minutes of heavy tcl interpreter use.
>
> The registry.db is pretty huge:
>
> $ ls -la
> total 1286264
> drwxr-xr-x     7 root  admin        238 21 Jun 11:06 .
> drwxr-xr-x@   15 root  admin        510 18 Dec  2018 ..
> -rw-r--r--     1 root  admin          0 21 Jun 10:59 .registry.lock
> -rw-rw-r--     1 root  admin          0  1 Oct  2015 .turd_MacPorts
> drwxr-xr-x  2954 root  admin     100436 21 Jun 10:59 portfiles
> drwxr-xr-x   185 root  admin       6290 13 Jun 19:51 portgroups
> -rw-r--r--     1 root  admin  658564096 21 Jun 11:06 registry.db

Another PowerMac 10.5 datapoint, if it is helpful:

$ ls -Al
total 115880
-rw-r--r--    1 root  admin         0 Jun 20 00:07 .registry.lock
-rw-rw-r--    1 root  admin         0 Oct  1  2015 .turd_MacPorts
drwxr-xr-x  207 root  admin      7038 Jun 20 00:07 portfiles
drwxr-xr-x   55 root  admin      1870 Jun 20 00:02 portgroups
-rw-r--r--    1 root  admin  59329536 Jun 20 00:07 registry.db

My MacPorts update recipe is as follows.

    /opt/local/bin/port selfupdate
    /opt/local/bin/port upgrade outdated
    /opt/local/bin/port clean inactive
    /opt/local/bin/port -f uninstall inactive

Jeff
Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Ken Cunningham
yeah, I don’t know.

On my 10.4 PPC system, I see my registry.db is even bigger at 883 MB, but uninstalling ports in instantaneous.

Not sure what the real problem is just at the moment with that 10.5 system.

Ken



> On Jun 21, 2020, at 11:56 AM, Jeffrey Walton <[hidden email]> wrote:
>
> On Sun, Jun 21, 2020 at 2:18 PM Ken Cunningham
> <[hidden email]> wrote:
>>
>> On one of my systems, 10.5 PPC, things are taking a long time when uninstalling a port - like 4-5 minutes of heavy tcl interpreter use.
>>
>> The registry.db is pretty huge:
>>
>> $ ls -la
>> total 1286264
>> drwxr-xr-x     7 root  admin        238 21 Jun 11:06 .
>> drwxr-xr-x@   15 root  admin        510 18 Dec  2018 ..
>> -rw-r--r--     1 root  admin          0 21 Jun 10:59 .registry.lock
>> -rw-rw-r--     1 root  admin          0  1 Oct  2015 .turd_MacPorts
>> drwxr-xr-x  2954 root  admin     100436 21 Jun 10:59 portfiles
>> drwxr-xr-x   185 root  admin       6290 13 Jun 19:51 portgroups
>> -rw-r--r--     1 root  admin  658564096 21 Jun 11:06 registry.db
>
> Another PowerMac 10.5 datapoint, if it is helpful:
>
> $ ls -Al
> total 115880
> -rw-r--r--    1 root  admin         0 Jun 20 00:07 .registry.lock
> -rw-rw-r--    1 root  admin         0 Oct  1  2015 .turd_MacPorts
> drwxr-xr-x  207 root  admin      7038 Jun 20 00:07 portfiles
> drwxr-xr-x   55 root  admin      1870 Jun 20 00:02 portgroups
> -rw-r--r--    1 root  admin  59329536 Jun 20 00:07 registry.db
>
> My MacPorts update recipe is as follows.
>
>    /opt/local/bin/port selfupdate
>    /opt/local/bin/port upgrade outdated
>    /opt/local/bin/port clean inactive
>    /opt/local/bin/port -f uninstall inactive
>
> Jeff

Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Craig Treleaven
In reply to this post by Ken Cunningham
Hi:

I can’t help but I have the same issue as you—uninstalls take a long time.  My registry:

$ ls -la /opt/local/var/macports/registry/
total 1453344
drwxr-xr-x     7 root  wheel        238 19 Jun 15:23 .
drwxr-xr-x@   17 root  admin        578 28 Feb  2017 ..
-rw-r--r--     1 root  wheel          0 19 Jun 15:20 .registry.lock
-rw-rw-r--     1 root  wheel          0 20 Nov  2014 .turd_MacPorts
drwxr-xr-x  4088 root  wheel     138992 19 Jun 15:21 portfiles
drwxr-xr-x   234 root  wheel       7956 29 May 09:12 portgroups
-rw-r--r--     1 root  wheel  744112128 19 Jun 15:23 registry.db

This is on a 2011 MacBook Pro (i7).

Craig

> On Jun 21, 2020, at 2:17 PM, Ken Cunningham <[hidden email]> wrote:
>
> On one of my systems, 10.5 PPC, things are taking a long time when uninstalling a port - like 4-5 minutes of heavy tcl interpreter use.
>
> The registry.db is pretty huge:
>
> $ ls -la
> total 1286264
> drwxr-xr-x     7 root  admin        238 21 Jun 11:06 .
> drwxr-xr-x@   15 root  admin        510 18 Dec  2018 ..
> -rw-r--r--     1 root  admin          0 21 Jun 10:59 .registry.lock
> -rw-rw-r--     1 root  admin          0  1 Oct  2015 .turd_MacPorts
> drwxr-xr-x  2954 root  admin     100436 21 Jun 10:59 portfiles
> drwxr-xr-x   185 root  admin       6290 13 Jun 19:51 portgroups
> -rw-r--r--     1 root  admin  658564096 21 Jun 11:06 registry.db
>
>
> As you can see, this install has been going along for 5 years, so I guess some bitrot could occur. I’m sure I could blow out the installation and start fresh, but that’s not terribly trivial on 10.5 PPC with no functional buildbot.
>
> Is there something else I can do to put that registry.db on a diet?
>
> Thanks,
>
> Ken

Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Bill Cole
In reply to this post by Ken Cunningham
On 21 Jun 2020, at 15:24, Ken Cunningham wrote:

> yeah, I don’t know.
>
> On my 10.4 PPC system, I see my registry.db is even bigger at 883 MB,
> but uninstalling ports in instantaneous.
>
> Not sure what the real problem is just at the moment with that 10.5
> system.

Diagnostic idea: ^T repeatedly while it's being slow can help in
identifying exactly what's being slow. With TCL there's a strong chance
that it's not technically TCL. As a data point, I see slowness on a
10.6.8 Core Duo (i.e. i386) machine that is largely in xcodebuild, for
talks like removal where it is unclear to me why that's being called at
all...

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Ken Cunningham
In reply to this post by Craig Treleaven
I decided to do the obvious, backed up all the built software, and reinstalled everything.

Like they said in “Aliens” .. “No other way to be sure.”

Ken




> On Jun 21, 2020, at 12:43 PM, Craig Treleaven <[hidden email]> wrote:
>
> Hi:
>
> I can’t help but I have the same issue as you—uninstalls take a long time.  My registry:

Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

ryandesign2
Administrator
In reply to this post by Ken Cunningham
On Jun 21, 2020, at 13:17, Ken Cunningham wrote:

> On one of my systems, 10.5 PPC, things are taking a long time when uninstalling a port - like 4-5 minutes of heavy tcl interpreter use.
>
> The registry.db is pretty huge:
>
> $ ls -la
> total 1286264
> drwxr-xr-x     7 root  admin        238 21 Jun 11:06 .
> drwxr-xr-x@   15 root  admin        510 18 Dec  2018 ..
> -rw-r--r--     1 root  admin          0 21 Jun 10:59 .registry.lock
> -rw-rw-r--     1 root  admin          0  1 Oct  2015 .turd_MacPorts
> drwxr-xr-x  2954 root  admin     100436 21 Jun 10:59 portfiles
> drwxr-xr-x   185 root  admin       6290 13 Jun 19:51 portgroups
> -rw-r--r--     1 root  admin  658564096 21 Jun 11:06 registry.db
>
>
> As you can see, this install has been going along for 5 years, so I guess some bitrot could occur. I’m sure I could blow out the installation and start fresh, but that’s not terribly trivial on 10.5 PPC with no functional buildbot.
>
> Is there something else I can do to put that registry.db on a diet?

Since the registry contains information about every port you have installed, and every file installed by those ports, it's natural that it will be larger the more ports you have installed. On our Catalina buildbot worker, for example, since they keep all ports installed, the registry is 1GB. If you don't have many ports installed, then that size of registry would be unexpected.

The registry doesn't immediately shrink if you uninstall a port, but after enough free space builds up in the registry MacPorts will perform the SQLite "vacuum" procedure to reclaim space. MacPorts does not print any status information to tell you when this is happening.

It is normal that it can take a long time for MacPorts to write the modified registry to disk after you've installed or uninstalled something if it is large because you have many ports installed, or if MacPorts happens to perform the vacuum procedure. How long would depend on the speed of your disk. The 300MB registry on my main Mac takes a few seconds to rewrite but that's on a fast SSD.


On Jun 22, 2020, at 00:28, Ken Cunningham wrote:

> I decided to do the obvious, backed up all the built software, and reinstalled everything.

Has that helped? Do you now have a much smaller registry with the same set of ports installed? If so that would be unexpected and might indicate a problem in how we're cleaning up the registry during normal use.

Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Dave Horsfall
In reply to this post by Jeffrey Walton
On Sun, 21 Jun 2020, Jeffrey Walton wrote:

> Another PowerMac 10.5 datapoint, if it is helpful:

On my MacBook Pro (Sierra), it's

-rw-r--r--  1 root  admin  62558208 Jun 22 08:09 /opt/local/var/macports/registry/registry.db

And yeah, as Ryan noted later I got lotsa ports installed :-)

> My MacPorts update recipe is as follows.
>
>    /opt/local/bin/port selfupdate
>    /opt/local/bin/port upgrade outdated
>    /opt/local/bin/port clean inactive
>    /opt/local/bin/port -f uninstall inactive

Is this the "official" care and feeding of MacPorts?  My routine (no real
reason) is:

Sunday: port -u uninstall # Clean out saved ports
Monday: port selfupdate # Update itself
and     port -p upgrade # And play dumb by answering the questions

I haven't timed it (but the build takes a heck of a long time sometimes)
and I'm asked if I want to do a "clean" after two weeks.

Oh, I use "-p" because I come across the occasional broken port ("guile"
springs to mind) and I really want to see what can be built after that,
instead of waiting for a fix.

-- Dave
Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

ryandesign2
Administrator
On Jun 22, 2020, at 17:33, Dave Horsfall wrote:

> On Sun, 21 Jun 2020, Jeffrey Walton wrote:
>
>> Another PowerMac 10.5 datapoint, if it is helpful:
>
> On my MacBook Pro (Sierra), it's
>
> -rw-r--r--  1 root  admin  62558208 Jun 22 08:09 /opt/local/var/macports/registry/registry.db
>
> And yeah, as Ryan noted later I got lotsa ports installed :-)

That's only 60MB so not very big compared to Ken's 628MB.


>> My MacPorts update recipe is as follows.
>>
>>   /opt/local/bin/port selfupdate
>>   /opt/local/bin/port upgrade outdated
>>   /opt/local/bin/port clean inactive
>>   /opt/local/bin/port -f uninstall inactive
>
> Is this the "official" care and feeding of MacPorts?

The official routine is:

sudo port selfupdate
sudo port upgrade outdated

Running "sudo port reclaim" should take care of cleaning up things that are no longer needed. (It could of course still be improved to handle additional cases.) MacPorts should prompt you about running reclaim if you haven't done so in awhile.


> My routine (no real reason) is:
>
> Sunday: port -u uninstall # Clean out saved ports

This is the same as "sudo port uninstall inactive". I advocate removing this confusing second usage of the -u flag. https://trac.macports.org/ticket/33933


> Monday: port selfupdate # Update itself
> and     port -p upgrade # And play dumb by answering the questions
>
> I haven't timed it (but the build takes a heck of a long time sometimes)
> and I'm asked if I want to do a "clean" after two weeks.

Ken's observation is about the amount of time it takes to deal with the registry after a port installation is finished, not the amount of time it takes to build.


> Oh, I use "-p" because I come across the occasional broken port ("guile"
> springs to mind) and I really want to see what can be built after that, instead of waiting for a fix.

You definitely should not use -p with upgrade or install. It will proceed to build ports despite failures to upgrade their dependencies, which you absolutely do not want to do. I've explained why many times before on this list. We really shouldn't allow -p to be used with upgrade or install or should at least print a much more stern warning.

Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Ken Cunningham
In reply to this post by ryandesign2
>>
>> I decided to do the obvious, backed up all the built software, and reinstalled everything.
>
> Has that helped? Do you now have a much smaller registry with the same set of ports installed? If so that would be unexpected and might indicate a problem in how we're cleaning up the registry during normal use.
>

I installed a subset of what I had previously installed. The registry.db is still small, but uninstalls are instantaneous again. I guess we see what happens over time.

All in all, all my 10.5 PPC port builds have been backed up, I made them all available as an archive source to a few interested users, and my system works like lightening again.

So i guess that's a win on all fronts.

Ken



$ ls -la
total 64624
drwxr-xr-x    6 root  admin       204 Jun 23 17:54 .
drwxr-xr-x@  13 root  admin       442 Jun 22 08:29 ..
-rw-r--r--    1 root  admin         0 Jun 23 17:53 .registry.lock
drwxr-xr-x  238 root  admin      8092 Jun 23 17:54 portfiles
drwxr-xr-x   26 root  admin       884 Jun 22 22:28 portgroups
-rw-r--r--    1 root  admin  33087488 Jun 23 17:54 registry.db

Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Jeffrey Walton
On Tue, Jun 23, 2020 at 9:01 PM Ken Cunningham
<[hidden email]> wrote:
>
> ...
> All in all, all my 10.5 PPC port builds have been backed up, I made them all available as an archive source to a few interested users, and my system works like lightening again.

That's weird. MacPorts on my G5 has been broken for months due to libcxx.

How do you manage to build packages?

Jeff
Reply | Threaded
Open this post in threaded view
|

Re: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?

Ken Cunningham

On 2020-06-23, at 8:50 PM, Jeffrey Walton wrote:

> On Tue, Jun 23, 2020 at 9:01 PM Ken Cunningham
> <[hidden email]> wrote:
>>
>> ...
>> All in all, all my 10.5 PPC port builds have been backed up, I made them all available as an archive source to a few interested users, and my system works like lightening again.
>
> That's weird. MacPorts on my G5 has been broken for months due to libcxx.
>
> How do you manage to build packages?
>


I'm sorry for not helping you better.

A very large amount of MacPorts (95%?) works on 10.5 PPC (and most things on 10.4 PPC), but base is broken in a few places and you can't just rely on it to choose the right compiler at present.

The base fixes are trivial <https://trac.macports.org/attachment/ticket/59717/patch-portconfigure-properly-support-less-than-snowleopard.diff> at least on the face of it.

I have not been contributing to base lately as -- well -- ...

Michael tells me he is getting a whole refurbished PPC infrastructure shortly, so perhaps the two of us might get some traction.

Where are you stuck? If you can point me to a ticket I'll get you through it, if I can.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: status of 10.5 PPC support on MacPorts (was: registry.db getting rather obese (> 600MB) and updates very slow. Anything to do?)

Ken Cunningham

On 2020-06-23, at 9:31 PM, Ken Cunningham wrote:
>
> A very large amount of MacPorts (95%?) works on 10.5 PPC (and most things on 10.4 PPC), but base is broken in a few places and you can't just rely on it to choose the right compiler at present.

My usual "acid test" for the state of MacPorts on 10.5 PPC is Octave, which is very demanding on compiler / supporting software, etc, and I'm happy to report it still seems healthy:

$ port -v installed octave
The following ports are currently installed:
  octave @5.2.0_2+accelerate+app+docs+gfortran+graphicsmagick+qt4+sound+sundials (active) platform='darwin 9' archs='ppc' date='2020-06-25T11:01:42-0700'


and runs OK.

Minor tweaks needed in building.

Ken