Build servers going offline due to inclement weather

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Build servers going offline due to inclement weather

ryandesign2
Administrator
The MacPorts build servers are located in Austin, TX which is experiencing its coldest temperatures since 1989, which is driving record electricity usage, which has prompted the local power company to warn of impending rolling power outages. The build servers do not take well to being powered off while builds are happening, so I'm shutting them down now. Hopefully the situation will improve within a day or so.

I'll keep the buildmaster on for now, since it also generates the portindexes and distributes them and the updated portfiles to the public rsync server, but if the power goes out I'll shut it down too because the uninterruptible power supply will not last as long as the projected duration of the power outages.

Binary archives of newly updated ports will not be produced during this time so MacPorts will compile from source on your computer when needed. Source distfiles will not be mirrored during this time either which may impact your ability to install some ports on older OS versions, particularly OS X 10.8 and earlier. If you get an error during the fetch phase, consult:

https://trac.macports.org/wiki/ProblemHotlist#fetch-failures

Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Dave Horsfall
Thanks for the advice; brrr...

And yes, I treat a UPS as giving you enough time to shut down the system
cleanly, which is why the better ones have some sort of a signal to start
the shutdown process automatically.

-- Dave
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

ryandesign2
Administrator
In reply to this post by ryandesign2
On Feb 15, 2021, at 02:42, Ryan Schmidt wrote:

> The MacPorts build servers are located in Austin, TX which is experiencing its coldest temperatures since 1989, which is driving record electricity usage, which has prompted the local power company to warn of impending rolling power outages. The build servers do not take well to being powered off while builds are happening, so I'm shutting them down now. Hopefully the situation will improve within a day or so.
>
> I'll keep the buildmaster on for now, since it also generates the portindexes and distributes them and the updated portfiles to the public rsync server, but if the power goes out I'll shut it down too because the uninterruptible power supply will not last as long as the projected duration of the power outages.
>
> Binary archives of newly updated ports will not be produced during this time so MacPorts will compile from source on your computer when needed. Source distfiles will not be mirrored during this time either which may impact your ability to install some ports on older OS versions, particularly OS X 10.8 and earlier. If you get an error during the fetch phase, consult:
>
> https://trac.macports.org/wiki/ProblemHotlist#fetch-failures

Internet service is out so the buildmaster is off now too. Port updates or other changes committed to the repository won't be reflected on the public rsync server until this is back online.

Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

ryandesign2
Administrator
On Feb 15, 2021, at 17:34, Ryan Schmidt wrote:

> On Feb 15, 2021, at 02:42, Ryan Schmidt wrote:
>
>> The MacPorts build servers are located in Austin, TX which is experiencing its coldest temperatures since 1989, which is driving record electricity usage, which has prompted the local power company to warn of impending rolling power outages. The build servers do not take well to being powered off while builds are happening, so I'm shutting them down now. Hopefully the situation will improve within a day or so.
>>
>> I'll keep the buildmaster on for now, since it also generates the portindexes and distributes them and the updated portfiles to the public rsync server, but if the power goes out I'll shut it down too because the uninterruptible power supply will not last as long as the projected duration of the power outages.
>>
>> Binary archives of newly updated ports will not be produced during this time so MacPorts will compile from source on your computer when needed. Source distfiles will not be mirrored during this time either which may impact your ability to install some ports on older OS versions, particularly OS X 10.8 and earlier. If you get an error during the fetch phase, consult:
>>
>> https://trac.macports.org/wiki/ProblemHotlist#fetch-failures
>
> Internet service is out so the buildmaster is off now too. Port updates or other changes committed to the repository won't be reflected on the public rsync server until this is back online.

Internet service has been restored and power has stayed on, but I'm keeping the servers offline to comply with the City of Austin's request to conserve power as much as possible. The State of Texas energy authority ERCOT has required Austin Energy to reduce its power consumption so drastically that all customers on circuits deemed non-critical (~40% of the city) have been without power for over 24 hours already (in well below freezing temperatures).

https://austinenergy.com/ae/about/news/news-releases/2021/update-what-you-need-to-know-about-austin-energys-outage-situa

The rest of the contiguous United States shares either the east or west power grid, but most of Texas uses its own separate grid managed by ERCOT. Unfortunately right now most of Texas is very cold and everybody is using more electricity running their heaters. At the same time, the weather has knocked out some power plants. And since our grid is separate, there's no way to get surplus power from other states. Texas was not designed with such cold temperatures in mind.

Hopefully the situation will improve in a day or two and I'll get the servers back online then.


Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Todd Doucet


The rest of the contiguous United States shares either the east or west power grid, but most of Texas uses its own separate grid managed by ERCOT. Unfortunately right now most of Texas is very cold and everybody is using more electricity running their heaters. At the same time, the weather has knocked out some power plants. And since our grid is separate, there's no way to get surplus power from other states. Texas was not designed with such cold temperatures in mind.

I read that Texas did import electrical power from Mexico a week or so ago.  I guess they threw the jumper cables over that "big, beautiful wall".

My understanding is that, also, there is a connection to the East US grid, but using it might subject Texas to dreaded Federal regulation of the power industry there.

Stay warm.


Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Peter West-2
The problem is the dependency on windmills. Texas has 25Gw (nameplate only) of wind generation. 12Gw was frozen out (as of a day ago.) On Monday morning, ERCOT wholesale prices hit the cap of $9,000 per Mw-hr (average $25).

This story is being played out in Europe, particularly Germany, during this cold spell.

In Australia, the Eastern and mid States (Qld, NSW, Vic, Tas, SA - more than half the area of the contiguous US) are on a grid. Renewables are heavily subsidised, which is driving baseload generators out of business. Our problems tend to happen in summer.  A couple of years ago the State of South Australia (the wind-power state, like Texas) had blackouts for almost a day when storms brought down extended transmission lines from wind farms, and others had to feather the blades because of strong winds. The grid interconnectors couldn’t keep up with the load, and one of them shut down.

Our wholesale cap is $14,000 per Mw-hr (average around $40) and that cap has been reached on a number of occasions, including the one above. Solar isn’t very effective for more than half of the day, on a good sunlight day.

Australia in the early 2000s had almost the cheapest electricity in the world.

Good fun.

Peter

“Why does this generation seek a sign? Truly, I say to you, no sign will be given to this generation.”

On 16 Feb 2021, at 11:15 pm, Todd Doucet <[hidden email]> wrote:



The rest of the contiguous United States shares either the east or west power grid, but most of Texas uses its own separate grid managed by ERCOT. Unfortunately right now most of Texas is very cold and everybody is using more electricity running their heaters. At the same time, the weather has knocked out some power plants. And since our grid is separate, there's no way to get surplus power from other states. Texas was not designed with such cold temperatures in mind.

I read that Texas did import electrical power from Mexico a week or so ago.  I guess they threw the jumper cables over that "big, beautiful wall".

My understanding is that, also, there is a connection to the East US grid, but using it might subject Texas to dreaded Federal regulation of the power industry there.

Stay warm.

Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

ryandesign2
Administrator


On Feb 16, 2021, at 18:38, Peter West wrote:

> On 16 Feb 2021, at 11:15 pm, Todd Doucet wrote:
>
>>> The rest of the contiguous United States shares either the east or west power grid, but most of Texas uses its own separate grid managed by ERCOT. Unfortunately right now most of Texas is very cold and everybody is using more electricity running their heaters. At the same time, the weather has knocked out some power plants. And since our grid is separate, there's no way to get surplus power from other states. Texas was not designed with such cold temperatures in mind.
>>
>> I read that Texas did import electrical power from Mexico a week or so ago.  I guess they threw the jumper cables over that "big, beautiful wall".
>>
>> My understanding is that, also, there is a connection to the East US grid, but using it might subject Texas to dreaded Federal regulation of the power industry there.
>>
>> Stay warm.
>
> The problem is the dependency on windmills. Texas has 25Gw (nameplate only) of wind generation. 12Gw was frozen out (as of a day ago.) On Monday morning, ERCOT wholesale prices hit the cap of $9,000 per Mw-hr (average $25).
>
> This story is being played out in Europe, particularly Germany, during this cold spell.
>
> In Australia, the Eastern and mid States (Qld, NSW, Vic, Tas, SA - more than half the area of the contiguous US) are on a grid. Renewables are heavily subsidised, which is driving baseload generators out of business. Our problems tend to happen in summer.  A couple of years ago the State of South Australia (the wind-power state, like Texas) had blackouts for almost a day when storms brought down extended transmission lines from wind farms, and others had to feather the blades because of strong winds. The grid interconnectors couldn’t keep up with the load, and one of them shut down.
>
> Our wholesale cap is $14,000 per Mw-hr (average around $40) and that cap has been reached on a number of occasions, including the one above. Solar isn’t very effective for more than half of the day, on a good sunlight day.
>
> Australia in the early 2000s had almost the cheapest electricity in the world.
>
> Good fun.

While Texas does use wind power (and of course solar), it's not our primary source of power at this time of year; natural gas is, and natural gas wells are freezing. 70-80 of Texas' 680 power generation facilities are down, so we have a shortfall of 45,000 MW of power, of which 15,000 MW is from wind and 30,000 MW is from coal and natural gas. (1 MW can power 200 homes, so 45,000 MW is 9,000,000 homes without power.) It's true that Texas can use power from Mexico, but that link only offers 450 MW. And Mexico is also having its own power outages due to not being able to get enough natural gas from Texas to run its own plants. Texas also has a small connection to the eastern power grid, but that power grid is also experiencing its own rolling blackouts already.

Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Peter West-2
Hi Ryan,

That’s interesting about the natural gas supplies.  I’m surprised that Texas does not have even two days worth of gas in storage. Maybe that had all been sold to Mexico. Considering that Russia supplies (from Siberia) significant quantities of natural gas to Europe, with plans for a doubling of the Nord Stream pipelines, they obviously have more experience in keeping gas flowing in freezing conditions. The Nord Stream pipelines do go under water, but the gas has to get to them.


Peter
“Why does this generation seek a sign? Truly, I say to you, no sign will be given to this generation.”

On 17 Feb 2021, at 12:51 pm, Ryan Schmidt <[hidden email]> wrote:



On Feb 16, 2021, at 18:38, Peter West wrote:

On 16 Feb 2021, at 11:15 pm, Todd Doucet wrote:

The rest of the contiguous United States shares either the east or west power grid, but most of Texas uses its own separate grid managed by ERCOT. Unfortunately right now most of Texas is very cold and everybody is using more electricity running their heaters. At the same time, the weather has knocked out some power plants. And since our grid is separate, there's no way to get surplus power from other states. Texas was not designed with such cold temperatures in mind.

I read that Texas did import electrical power from Mexico a week or so ago.  I guess they threw the jumper cables over that "big, beautiful wall".

My understanding is that, also, there is a connection to the East US grid, but using it might subject Texas to dreaded Federal regulation of the power industry there.

Stay warm.

The problem is the dependency on windmills. Texas has 25Gw (nameplate only) of wind generation. 12Gw was frozen out (as of a day ago.) On Monday morning, ERCOT wholesale prices hit the cap of $9,000 per Mw-hr (average $25).

This story is being played out in Europe, particularly Germany, during this cold spell.

In Australia, the Eastern and mid States (Qld, NSW, Vic, Tas, SA - more than half the area of the contiguous US) are on a grid. Renewables are heavily subsidised, which is driving baseload generators out of business. Our problems tend to happen in summer.  A couple of years ago the State of South Australia (the wind-power state, like Texas) had blackouts for almost a day when storms brought down extended transmission lines from wind farms, and others had to feather the blades because of strong winds. The grid interconnectors couldn’t keep up with the load, and one of them shut down.

Our wholesale cap is $14,000 per Mw-hr (average around $40) and that cap has been reached on a number of occasions, including the one above. Solar isn’t very effective for more than half of the day, on a good sunlight day.

Australia in the early 2000s had almost the cheapest electricity in the world.

Good fun.

While Texas does use wind power (and of course solar), it's not our primary source of power at this time of year; natural gas is, and natural gas wells are freezing. 70-80 of Texas' 680 power generation facilities are down, so we have a shortfall of 45,000 MW of power, of which 15,000 MW is from wind and 30,000 MW is from coal and natural gas. (1 MW can power 200 homes, so 45,000 MW is 9,000,000 homes without power.) It's true that Texas can use power from Mexico, but that link only offers 450 MW. And Mexico is also having its own power outages due to not being able to get enough natural gas from Texas to run its own plants. Texas also has a small connection to the eastern power grid, but that power grid is also experiencing its own rolling blackouts already.

Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Dave Horsfall
In reply to this post by ryandesign2
[ Power cuts in Texas ]

Amazing what you can learn here :-)  Seriously, I really feel for you
guys; I've seen the stuff on TV, and, well...

-- Dave
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Richard Bonomo TDS personal

Clearly, Texas needs new nuclear power plants!

During the Blizzard of 1978, coal barges were frozen in the Ohio River.  Ohio (and the eastern grid in general) managed to keep things
going because places like Wisconsin had enough nuclear power plants in those days (no longer, though) to take up the slack from the coal-fired
electrical generation plants there were idled or operating at reduced power output because it was not possible to get coal to them for
some weeks.

Rich

----- Original Message -----
From: "Dave Horsfall" <[hidden email]>
To: "macports-users" <[hidden email]>
Sent: Tuesday, February 16, 2021 10:00:50 PM
Subject: Re: Build servers going offline due to inclement weather

[ Power cuts in Texas ]

Amazing what you can learn here :-)  Seriously, I really feel for you
guys; I've seen the stuff on TV, and, well...

-- Dave
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

James Linder


On 17 Feb 2021, at 12:25 pm, Richard Bonomo TDS personal <[hidden email]> wrote:


Clearly, Texas needs new nuclear power plants!

During the Blizzard of 1978, coal barges were frozen in the Ohio River.  Ohio (and the eastern grid in general) managed to keep things
going because places like Wisconsin had enough nuclear power plants in those days (no longer, though) to take up the slack from the coal-fired
electrical generation plants there were idled or operating at reduced power output because it was not possible to get coal to them for
some weeks.

Rich

----- Original Message -----
From: "Dave Horsfall" <[hidden email]>
To: "macports-users" <[hidden email]>
Sent: Tuesday, February 16, 2021 10:00:50 PM
Subject: Re: Build servers going offline due to inclement weather

[ Power cuts in Texas ]

Amazing what you can learn here :-)  Seriously, I really feel for you
guys; I've seen the stuff on TV, and, well...

Rich while macports servers are in a chaotic state it's fun to play completely OT.

The single (engineering, real) problem with nuclear power is that it makes a mess that lasts millions of years. The consequence of this is it turns out to be a very expensive way to go.
(The 100 year.cost of the, now abandonedish Yucca Mountain repository  (politics) was estimated at $100 billion)

"Earth scientist discovered how to turn them (aliens) into disgusting lumps of jelly ..."
Even if the touted thorium reactors can render waste short lived the question is "who pays"
again making the nuclear option very expensive.


So we need a solution, not condemnation of those who try (ie Germany)
james
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Bill Cole
In reply to this post by Peter West-2
On 16 Feb 2021, at 22:01, Peter West wrote:

> Hi Ryan,
>
> That’s interesting about the natural gas supplies.  I’m surprised
> that Texas does not have even two days worth of gas in storage. Maybe
> that had all been sold to Mexico. Considering that Russia supplies
> (from Siberia) significant quantities of natural gas to Europe, with
> plans for a doubling of the Nord Stream pipelines, they obviously have
> more experience in keeping gas flowing in freezing conditions. The
> Nord Stream pipelines do go under water, but the gas has to get to
> them.

If you *plan* for supplying gas and operating power plants (which need
non-frozen water) in absurdly cold conditions, it is not hard to do,
although it does cost money to implement. After all, people do live in
Canada and the northern tier of the US, and both the wind turbines and
the gas plants are running just fine in Minnesota.  No one ever planned
for essentially all of Texas pl,us adjacent areas south and east to be
as cold as it has been for the past week. There are also quirks of the
way Texas has (not) regulated the energy sector that have encouraged
minimizing prices in normal times, at the cost of the whole system
collapsing in a cold snap.


--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Bill Cole
In reply to this post by Richard Bonomo TDS personal
On 16 Feb 2021, at 23:25, Richard Bonomo TDS personal wrote:

> Clearly, Texas needs new nuclear power plants!

Or they could just winterize the ones they have. e.g. there's a nuclear
plant south of Houston that is offline because its cooling water is
currently ice.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Richard Bonomo TDS personal
Dear Bill et al.,

Does that one rely on a river or other body of water, or does it have cooling towers?

As far as I know, our natural gas facilities and wind turbines are working just fine
here in Wisconsin in our recent sub-zero temperatures, but they are all built for it.

Dear James et al.,

Reprocessing could get around the bulk of the waste/storage issues, but people are
afraid of the security issues in our frightening world.  I am not a great fan
of nuclear fission for general power use, but it has its place for the time being.

I work in Nuclear Fusion, as that is really the only truly long-term (centuries,
millennia) way to power a technically advanced and populous society.

https://www.youtube.com/watch?v=ioq2qNdLQL4

Rich

----- Original Message -----
From: "Bill Cole" <[hidden email]>
To: "macports-users" <[hidden email]>
Sent: Wednesday, February 17, 2021 12:30:43 AM
Subject: Re: Build servers going offline due to inclement weather

On 16 Feb 2021, at 23:25, Richard Bonomo TDS personal wrote:

> Clearly, Texas needs new nuclear power plants!

Or they could just winterize the ones they have. e.g. there's a nuclear
plant south of Houston that is offline because its cooling water is
currently ice.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Dave Horsfall
On Wed, 17 Feb 2021, Richard Bonomo TDS personal wrote:

> I work in Nuclear Fusion, as that is really the only truly long-term
> (centuries, millennia) way to power a technically advanced and populous
> society.

Agreed.

-- Dave
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

MacPorts - Users mailing list
Best mailing list ever?

Michael A. Leonetti
As warm as green tea

On 2/17/21 10:58 PM, Dave Horsfall wrote:
> On Wed, 17 Feb 2021, Richard Bonomo TDS personal wrote:
>
>> I work in Nuclear Fusion, as that is really the only truly long-term
>> (centuries, millennia) way to power a technically advanced and
>> populous society.
>
> Agreed.
>
> -- Dave
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

ryandesign2
Administrator
In reply to this post by ryandesign2
Build servers remain offline. Since my last update, power was out here for 32 hours but is back on now. Power is restored in most of Austin but the Texas electric grid operator still asks us to conserve electricity.

https://twitter.com/ERCOT_ISO/status/1362780731001499650

Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Rainer Müller-4
Hello Ryan,

On 19/02/2021 16.38, Ryan Schmidt wrote:
> Build servers remain offline. Since my last update, power was out here for 32 hours but is back on now. Power is restored in most of Austin but the Texas electric grid operator still asks us to conserve electricity.

Thank you for the update. I hope you are well and stay warm!

Rainer
Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Todd Doucet
Many new users, like me, might not know what the consequences of the build servers being down are, exactly.  I am speculating that this does not mean that MacPorts is not available generally, but instead that if I ask for a port it might build it on my machine instead of pulling a pre-built binary for me.

Hello Ryan,

On 19/02/2021 16.38, Ryan Schmidt wrote:
> Build servers remain offline. Since my last update, power was out here for 32 hours but is back on now. Power is restored in most of Austin but the Texas electric grid operator still asks us to conserve electricity.

Thank you for the update. I hope you are well and stay warm!

Rainer


Reply | Threaded
Open this post in threaded view
|

Re: Build servers going offline due to inclement weather

Bill Cole
On 19 Feb 2021, at 15:12, Todd Doucet wrote:

> Many new users, like me, might not know what the consequences of the
> build servers being down are, exactly.  I am speculating that this
> does not mean that MacPorts is not available generally, but instead
> that if I ask for a port it might build it on my machine instead of
> pulling a pre-built binary for me.

Correct. In some cases there could also be problems with getting source
distributions of some ports automatically.

 From Ryan's initial message:

> Binary archives of newly updated ports will not be produced during
> this time so MacPorts will compile from source on your computer when
> needed. Source distfiles will not be mirrored during this time either
> which may impact your ability to install some ports on older OS
> versions, particularly OS X 10.8 and earlier. If you get an error
> during the fetch phase, consult:
>
> https://trac.macports.org/wiki/ProblemHotlist#fetch-failures

That link explains that if you can find and manually fetch the right
distfile, you can work around a 'port fetch' failure.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
12