The Daily Parker

Politics, Weather, Photography, and the Dog

All the rows in the world

When I launched the final weather archive import on Tuesday, I predicted it would finish around 1pm today. See my accuracy for yourself:

2022-04-08 12:54:05.0975|INFO|Moved 118,773,651 weather archives from v3 to v5
2022-04-08 12:54:05.0975|INFO|Finished importing; duration 3.03:41:19.2445019
2022-04-08 12:54:05.0975|INFO|Import finished

Not a bad prediction.

So Weather Now 5 now has about 260 million historical records going back to 2006, including Chicago's weather from 15 years ago this hour. And where the weather station reported climate records, we've got those too.

Microsoft Azure recalculates storage use daily around 11 am Central time, so I don't have the complete picture yet, but it looks like I transferred about 245 GB of data. I'll find out for sure tomorrow, and in 3-4 days I'll get an accurate view of the storage cost.

Whew. I'm glad that's over.

Justice Ketanji Brown Jackson

The US Senate did something pretty cool yesterday:

The Senate on Thursday confirmed Judge Ketanji Brown Jackson to the Supreme Court, making her the first Black woman to be elevated to the pinnacle of the judicial branch in what her supporters hailed as a needed step toward bringing new diversity and life experience to the court.

Overcoming a concerted effort by Republicans to sully her record and derail her nomination, Judge Jackson was confirmed on a 53-to-47 vote, with three Republicans joining all 50 members of the Democratic caucus in backing her.

Not everyone shared in the joy of the day. As applause echoed from the marbled walls, Senator Mitch McConnell, Republican of Kentucky and the minority leader, turned his back and slowly walked out, as did most of the few Republicans remaining on the floor, leaving half of the chamber empty as the other half celebrated in a stark reflection of the partisan divide.

“When it came to one of the most consequential decisions a president can make, a lifetime appointment to our highest court, the Biden administration let the radicals run the show,” Mr. McConnell had said earlier, making one last argument against Judge Jackson, whose nomination he framed as an example of extremists taking control of the Democratic Party. “The far left got the reckless inflationary spending they wanted. The far left has gotten the insecure border they wanted. And today, the far left will get the Supreme Court justice they wanted.”

Senator McConnell is full of shit, of course, and he knows it. Jackson would have made any Republican Senator's heart sing only five years ago. But, hey, thanks to Senators Susan Collins (R-ME), Lisa Murkowski (R-AK), and Mitt Romney (R-UT), we may have gotten the last bipartisan confirmation in our lifetimes.

On July 1st, Justice Jackson will become only the 7th person to sit on the Court who wasn't a white guy.

The artist and her work

Cassie has spent the last two weeks creating found art out of one of my area rugs. Yesterday the "found" part got too much for me and I let the rug go. Pity, too; I won it at a silent auction for $300 only in 2016, and neither Parker nor Cassie tried to destroy it until this spring.

Here's Cassie's final expression of the piece. Note not only the center section, which Cassie exfiltrated from the house a small bit at a time, but also the left edge, where she expressed a more compelling feeling of the interplay between organic lines and straight edges:

Somebody call lunch!

I've gotten two solid nights of sleep in a row, and I've got a clean desk for the first time in weeks. I hope that this becomes the norm, at least until November, when I'll have a packed musical schedule for six weeks as the Apollo Chorus rehearses or performs about 30 times. But that's seven months off.

That gives me plenty of time to listen to or read these:

And finally, in compiling geographic source data for Weather Now, I discovered that the International Civil Aviation Organisation (ICAO) assigned an official designator the location where the Ingenuity helicopter landed on Mars: JZRO, for Jezero Crater.

Early afternoon roundup

Now that I've got a few weeks without travel, performances*, or work conferences, I can go back to not having enough time to read all the news that interests me. Like these stories:

Finally, Michelin has handed out its 2022 stars for Chicago. Nothing surprising on the list, but I now have four more restaurants to try.

* Except that I volunteered to help a church choir do five Messiah choruses on Easter Sunday, so I've got two extra rehearsals and a service in the next 12 days.

Bonus update: the fog this morning made St Boniface Cemetery especially spooky-looking when Cassie and I went out for her morning walk:

Archive import finished; final archive import starting today

On Friday, I used Arithmetic™ to predict that the 162-million-row weather data transfer from Weather Now v3 to v5 would end around 7pm last night. Let's check the logs:

2022-04-04 18:48:30.7196|INFO|Clearing v3 archival records for ZYTX
2022-04-04 18:49:27.7471|INFO|Moved 157,408,921 weather archives from v3 to v5
2022-04-04 18:49:27.7471|INFO|Finished importing; duration 4.04:14:55.0952715 

Nice prediction. (It logged 157 million rows because I made a performance tweak and re-started the app after 5 million.)

As I've mentioned, those 162 million rows only go back to September 2009. But v3 launched in January 2007. And it turns out I have a second archive, also containing about 25 GB of data, going back to August 2006. I had to think back to decisions I never documented to piece together why.

In August 2006, I had a single big machine that served as my domain controller, Web server, and Exchange endpoint, and a second big database server. In this photo from 31 July 2006, the Exchange/Web/DC server is the white one on the right ("DOPPELKUH") and the SQL server ("BULLE") is the big black one on the left:

Now, BULLE was a huge machine for the time, with about 200 GB of disk space and (I think) 16 GB of memory. That 200 GB expanse tempted me to turn off a data-purge feature from more parsimonious days, and apparently I deployed that change around 9am on 11 August 2006. (Sadly, the purge feature worked as designed, and I have no archival data before then.)

In October 2006, I finally bought a server rack and moved the database to an even bigger set of disks:

Everything ticked along until around the time I deployed the Weather Now v3.5 refresh and discovered that the un-purged data file had grown to 25 GB. So on 3 September 2009, I simply created a new database and changed the data connection strings to point to it. That new database kept growing until I switched the archival data store to the Cloud in 2013.

And now, I get to take advantage of the triviality of that change by making an equally trivial change to the import controller's data connection string. The 2006 data file has 118,774,028 rows covering 3,988 stations from 11 August 2006 to 3 September 2009. At 435 rows per second, the final archival import should finish around...let's see...1 pm on Friday. At that point, every single byte of data Weather Now has collected in the past 15 years will be available through the App for you to see.

More from the archives:

Mondays are still long

I realize posting has slipped a little in the past couple of weeks. It should resume its normal frequency tomorrow, as I actually have five consecutive weeks of a routine schedule coming up.

That routine includes rehearsals on Mondays, though, so nothing new today.

Quick update

The Apollo Chorus performed last night at the Big Foot Arts Festival in Walworth, Wis., so I haven't done a lot of useful things today. I did take a peek at the other weather archive I have lying around, and discovered (a) it has the same schema as the one I'm currently importing into Weather Now 5, and (b) it only goes back to August 2006.

Somewhere I have older archives that I need to find... But if not, NOAA might have some.

The update rolls on...

As of 17:16 CDT, the massive Weather Now v3 to v5 import had 115,441,906 records left to transfer. At 14:28 CDT yesterday, it was at 157,409,431, giving us a rate of ( 41,967,525 / 96,480 seconds = ) 435 records per second. A little more math gives us another 265,392 seconds to go, or 3 days, 1 hour, 43 minutes left.

So, OK then, what's the over-under on this thing finishing before 7pm Monday?

It's just finished station KCKV (Outlaw Field, Clarksville, Tenn.), with another 2,770 stations left to go. Because it's going in alphabetical order, this means it's finished all of the Pacific Islands (A stations), Northern Europe (B and E), Canada (C), Africa (D, F, G, H). There are no I or J stations (at least not on Earth). K is by far the largest swath as it encompasses all of the continental US, which has more airports than any other land mass.

Once it finishes the continental US, it'll have only 38 million left to do! Whee!

Contradictory transit incentives

Two stories this morning seemed oddly juxtaposed. In good news, the City of Chicago announced plans to spend $15 million on 77 km of new bike and pedestrian trails over the next couple of years:

Several of the projects, including plans to convert an old railroad into a trail in Englewood, are still in the planning and design phases. Others, like Sterling Bay’s planned extension of the 606 Bloomingdale Trail into Lincoln Yards, are set to come to fruition through private partnerships. 

The news release lists 12 projects, including several that had been previously announced, that are set to be funded with a $15 million “commitment to jumpstart” the “key projects citywide.” The $15 million comes from a combination of “federal, state and local sources, including general obligation bonds, Tax Increment Financing, and Open Space Impact Fees,” according to a spokesperson for the city’s planning department.

The City also plans to give away 5,000 bicycles to encourage people to get out of their cars. But at the same time, the City announced it would give away 50,000 gas cards to encourage people not to get out of their cars:

The plan, which comes weeks after possible 2023 mayoral candidate Willie Wilson spearheaded several rounds of free gas giveaways, includes gas cards worth $150 each for as many as 50,000 drivers, and transit cards worth $50 each for as many as 100,000 riders. Wilson on Thursday blasted the mayor’s plan and called it a “political stunt.”

Three-quarters of the transit cards would be prioritized for residents in low-income neighborhoods who use the CTA often. The remainder would be distributed throughout the city.

“It will benefit CTA riders across the city, but especially on the South and West sides,” CTA President Dorval Carter said. “Areas that saw the lowest ridership declines during the pandemic, areas where public transit is the best and sometimes the only option.”

Let's pause for just a moment to give political-stunt-incarnate Willie Wilson a golf clap for calling anything a "political stunt."

I get fretting about gasoline prices if you do what you can to save gas and need your car to survive. But on my trip last week, I got passed by idiots in two-ton SUVs who no doubt complain it costs them $100 to fill their tank.

My little Prius got to and from Kentucky on less than $80 of gas, and even with that trip I've still gotten an average of 2.2 liters per 100 km (156 MPG) so far this year. In fact, the second-worst economy I've ever gotten for a tank of gas in this car was on the return trip from Berea, when I got 5.5 L/100 km (43 MPG) over 610 km (400 miles). Of course, since I got back I've averaged 2 L/100 km (140 MPG).

So maybe if people didn't burn as much gasoline, the city wouldn't feel like giving away gasoline was an option? Just a thought.