The Daily Parker

Politics, Weather, Photography, and the Dog

Chaining LINQ predicates

I've spent a good bit of free time lately working on migrating Weather Now to Azure. Part of this includes rewriting its Gazetteer, or catalog of places that it uses to find weather stations for users. For this version I'm using Entity Framework 5.0, which in turn allows me to use LINQ extensively.

I always try to avoid duplicating code, and I always try to write sufficient unit tests to prevent (and fix) any coding errors I make. (I also use ReSharper and Visual Studio Code Analysis to keep me honest.)

There are two methods in the Gazetteer's PlaceFinder class that search for places by distance. The prototypes are:

public static IEnumerable FindNearby(ILocatable center, Length radius)

and:

public static IEnumerable FindNearby(ILocatable center, Length radius, Expression<Func<Place, bool>> predicate)

But in order for the first method to work, it has to create a predicate of its own to draw a box around the center location. (The ILocatable interface requires Latitude and Longitude. Length is a class in the Inner Drive Extensible Architecture representing a measurable two-dimensional distance.) So in order for the second method to work, it has to chain predicates.

Fortunately, I found Joe and Ben Albahari's library of LINQ extensions. Here's the second method:

public static IEnumerable<PlaceDistance> FindNearby(
	ILocatable center,
	Length radius,
	Expression<Func<Place, bool>> predicate)
{
	var searchPredicate = 
		SearchDistancePredicate(center, radius)
		.And(predicate);

	var places = Find(searchPredicate);

	return SearchDistanceResults(places, center, radius);
}

This allows me to use a single Find method that takes a predicate, engages a retry policy, and returns exactly what I'm looking for. And it allows me to do this, which just blows my mind:

var results = PlaceFinder.FindNearby(TestNode, TestRadius, p => p.Feature.Name == "airport");

Compared with the way Weather Now works under the hood right now, and how much coding the existing code took to achieve the same results, I'm just stunned. And it will make migrating Weather Now a lot easier.

Upgrading to Azure Storage Client 2.0

Oh, Azure Storage team, why did you break everything?

I love upgrades. I really do. So when Microsoft released the new version of the Windows Azure SDK (October 2012, v1.8) along with a full upgrade of the Storage Client (to 2.0), I found a little side project to upgrade, and went straight to the NuGet Package Manager for my prize.

I should say that part of my interest came from wanting to use some of the .NET 4.5 features, including the asynchronous helper methods, HTML 5, and native support for SQL 2012 spatial types, in the new version of Weather Now that I hope to complete before year's end. The Azure SDK 1.8 supports .NET 4.5; previous version didn’t. And the Azure SDK 1.8 includes a new version of the Azure Emulator which supports 4.5 as well.

To support the new, Azure-based version (and to support a bunch of other projects that I migrated to Azure), I have a class library of façades supporting Azure. Fortunately, this architecture encapsulated all of my Azure Storage calls. Unfortunately, the upgrade broke every other line of code in the library.

0. Many have the namespaces have changed. But of course, you use ReSharper, which makes the problem go away.

1.The CloudStorageAccount.FromConfigurationSetting() method is gone. Instead, you have to use CloudStorageAccount.Parse(). Here is a the delta from TortoiseHg:

- _cloudStorageAccount = CloudStorageAccount.FromConfigurationSetting(storageSettingName);
+ var setting = CloudConfigurationManager.GetSetting(storageSettingName);
+ _cloudStorageAccount = CloudStorageAccount.Parse(setting);

2. BlobContainer.GetBlobReference() is gone, too. Instead of getting a generic IBlobContainer reference back, you have to specify whether you want a page blob or a block blob. In this app, I only use page blobs, so the delta looks like this:

- var blob = _blobContainer.GetBlobReference(blobName);
+ var blob = _blobContainer.GetBlockBlobReference(blobName);

Note that BlobContainer also has a GetPageBlobReference() method. It also has a nearly-useless GetBlobReferenceFromServer method that throws a 404 error if the blob doesn’t exist, which makes it useless for creating new blobs.

3. Blob.DeleteIfExists() works somewhat differently, too:

- var blob = _blobContainer.GetBlobReference(blobName);
- blob.DeleteIfExists(new BlobRequestOptions 
-	{ DeleteSnapshotsOption = DeleteSnapshotsOption.IncludeSnapshots });
+ var blob = _blobContainer.GetBlockBlobReference(blobName);
+ blob.DeleteIfExists();

4. Remember downloading text directly from a blob using Blob.DownloadText()? Yeah, that’s gone too. Blobs are all about streams now:

- var blob = _blobContainer.GetBlobReference(blobName);
- return blob.DownloadText();
+ using (var stream = new MemoryStream())
+ {
+ 	var blob = _blobContainer.GetBlockBlobReference(blobName);
+ 	blob.DownloadToStream(stream);
+ 	using (var reader = new StreamReader(stream, true))
+ 	{
+ 		stream.Position = 0;
+ 		return reader.ReadToEnd();
+ 	}
+ }

5. Because blobs are all stream-based now, you can’t simply upload files to them. Here’s the correction to the disappearance of Blob.UploadFile():

- var blob = _blobContainer.GetBlobReference(blobName);
- blob.UploadByteArray(value);
+ var blob = _blobContainer.GetBlockBlobReference(blobName);
+ using (var stream = new MemoryStream(value))
+ {
+ 	blob.UploadFromStream(stream);
+ }

6. Microsoft even helpfully corrected a spelling error which, yes, broke my code:

- _blobContainer.CreateIfNotExist();
+ _blobContainer.CreateIfNotExists();

Yes, if not existS. Notice the big red S, which is something I’d like to give the Azure team after this upgrade.*

7. We’re not done, yet. They fixed a "problem" with tables, too:

  var cloudTableClient = _cloudStorageAccount.CreateCloudTableClient();
- cloudTableClient.CreateTableIfNotExist(TableName);
- var context = cloudTableClient.GetDataServiceContext();
+ var table = cloudTableClient.GetTableReference(TableName);
+ table.CreateIfNotExists();
+ var context = cloudTableClient.GetTableServiceContext();

8. Finally, if you have used the CloudStorageAccount.SetConfigurationSettingPublisher() method, that’s gone too, but you don’t need it. Instead, use the CloudConfigurationManager.GetSetting() method directly. Instead of doing this:

if (RoleEnvironment.IsAvailable)
{
	CloudStorageAccount.SetConfigurationSettingPublisher(
		(configName, configSetter) => 
		configSetter(RoleEnvironment.GetConfigurationSettingValue(configName)));
}
else
{
	CloudStorageAccount.SetConfigurationSettingPublisher(
		(configName, configSetter) => 
		configSetter(ConfigurationManager.AppSettings[configName]));
}

You can simply do this:

var someSetting = CloudConfigurationManager.GetSetting(settingKey);

The CloudConfiguration.GetSetting() method first tries to get the setting from Azure, then from the ConfigurationManager (i.e., local settings).

I hope I have just saved you three hours of silently cursing Microsoft’s Azure Storage team.

* Apologies to Bill Cosby.

Time to stock up on Twinkies

Texas-based Hostess Brands has shut down in preparation for liquidation:

Hostess said a strike by members of the Bakery, Confectionery, Tobacco Workers and Grain Millers International Union that began last week had crippled its ability to produce and deliver products at several facilities, and it had no choice but to give up its effort to emerge intact from bankruptcy court.

The Irving, Texas-based company said the liquidation would mean that most of its 18,500 employees would lose their jobs.

In the Chicago area, Hostess employs about 300 workers making CupCakes, HoHos and Honey Buns in Schiller Park. Hostess also has a bakery in Hodgkins, where 325 workers make Beefsteak, Butternut, Home Pride, Nature’s Pride and Wonder breads.

Union President Frank Hurt said the company's failure was not the fault of the union but the "result of nearly a decade of financial and operational mismanagement" and that management was trying to make union workers the scapegoats for a plan by Wall Street investors to sell Hostess.

Twinkies, legend has it, were invented right here in Chicago in 1930.

I will now go see if the 7-11 across the street still has any left...

All the stuff I've sent to Instapaper

Starting before 8am with an international conference call usually means I'll have a full day. Fortunately there's Instapaper, which lets me shove all the interesting things I find during the day onto my Android pad for tomorrow's bus ride to work.

So far today:

And...now back to work.

Why you shouldn't read emails right before going to bed

Because you might learn things neatly summed up in the headline "Petraeus Apparently Most Mentally Balanced Individual in His Own Scandal" "(From the Times …)":

Ms. Kelley, a volunteer with wounded veterans and military families, brought her complaint to a rank-and-file agent she knew from a previous encounter with the F.B.I. office, the official also said. That agent, who had previously pursued a friendship with Ms. Kelley and had earlier sent her shirtless photographs of himself, was “just a conduit” for the complaint, he said. He had no training in cybercrime, was not part of the cyber squad handling the case and was never assigned to the investigation.

But the agent, who was not identified, continued to “nose around” about the case...

Later, the agent became convinced — incorrectly, the official said — that the case had stalled. Because of his “worldview,” as the official put it, he suspected a politically motivated cover-up to protect President Obama. The agent alerted Eric Cantor, the House majority leader, who called the F.B.I. director, Robert S. Mueller III, on Oct. 31 to tell him of the agent’s concerns.

Says Josh Marshall:

So basically this entire scandal both at the outset and in the denouement was driven by Freakshow FBI Agent X who both wanted to bed the victim of the alleged harassment and also decided that the FBI was covering up it’s investigation of the Tampa socialite to protect President Obama. And this because of his “worldview”. Please let us meet this awesome example of American law enforcement.

Now, I understand why David Patraeus lost his security clearance and, thus, his job, because of this affair. I also agree with commentators (including one of my co-workers) who said, "he resigned because of an affair? Really?" I don't think there's anything simple about the DCI conducting an affair using a pseudo-anonymous account on GMail that someone could discover without resorting to Bond-esque espionage.

But I think our nation's top law-enforcement agency, the FBI, deserve props for deciding that an investigation of a top political official that turned up nothing criminal should be announced only after one of the most divisive elections of the last 50 years. Law enforcement should not ever be political. We're a nation of laws, not men, as many have observed. Patraeus acted stupidly, but not criminally, and he's given up his career for it. Let him go quietly into the night.

As for the petty, partisan little man that broke the law to use Patraeus's resignation for political gain, he deserves the undying scrutiny of the U.S. Attorney for whatever district he lives in.

And full disclosure: I believe strongly that the DCI needs to give up certain things upon taking office, including GMail, Facebook, and bits on the side. I have no problem with human failings; but I object in the strongest terms to people in certain offices—Director of Central Intelligence and President of the United States included—engaging in any kind of personal deceit. Professional deceit? No problem with DCI or POTUS; that's part of the job. But Bill Clinton disappointed me deeply, and now, so did David Patraeus. And that's even before we talk about illegal drone strikes.

Was last Tuesday a close shave? Occam says "no"

Talking Points Memo has some smart readers. One of them crystallized the debate about whether the Romney campaign's shock after losing was genuine, or a ploy to inoculate themselves against the wrath of their donors:

There’s an old rule of political research: never ascribe to conspiracy what can be more easily explained by stupidity. Was the Romney campaign brilliant masterminds of a coordinated PR strategy to make themselves look stupid? Or we’re they just stupid?

Occam’s razor says the latter.

For background, read Josh Marshall's angst about how the Romney campaign could have missed the facts so completely, even though the campaign ran on a platform of denying evidence that disagreed with them.

Going, going...

Two Chicago businesses find themselves on the ropes this afternoon, according to the Tribune. The first company, Groupon, has problems many people predicted long before their IPO:

Groupon and its compatriots in the much-hyped daily deals business were supposed to change the very nature of small-business advertising. Instead, it is the daily deal vendors that are racing to change as evidence mounts that their business model is fundamentally flawed.

Critics say the torrid growth that enabled Groupon to go public at $20 a share just a year ago was fueled by merchants buying into a new type of marketing that they didn't fully understand. The discounts offered through the Groupon coupons have turned out to be costly, and the repeat business they generate uncertain.

A Raymond James survey of roughly 115 merchants that used daily deals services during the fall found that 39 percent of merchants said they were not likely to run another Groupon promotion over the next couple of years. The top reasons cited were high commission rate and low rate of repeat customers gained through offering a promotion.

Yeah, no kidding. These flaws have been obvious from Groupon's beginning. I have one outstanding Groupon and one outstanding Living Social deal, and I'm not sure when I'll get to use either. After that, meh.

The other company, Hostess, has hit an unexpected and possibly fatal labor dispute that may force it into out of reorganization and into liquidation:

On Friday, Hostess-employed members of the Bakery, Confectionery, Tobacco Workers and Grain Millers International Union began to strike nationwide, blaming a “horrendous contract” that they claimed could cut wages and benefits up to 32%.

Workers picketed or honored the strike in Sacramento, Los Angeles, Oakland and Seattle and in states such as Ohio, Tennessee, Illinois and Montana. Hostess said in a statement that the walk-offs could lead to layoffs for most of its 18,300-member workforce and a sale of its assets “to the highest bidders.”

“A widespread strike will cause Hostess Brands to liquidate if we are unable to produce or deliver products,” according to the statement. The Irving, Tex.-based company, which was founded in 1930, acknowledged that “the concessions are tough.”

So if you like Twinkies and Ho-Hos, you'd better stock up. (Don't worry; they keep.)

Gotta love Chicago weather

In the past 24 hours, the temperature here dropped from 21°C at 2pm yesterday to -2°C at 8am today. The cold front responsible drove the temperature down 6°C in three hours at one point, making for uncomfortable walks home from dinner.

We had two delightful September days, but now it's November again. The Climate Prediction Center expects normal temperatures in Chicago through the end of January. Winter is coming...

The Romney campaign's software epic fail

Politico's Burns & Haberman explain:

[ORCA] was described as a mega-app for smartphones that would link the more than 30,000 operatives and volunteers involved in get-out-the-vote efforts. PBS profiled it a few days before the election. The app was created and managed by the Romney campaign and was kept a secret among a close circle in Boston, according to POLITICO sources.

It's been reported the system crashed at 4 p.m., but multiple sources familiar with the war room operation said it had actually been crashing throughout the day. Officials mostly got information about votes either from public news sources tracking data, like CNN.com, or by calling the counties for information, the source said. Officials insisted the day after the election that they had still believed they were close, and that they had hit their numbers where they needed to, even as Fox News and other outlets called the race.

The post links back to a Romney volunteer's description of how ORCA failed in the field:

On one of the last conference calls (I believe it was on Saturday night), they told us that our packets would be arriving shortly. Now, there seemed to be a fair amount of confusion about what they meant by "packet". Some people on Twitter were wondering if that meant a packet in the mail or a pdf or what. Finally, my packet arrived at 4PM on Monday afternoon as an emailed 60 page pdf. Nothing came in the mail. Because I was out most of the day, I only got around to seeing it at around 10PM Monday night. So, I sat down and cursed as I would have to print out 60+ pages of instructions and voter rolls on my home printer. ... They expected 75-80 year old veteran volunteers to print out 60+ pages on their home computers? The night before election day? From what I hear, other people had similar experiences. In fact, many volunteers never received their packets at all.

Now a note about the technology itself. For starters, this was billed as an "app" when it was actually a mobile-optimized website (or "web app"). For days I saw people on Twitter saying they couldn't find the app on the Android Market or iTunes and couldn't download it. Well, that's because it didn't exist. It was a website. This created a ton of confusion. Not to mention that they didn't even "turn it on" until 6AM in the morning, so people couldn't properly familiarize themselves with how it worked on their personal phone beforehand.

The project management antipatterns are apparent: Blowhard Jamboree, Smoke and Mirrors, Throw It Over the Wall, and basic Project Mismanagement, for starters. I haven't seen the code, but I can't imagine the management and deployment problems didn't lead to design and construction problems as well.

We software professionals have learned, through painful experience, that software developers have a better understanding of how to develop software than corporate executives. Go figure. Since Mitt Romney ran as a father-knows-best, authoritarian candidate, it should surprise no one that the people he hired couldn't run a software project.