The Daily Parker

We're just 45 minutes from releasing a software project to our client for user acceptance testing (UAT), and we're ready. (Of course, there are those 38 "known issues..." But that's what the UAT period is for!)

When I get back from the launch meeting, I'll want to check these out:

• I still haven't seen President Obama's press conference from Monday.
• "I Haven't Sat Down In Six Months Now" is a 25-minute video tour of the ISS.
• An economist makes the case for eliminating parking tickets as a way to manage parking more fairly.
• Shocker: The U.S. sucks for walking.
• A cluster of 73 quasars 4 billion light years across is raising questions about Einstein.
• Dr. Dobbs has a long technical view of President Obama's re-election IT.
• No one is really sure what a kilogram weighs anymore...

Off to the client. Then...bug fixes!

My efforts to move Weather Now up to Microsoft Azure took on some new urgency today when I noticed this:

That particular error code means the RAID battery has less than 24 hours of charge in it. Fortunately, this means only that the disk will slow down if the battery dies, unless there's a sudden power failure, in which case I could lose the entire RAID volume.

This is exactly the sort of thing that made me want to move all my applications to the Cloud in the first place.

I just hope I can finish the port before...well, before ol' Sparky dies...

I'm in the home stretch moving Weather Now to Azure. I've finished the data model, data retrieval code, integration with the existing UI, and the code that parses incoming weather data from NOAA, so now I'm working on inserting that data into the database.

To speed up development, improve the design, and generally make my life easier, I'm using Entity Framework 5.0 with database-first modeling. The problem that consumed me yesterday afternoon and on into this morning has been how to ramp up to realistic volumes of data.

The Worker Role that will go out to NOAA and put weather data where Weather Now can use it will receive somewhere around 60,000 weather reports every hour. Often, NOAA repeats reports; sometimes, NOAA sends truncated copies of reports; sometimes, NOAA sends garbled reports. The GetWeather application (soon to be Azure worker task) has to handle all of that and still function in bursts of up to 10,000 weather reports at once.

The WeatherStore class takes parsed METARs and stores them in the CurrentObservations, PastObservations, and ClimateObservations tables, as appropriate. As I've developed the class, I've written unit tests for each kind of thing it has to do: "Store single report," "Store many reports" (which tests batching them up and inserting them in smaller chunks), "Store duplicate reports," etc. Then yesterday afternoon I wrote an integration test called "Store real-life NOAA file" that took the 600 KB, 25,000-line, 6,077-METAR update NOAA published at 2013-01-01 00:00 UTC, and stuffed it in the database.

Sucker took 900 seconds—15 minutes. In real life, that would mean a complete collapse of the application, because new files come in about every 4 minutes and contain similarly thousands of lines to parse.

This morning, I attached JetBrains dotTrace to the unit test (easy to do since JetBrains ReSharper was running the test), and discovered that 90% of the method's time was spent in—wait for it—DbContext.SaveChanges(). As I dug through the line-by-line tracing, it was obvious Entity Framework was the problem.

I'll save you the steps to figure it out, except to say Stack Overflow is the best thing to happen to software development since the keyboard.

Here's the solution:

using (var db = new AppDataContext())
{
	db.Configuration.AutoDetectChangesEnabled = false;

// do interesting work

	db.SaveChanges();
}

The result: The unit test duration went from 900 seconds to...15. And that is completely acceptable. Total time spent on this performance improvement: 1.25 hours.

Sigh. Zipcar, short-term rental car service that has occasionally made my life a lot easier, just got swallowed by Avis:

Zipcar Inc. has been growing as more people in urban areas forgo owning a car and instead tap car-sharing and hourly rental services when they need a vehicle. The company’s third-quarter sales grew 15% to $78.2 million while its membership (renters) grew 18% to more than 767,000. Zipcar earned $4.3 million in the three-month period and has said it expected 2012 to be the first full year for which it posts a profit.

Avis Budget, the nation’s third-largest car rental company -- after Enterprise Holdings and Hertz -- will pay $12.25 a share in cash for Zipcar, a 49% premium over the stock's closing price Monday.

Avis believes it can whittle $50 million to $70 million of expenses out of the combined operations of the companies by eliminating duplication of functions such as the cost of maintaining Zipcar as a publicly traded company.

Just once I'd like to see a cool, niche company grow to a sustainable size without being acquired by a huge corporation.

I don't know how Amazon figures out what to recommend, nor do I know who's buying what from them. Sometimes I wonder, though, like when it gives me these helpful suggestions:

But of course it's important to have cookies when the revolution comes...

Google Analytics have created some videos imagining if supermarkets were designed like websites:

Heh.

As I've noted before, only one Web application still lives in my living room the Inner Drive Technology Worldwide Data Center: Weather Now. In the last few days, it's showing one more good reason that it needs to get to Windows Azure pronto.

Take a look at my Google Analytics view of incoming visitors:

What is going on? How do I go from 300 daily unique visitors to 1,800 in two days? Take a look at where they're coming from:

Yes, that's right. Close to 40% of Weather Now's traffic came from the Yukon Territory yesterday. And another 40% came from Alaska. And they're all going to this page for some reason. This might be why:

So how does Azure enter into it? Simply, if you have a Web application running on your own server, and you get a 750% increase in traffic, your server may not be able to handle it. Or, worse in a way, you might have been running the server capable of handling the peak load all the time, at great expense in electricity and hardware.

With Azure, you can simply bring another instance online, or increase the size of your running instance, or do any number of things to adapt quickly to the increased load, without having to buy or move the hardware. Then, when the load returns to normal, you can spin down the idle capacity. The trick is, you only pay for the capacity you're actually using.

I'm getting a lot closer to moving Weather Now, but a deadline looming at my paying job tomorrow has my attention at the moment. So more on this stuff later. Meanwhile, if you're in the Yukon or in central Alaska, stay warm, folks!

Three unrelated stories drew my notice this evening:

PATH service has resumed to Hoboken. Some years ago—never mind how long precisely—I lived in Hoboken, N.J., the birthplace of Frank Sinatra (really) and baseball (not really). I took the Port Authority Trans-Hudson train almost every day when I worked in SoHo, and about every third day when I worked in Midtown. Having experienced other ways of commuting to New York—in fact, the switch up to 53rd and Park finally got me to return to Chicago, after my commute stretched to an hour and 15 minutes and required three transit changes—I have a lot of sympathy for the people living in Hoboken and Jersey City who have had to make their ways across the Hudson without the PATH.

In the first days after 9/11, both the PATH and the MTA worried that the Twin Towers' collapse would breach the "bathtub" (the Towers' foundation) and flood both the PATH and the New York subway. No one knew how bad the damage would be, and were thankful when it didn't happen. Eleven years later, Hurricane Sandy showed everyone.

So reading today that the PATH Hoboken to 33rd St. line reopened after seven weeks made me smile. Not as much, I expect, as the thousands of people whose commutes can now return to tolerable lengths.

I'm visiting New York in a few weeks; I'll make sure to post a few photos in homage of the PATH.

Facebook's change to Instagram's terms of service has rightly outraged everyone paying attention. Instagram, a photo-sharing service that Facebook bought recently for $1 bn, this week published new terms of service that allow them to use posted photos any way they want, any time they want. Their goal, not surprisingly, is to make money. The people who use Instagram just want to share their photos with their friends.

The Times quoted Santa Clara University Law Professor Eric Goldman saying, "The interest of the site is never 100 percent aligned with the users, and the divergence inevitably leads to friction. It’s unavoidable." Well, yes, because Instagram's users are not Instagram's customers, as they are just discovering, because the customer is the one who pays you. If you use a service that is free to you, you are not the customer and therefore have nothing to say to the service's owners. I find the flap about Instagram's TOS so interesting because it seems as if none of their users has realized this key point yet.

Instagram swears up and down that the users continue to own their own photos. Of course they do. And of course you keep ownership. But if you post on Instagram, "you hereby grant to Instagram a non-exclusive, fully paid and royalty-free, worldwide, limited license to use, modify, delete from, add to, publicly perform, publicly display, reproduce and translate such Content, including without limitation distributing part or all of the Site in any media formats through any media channel...." So they don't own your photos, exactly, but they can act as if they do.

Under U.S. copyright law, the creator of a work owns it, unless he has signed away the creation right explicitly. (Example: I work for a great software company. I write software for them, under a work-for-hire agreement. Therefore, except for some explicit, written exceptions, all the software I produce that has commercial value is owned by my employer. If I write code in one of my employer's client's applications that makes 10th Magnitude a billion dollars, I don't own it, 10th Magnitude does. That's the deal I made when I took this job. I trust, however, that if I made my boss a billion dollars, he'd share.)

So if you take a photo on your phone, you own it. It's your photo. And Instagram's new TOS says, yes, of course you own it, but we can sell it if we want and pay you nothing.

Now, I've experienced a variety of contractual arrangements in my life as a creative person, so I'm not shocked when someone wants a piece of my income as a fee for finding the income-producing gig. As a software contractor, I've routinely signed away 25% or 30% of my earnings off the top, in exchange for someone else doing the legwork to find the income-producing gig on my behalf. (It's really hard to find gigs while you're working full time on one, it turns out.) And, as someone who hires software contractors now, I expect they'll agree, too. We call this a "commission," as have people in other professions for millennia.

Instagram, effectively, demands a 100% commission off your work. Not only that, but if Instagram finds that one of your photos makes Ansel Adams weep, they can market the crap out of it. You'll never see a dime. Why would someone license the rights from you, when Instagram is selling them cheap? And you can't stop Instagram from destroying the market for your work, because you consented to it by posting your photo.

Let me put it another way. Instagram is saying, "You own your car, of course. But if you park it in our garage, we get to use it as a taxi, without paying you a dime."

To sum up: the people railing against Instagram's new TOS are exactly right. It sucks. And I will never, ever post any of my intellectual property there, even if they change the TOS in response to the approbation they've received, because (repeat after me) I am not their customer.

Finally—and I assure you, this is not related to Instagram—I recoiled in horror at the latest religious stupidity, that the Taliban have started killing anti-polio workers in Afghanistan.

Full disclosure: I was a member of Rotary International for a few years, and I wholly support the organization's amazingly-successful efforts to destroy polio the way we destroyed smallpox. Polio is a sufficiently complex organism that it can't evolve as quickly as we can kill it, making it an ideal target for eradication (like smallpox). But you have to get immunized, and sufficient numbers of your neighbors do, too, or it will keep spreading.

So, these idiot religious fundies, who subscribe to any number of irrational fantasies already, have apparently decided that the people trying to keep their babies from dying of an entirely preventable disease are, in fact, American spies. As the Times reports, "the killings were a serious reversal for the multi-billion-dollar global polio immunization effort, which over the past quarter century has reduced the number of endemic countries from 120 to just three: Pakistan, Afghanistan and Nigeria."

Does anyone else see a coincidence between the three last outposts of a crippling, preventable disease and religious nuttery? Part of Rotary's success, by the way, has been in reassuring local populations that eradicating polio is no more and no less than it seems: a humanitarian effort to end a horrible disease forever. Wars have stopped to allow Rotary and the Gates Foundation to conduct immunizations. But the Taliban do not believe in reason. They would rather have hundreds of their children dead or crippled than accept the possibility that some American- (and British- and French- and Japanese- and South-African- and Namibian- and Saudi- and...) funded organization wants to prevent their children dying or becoming crippled.

Three countries still have polio. They also have air travel. Not everyone in the OECD has polio vaccinations today. So, if I can mention the self-interest of everyone able to read this blog post, who must therefore speak English and have an Internet connection, the religious nutters killing health workers who, but for being shot, would have eradicated a disease that has crippled millions, have made your life more perilous.

</ rant>

All right. Time to walk the dog.

PS: You may need to subscribe to the New York Times to read the linked stories. I apologize if this inconveniences you, but I recommend subscribing anyway. For $15 a month you not only get the entire newspaper online (and on any tablets you own), but you get to feel good about yourself. You also get to live Kant's categorical imperative, by behaving in such a way that the behavior could be universal. Isn't $15 an incentive worth aligning?

Microsoft veteran Raymond Chen explains:

The 64-bit version of Pinball had a pretty nasty bug where the ball would simply pass through other objects like a ghost. In particular, when you started the game, the ball would be delivered to the launcher, and then it would slowly fall towards the bottom of the screen, through the plunger, and out the bottom of the table.

Games tended to be really short.

Two of us tried to debug the program to figure out what was going on, but given that this was code written several years earlier by an outside company, and that nobody at Microsoft ever understood how the code worked (much less still understood it), and that most of the code was completely uncommented, we simply couldn't figure out why the collision detector was not working.

We had several million lines of code still to port, so we couldn't afford to spend days studying the code trying to figure out what obscure floating point rounding error was causing collision detection to fail. We just made the executive decision right there to drop Pinball from the product.

Chen's blog often goes into technical detail that many people might find off-putting, but he's a good person to read if you want to know more about how Microsoft works.

Last night I continued reducing local computing costs by turning off my home desktop PC. The old PC has a ton of space and a lot of applications that my laptop doesn't have, plus a nifty dual-DVI video card. But a couple of things have changed since 2008.

First, my current laptop, a Dell Latitude E6420, has a faster processor, the same amount of RAM, and a solid-state drive, making it about twice as fast as the desktop. Second, Dell has a new, upgraded docking station that will drive two big monitors easily. (Sadly, though the docking station can drive two DVIs, my laptop's video chip can only do one DVI and one VGA.) Third, the laptop uses buttloads less power than the desktop. Fourth, portable terabyte drives are a lot less expensive today than in 2008—and a lot smaller. And finally, I take my laptop to and from work, meaning I have a minor hassle keeping it synchronized with my desktop.

Here's my office about three years ago (January 2010):

A few months later I got a second 24-inch monitor (November 2010 photo):

Notice the printer has moved to make room for the second monitor, but otherwise the setup remains the same. The monitors connect to the desktop under the desk to the left, while the laptop has its own cradle to the right.

Now this afternoon:

The printer has landed on the floor directly under where it used to sit (I print about 3 pages per month, so this isn't the inconvenience it seems), the laptop has moved over to the printer's old spot (and has connected to the monitors), and the old desktop machine sits quietly consuming 225 fewer Watts per hour. I also replaced the 10-year-old, no-longer-functioning 2+1 speaker set with a more compact set. The round thing between the keyboard and the laptop near the center of the photo is a speakerphone that I use with Skype.

I think everyone knows the dog under the desk by now, too. He's not happy that I rearranged his favorite sleeping cave, so I might get a couple of weeks without mounds of dog hair under my desk until he decides the printer is harmless.

So far today I have been unusually productive, whether because of the novelty or because I have a fire-under-the-ass deadline at work. So back to it.