Friday, February 25, 2011

Apparently, You CAN Break the Internet

Nice day, little overcast and chilly, but no weather in sight -- no freezing rain to break power lines, no heavy rains to flood places, no snow, no tornadoes.

Imagine our surprise when the first text comes in: the power is out at the ADSB building.

Huh? No, seriously, why is the wireless down. Wait, the wired connections are down. Text from the track and Barnhill -- no internet.

Right in the start of the 8th inning for baseball, the close of the multis for SEC Indoor Track and the warm-ups for gymnastics hosting #1 Florida.

What to do? Well, grab your trusty iPhone and keep the CoverItLive blog going at least, and start making Tweets by phone to alert fans that yes, you have lost your RazorVision streams, but no, this one is beyond our ability to reset or fix.

My top priorities were to keep the fans out there informed, and to do our best to keep results moving. Especially the baseball game, that had just seen Arkansas rally to a 3-2 lead into the top of the ninth.

This is one of those times your drills and plans come in handy. We weren't able to stream gymnastics most of the night while the repairs were being made, and we never got official word on what happened. Fortunately, we've got folks who keep us connected, and that was critical. Why? Because if we aren't out front to let folks know, it's just another "fail" by RazorVision. That's exactly why we jumped immediately on the connection crippled blog, our own Twitter and one of the major gymnastics boards to alert them -- hey, this is act of God stuff.

It helped some when the wave of FIX RAZORVISION (yes, lots of all caps yelling) started. We were upfront and told the viewers -- hey, this is likely a replay night. Unfortunate as we upset #1 Florida, but some folks got to see the final rotation live, and the replay is going up now as I type.

Next step was to reach out to our message board community with this message under my name:

This applies to track, baseball and gymnastics -- the internet literally broke on campus. A power outage around 5 p.m. at one of the main hubs for the campus (and the main server room) tripped the backups and then something went horribly wrong. I don't have anything official, just know we lost internet at Baum, Tyson, Barnhill for most of the late afternoon and early evening while it got solved. We kept the blogs going off cell phone cards, but the live video streams died for lack of hardwire connection.

I've made provisions to make sure we can keep the blogs up for tomorrow, and all the events were captured and we're loading them up now that we've got some connectivity back.

Sorry about that -- I know we had a lot of folks talking about it on the boards and blogs -- but this was a freak catastrophic failure that was out of all of our hands.

When I made my first Tweet about connection trouble, I know I may have been breaking the news for the whole campus, but in real-time I had to make a quick decision. It's FEMA Public Information Officer 101: get information out to people in a crisis that will allow them to make good decisions, but make sure your information is accurate and give no more than what you can confirm. It's the old line the taught us in the ICS PIO courses: Trust in God, verify everyone else. And in this case, we had confirmation from ADSB direct of what was happening. Notice I didn't speculate about repair time, or give the unconfirmed other details about what did or didn't happen with the power backup -- that seems still a bit up in the air.

All from desktop to iPhone back to desktop then on to cell phone card -- our Sprint Overdrive cell access point that has now saved our bacon more than once this year (if you don't have at least one, go get one tomorrow; I with I had two or three).

Very, very proud of our team -- new media kept rolling while the rest of this was kind of chaos and then rippling through other stuff. We didn't lose the replay, we kept the blogs flowing with low bandwidth solutions and -- hey -- we beat #1.

Not a bad night's work.

Let's go do it again tomorrow!

Except without the breaking the internet thing. That wasn't cool.

No comments: