Major Chronicle Bug Fixed

Over the weekend I fixed a major bug in Chronicle's register reconstruction code. The bug was that if an instruction block had an early exit (e.g. a conditional branch that went a different way from what Valgrind expected), register effects associated with the not-executed instructions in the block were still applied as we replayed the register log! This could lead to subtle (or not so subtle) errors in register values returned by the query engine. I pushed the fix to the Subversion repository so if you're using Chronicle you really want to update.

Ozzie Oddity

Fairly standard Microsoft press conference contains one very odd quote from Ray Ozzie:

Chief Software Architect Ray Ozzie, who spearheads Microsoft's push into Web services, argued that leading-edge technologies began with consumers. Microsoft's access to consumers gave it a leg up over rivals focused on serving corporations and large organisations, he said.

Who are Microsoft's key rivals? Apple, Google, maybe Yahoo, maybe Adobe, maybe even Mozilla ... all far more end-user focused and less enterprise-oriented than Microsoft. So what is Ozzie talking about? IBM? Surely that's not who Microsoft fear.

I found this comment particularly ironic since Microsoft tends to favour the interests of enterprise customers over other users, e.g. in the timing of IE updates.

Upcoming Events

A few upcoming events:

• Friday August 3 (that's next week!) I'm signed up to give a Software Engineering Seminar at Auckland University. I'm going to talk about Chronicle and give a demo of the new UI I've been developing. That gives me an incentive to finish off the last couple of killer features, after which I promise to release it! And blog about it of course :-).
  
• Tuesday August 7 I'm visiting the University of Otago and giving a guest lecture in a class on object-oriented programming and UI. I'll be talking about "Developing Firefox": why we do it, why it's hard, how we do it, what works, what doesn't, what we might change, and how people can help us do it. I'll be flying down to Dunedin for the day.
  
• Sunday August 12 to Friday August 19 I'll be in Mountain View, primarily for the MoCo all-hands meeting. Should be fun. On Monday the 13th I'm planning to visit VMWare to catch up with a couple of friends and give the Chronicle talk.
  

Fun times. My not-very-hidden agenda with the university visits is to remind people that the Mozilla team is here and hiring...

Brrrrrr

Today was the last scheduled day for major Web-facing features to be landed for Gecko 1.9. As always this "freeze day" was very exciting as many people raced to get large last-minute patches completed, reviewed and safely landed. There was tree breakage, as always, but in general I felt today was less frantic than freeze days in the past. I'm not sure why, because a lot of cool stuff did land!

I landed my CSS 'text-rendering' patch that I discussed earlier. This means, among other things, that on trunk you can now set browser.display.auto_quality_min_font_size to 0 to get maximum quality text on all sites that don't override text-rendering (i.e., all sites!). So on Linux it's now easy to use Pango for everything --- something I know some people have wanted to be supported properly for a long time.

Eli Friedman landed a patch that exposes "full zoom" functionality from Gecko to chrome; there's no UI yet but it is now fairly trivial to create Firefox UI (or an extension!) to scale all the contents of a Web page. This will be great. This will interact nicely with my text-rendering patch because as you zoom in, the font size in device pixels increases so we'll engage high quality mode automatically. It would be really helpful if there was an extension we could use right now for testing ... hint hint ...

Support for 'oncopy', 'onpaste' and 'oncut' events landed --- that's something I know Web authors have been asking for for a while; thanks to Mathieu Fenniak.

Dave Camp's support for client-side access to the conents of local files via <input type="file"> landed (actually a couple of days ago). This is a really neat feature; instead of script having to upload files to a server to work with them, script can now just read the files locally. This could be cool for Webmail previews of attachments being uploaded, for example, or script could just open local text files and interpret their contents. It is also of course very useful for offline apps. He worked with Hixie on this so I think a spec should show up at WHATWG soon.

We also landed fantasai's patch for "overflow containers". This is a mighty piece of work that significantly extends our frame model. It is designed to handle situations where an element B overflows its parent element A vertically and B does not fit in a constrained-height container (e.g. a page) although A does. We need to place a box for B on the next page but its parent A has no box on the page, so what should the box tree look like? There's no easy answer but we decided the least of evils would be to create an "overflow container" box for A: a sort of ghost box that doesn't map any of A's actual content, but only serves as a container for A's overflowing children. This is a difficult part of our code and I'm very pleased that we have a solution. The best part is that it means we have a fighting chance of fixing in 1.9 the infamous "cannot split absolutely positioned elements" bug --- the bug that means printing a document that's mostly inside an absolutely positioned element only ever prints the first page.

Lots of other stuff landed that I have no time to mention here. Hopefully someone more diligent than I will get around to writing it all down.

It's been a long day --- up at 6am for the Gecko 1.9 phone call, and now it's nearly 1am. But it's a fun day, one of the more exciting days in Gecko development. Someone should make CODE RUSH 2 on corresponding day in the next release cycle!

Chronicle Contribution

Andrew Sutherland has announced some tools he's been building on top of the Chronicle engine. They look cool! Even better, he's got some patches to Chronicle that fix some bugs and add a couple of significant new features to the query engine. I plan to integrate those patches soon, and I'll be using the features in my debugger very soon. Yay open source!

It's neat to see that he's using the engine from Python --- that's one of the possibilities I had in mind when I decided to make the engine export a JSON interface.

Status

A number of interesting things are going on at the moment in my corner of Mozilla-land. While some of my textframe regression fixes wait for reviews, I've been working on a few other items.

I revisited an old bug about spoofing attacks based on having people type into the text input field of a file input control. This had led to the current trunk behaviour where the text input field is disabled and clicking on it pops up the platform file chooser dialog. A better alternative was suggested: allow the field to be focused and manipulated, and pop up the file chooser only when the user actually types in text. I implemented this and it works pretty well.

In my frame display list work early last year, I followed the z-ordering specification in CSS 2.1 appendix E to the letter, drawing 'outlines' very high up in z-order, even above positioned elements in the same stacking context --- which is what appendix E recommends, although it allows for some flexibility about the exact z-ordering of outlines. People complained about this and so after some discussion I moved outlines to be drawn above all regular non-positioned content in the stacking context, but below positioned elements. This matches expectations better and also happens to be compatible with IE and Webkit. Appendix E should probably be altered to designate this as the preferred layer. The good news is that it only requires changing a couple of lines of code --- I'm pleased, because I designed frame display lists with the goal of making this kind of change easy.

I created a patch that implements the 'text-rendering' SVG property for all elements. This allows authors to specify 'text-rendering:optimizeLegibility' to force the highest quality typography the platform supports (i.e. ligatures and kerning, on all our platforms), or 'text-rendering:optimizeSpeed' to select the lowest quality the platform supports (currently no effect on Mac, disables ligatures on Windows, and disables ligatures and kerning for Latin-1 text on Linux). The default setting is 'text-rendering:auto'. In that case we optimize for speed for text up to a certain size, and optimize for quality at or above that size. The idea is that quality is more needed at larger font sizes, and its cost will be relatively small at those sizes. The size is a preference, and setting it to zero means 'auto' will always choose quality. The default is currently 20 device pixels. An extension or even Firefox itself could easily expose this preference as a slider or just a toggle. (Yes, dynamic changes to the preference work correctly.)

'text-rendering' actually only applies to the rendering of DOM text nodes. Other text drawing, such as the drawing of XUL label attributes and page headers and footers, will always take the quality path. I think we should probably add 'text-rendering:optimizeLegibility' to the XUL <window> and <dialog> elements so all chrome looks nice, and also to text input boxes so they look nice.

I've got another patch that fixes a long-standing issue: the glyph bounds problem. The patch tries to avoid performance issues via aggressive caching of glyph bounds. When tight bounds are not needed (the common case where we only care if the glyph extends outside the font-box), we cache just the width of the glyphs that do not overflow the font-box, each packed into 16 bits, all in an array stored in our font object. This is fast and relatively small and allows us to reconstruct the loose bounds for a string in the presence of kerning. When we need tight bounds, or the glyph overflows the font-box, or in certain other corner cases, we cache the exact bounds in a hash table. Performance measurements show that on my Mac the cost of this patch is negligible; on Linux the cost is 1-2% on a standard page load benchmark. That seems pretty good to me, and the Linux number could be improved a little bit by bypassing cairo to measure glyphs, but I still need to get Windows numbers and I have yet to figure out to get stable performance numbers out of a Windows box :-(. Another option is to only enable accurate glyph bounds for 'quality' text as defined by text-rendering above.

There's been interesting work on text line breaking in the last couple of weeks. Masayuki Nakano landed a patch that uses JISx4051-like rules to break Latin-1 text. (We used to use these rules when a CJK character was detected in the vicinity, but that was rather quirky.) The new-textframe work means that these rules are now applied correctly no matter how the text is broken up into inlines. The practical impact is that now for the first time ever, Gecko will break lines after hyphens, slashes, and other punctuation. There's some contextual analysis to avoid silly breaks. I expect a certain amount of tweaking will be required but what we do is mostly IE-compatible so hopefully we'll be OK.

Another great piece of work that just landed was a patch by Theppitak Karoonboonyanan. His patch collects runs of "hard" characters (currently Thai and Lao) and calls Pango to perform line-breaking on them. Thai line-breaking is a hard problem, requiring dictionary-based analysis, and modern Pangos can hook up to libthai to make this happen. This analysis requires unbounded context, but again thanks to the new-textframe work, we can provide all the context needed. So as of yesterday our Thai support should be greatly improved on modern Linux distributions. Someone still needs to write the glue code to hook up Uniscribe and ATSUI (or other libraries?) in the same way. This work is easily extensible to any other "hard" languages we may discover. It's also quite efficient because the new-textframe infrastructure caches the results of line-break analysis in textruns.

We had to reconcile the desire to use platform linebreaking facilities with the need to provide consistent cross-platform linebreaking. Our solution is to use Masayuki's cross-platform linebreaking code for everything except the (few) languages it can't realistically handle. So the breaks between Thai characters will be determined by Pango, and could vary across platforms, but all other breaks will be determined by Gecko and will be consistent across platforms.

Chris Double has made fantastic progress with his work on the <video> element. He's got sound and video working on all platforms, good AV sync, fairly stable, both stored streams and live streaming, and even Javascript volume control. Also I don't think I mentioned it here before, but Karl's landed support for windowless plugins on Linux. We still need plugin authors to actually update their plugins to use it, of course.

In somewhat unrelated news I've made quite a bit of progress on the Chronologer debugger UI. I've got a timeline pane to which you can add queries for the invocation of named functions. I've got a call stack pane and an implementation of history-based stack reconstruction that produces a good call stack for the currently selected time, with display of parameters. I've got a rudimentary data pane that displays the values of variables in memory at the currently selected time. I just need a few more features to have something kinda useful --- a "go back to the last write to this variable" command, and a source-code view with history woven in. I'd like to get these done soon because I've captured a Chronicle trace of Firefox which includes a cycle collector fault, and I'd really like to find a real bug with this stuff :-).

I'm still unsatisfied with all the suggested names, though. My newest idea: Revelation.

Camera Features I Want

My new point-and-shoot camera has something I've been wanting for a long time: an orientation sensor so photos are automatically assigned landscape or portrait orientation. Yay!

I still have a few items on my wish-list:

• Automatic horizon alignment. Maybe there's something wrong with me, but I find it hard to take photos where the horizon is perfectly ... horizontal ... and photos with non-horizontal horizons bug me. This could be fixed: make the camera's orientation sensor reasonably precise, detect when the angle from horizontal is quite small (say five degrees either way), and in that case rotate the image to compensate.
  
• Stability detection. I'd like the camera to have accelerometers so it can wait a little and take a photo precisely during a stationary moment. I think this could really help with photos involving vehicles and shaky hands.
  
• Wifi photo sharing. I'm sick of having to give people my camera for them to take a photo of me, and I'm really sick of big group photos where some photographer has a pile of fifty cameras she has to go through so everyone gets their copy. Solution: give cameras a "share my photos" setting and a "collect other people's shared photos" setting (or combine them into one). When "share my photos" is enabled, every time the camera takes a photo, it's broadcast to all other cameras in range; those cameras with "collect" enabled, collect it. This would be particularly fun with lots of people around taking photos.
  
• Enough dynamic range so I can take a photo of a landscape without whiting out the sky...
  
• All this in a camera that's cheap enough that I don't have to become a photo freak to justify purchasing.
  

Fashionable Controversies

Why are people so upset that the Pope is reaffirming the stance that non-Catholic Christian groups are "defective"? If he didn't think so, would it make sense for him to be Catholic, let alone Pope? You know the old joke...

Look at it another way: Protestants (like me) presumably hold that the Catholic church is "defective" in its own way. Muslims think Christianity is defective. Atheists think it's all defective... We might as well be honest about it (without being unnecessarily abrasive).

I desire Christian unity, but I don't think ignoring over such differences is helpful. I would rather have fellowship with Catholics who take our differences seriously than those who would gloss over them for the sake of ecumenism.

On another note, here's a quite interesting attack by Scott Atran on the anti-religion program proposed (or rather, re-proposed) by Sam Harris, Richard Dawkins and many others. Naturally I would disagree with Atran (an atheist) on lots of things, but it's interesting to see him argue that the proposed program abandons reason, evidence and science.

Holiday Photos

Some photos from last week's trip...

View from Mount Manganui summit, southeast over the town and across the Bay of Plenty. I believe the mountain in the distance is Mount Edgecumbe, maybe 100km away.

View south/southeast from the ridge road near Waimangu, over hills and plains, over Lake Taupo (not visible), all the way to Mt Ruapehu nearly 200km away! You can also see steam rising from some geothermal area, probably Waiotapu.

Classic NZ rural landscape. (I wonder how long it will be before low-end cameras have adequate dynamic range...)

View from the bottom of Waimangu valley, over Lake Rotomahana to Mount Tarawera. Eerie to think of the devastation that poured out of this tranquil basin 121 years ago...

A boiling puddle in Kuirau Park. That's my foot. This is probably a nascent fumarole; most likely it's not really boiling and the bubbles are geothermal gases. But I didn't test it to find out.

The Huka Falls. Here the Waikato river narrows from 100 metres wide and 4 metres deep to 15 metres wide and 10 metres deep, forming a raging cataract. Standard tourist spot but still awesome.

The Hell's Gate mud volcano. It's about two metres high and looks amazingly like Ruapehu. It even experiences similar phenomena, such as lahars (over much smaller time and size scales of course!).

View south from the north end of Hakarimata Scenic Reserve. I'm not sure how far you can see but one of those clusters is Hamilton. The explanatory plaque claimed that it's possible to see Ruapehu but that would be nearly 300km and the conditions were nowhere near good enough.

Rotovegas

Well, I'm back ... from our family trip to Rotorua and nearby regions. Here's a rough breakdown:

Saturday Weather overcast, but at least not raining outside Auckland. Drove from Auckland to Rotorua, via Mt Manganui. I'd never been to the Mount, and it was fun to finally see it. The town seemed fun and the Mount itself was spectacular, although a slightly harder ascent than I expected. The view around the Bay of Plenty was simply amazing. I believe I could see White Island. We arrived in Rotorua in the evening and the town was packed --- a combination of school holidays, "Young Farmer Of The Year" competition, and some mountain biking event.

Sunday Perfect blue-sky weather, although cold. Drove to the Waimangu geothermal valley in the morning; along the way, we had an amazing view across hills and plains all the way to the volcanic plateau and snow-capped Ruapehu. Waimangu claims to be the newest geothermal area in the world, since prior to the 1886 Tarawera eruption, it was just regular scrubland. Now it contains an array of steaming lakes, vents and streams. There's a gentle two-hour walk from the road through the valley to the edge of Lake Rotomahana, then you can catch a bus back to the road. Highly recommended.

The two biggest lakes in the valley are actually vents from the 1886 eruption. In tthat event, first the three main peaks running 6km roughly northeast to southwest erupted, starting from the northeast, and then the eruption continued along that axis with new vents opening for a further 10km. (That included the bed of Lake Rotomahana, which blew out in a massive explosion, destroying the famous Pink and White Terraces, annihilating two nearby Maori villages, and burying most of another village (see below). The explosion was apparently heard as far away as Auckland!) Waimangu is the tail end of that axis; it's clear the volcano is only sleeping!

In the afternoon I thought we might try to reach Lake Waikaremoana in the Urewera mountains to the east of Rotorua, but it became clear that we didn't have time, so instead we visited the remains of Fort Galatea, an outpost during the wars of the late 19th century. There's not much there now but the redoubt's moat and the homestead's chimney are still clearly visible.

Late in the day we returned to Rotorua and visited Kuirau Park. This park is great --- just like a regular city park, except pocked with steaming fumaroles and springs. It's clear the thermal activity keeps shifting --- not all of the fumaroles are fenced. In one place a normal-looking puddle by a path, on closer inspection appeared to be boiling...

Monday The weather prediction was for Monday to be sunny, but it was actually the worst day of our trip, overcast and drizzly. Nevertheless we went ahead with our plan to visit Taupo. The Huka Fulls were impressive as always, and we had a good lunch in town. The southern end of Lake Taupo looked murky but we drove down there anyway, hoping the weather would lift and we'd get a look at at the great volcanoes of the central plateau. It was a nice drive but as we reached the southern end of the lake and climbed up to the plateau it became clear we were just going to be inside a cloud. So we started looking for something else to do. My Tongariro topo map showed a thermal area accessible by track near the tiny town of Waihi, west of Turangi. We went looking for it and found the track, which was overgrown but had been recently used by a four-wheeled bike by the look of it ... but after forty-odd minutes of rather damp walking we gave up and returned to the car. I still wonder where that track leads.

We got back to Taupo too late to visit Craters of the Moon or do anything else so we just packed it in and went back to Rotorua. Not a great day overall, partly due to my poor choices, but the drive around the lake was nice and the kids were quite amused throwing pumice stones from the beach into the lake and watching them float back to shore.

Tuesday Another overcast day but no drizzle --- although away to the north NZ was apparently thrashed by high winds and heavy rain. We kicked off by heading out to Lake Tarawera and visiting the Buried Village --- buried in ash and mud by the 1886 Tarawera eruption that I mentioned earlier. The historical reconstructions were quite good and the Te Wairoa stream and waterfall are also very nice. The kids got to feed rainbow trout in the stream, which I guess is something that first-time visitors to the region really should do!

A couple of the historical characters caught my imagination. One was Alfred Warbrick. Born in NZ in 1860, he toured Britain in a rugby team, but more pertinently, he was close to Mount Tarawera when it erupted --- in a hut with some other hunters --- survived, and then led the first search party to discover what was left of the villages close to the mountain. This must have been incredibly difficult, dangerous and ultimately heart-breaking. He spent most of the rest of his life leading tourists around the area.

Another was Sophia Hinerangi. Born in 1832 or thereabouts she may have witnessed the signing of the treaty of Waitangi. Somehow she survived bearing 17 children and became the principal guide for tourists visiting the Pink and White Terraces. She reported sighting the "phantom canoe" on Lake Tarawera before the eruption. She was in the village of Te Wairoa when it was buried and her whare (house), built stronger than most others, sheltered the majority of the survivors in that village. She eventually died in 1911. What a life!

We had lunch at the tearooms there and returned to Rotorua for a walk around part of the lakefront. The sulphurous waters make it clear where the stench of hydrogen sulphide comes from that pervades the town! There's also a neat little thermal area with fumaroles and hot pools. After that walk we went to Hell's Gate, a commercially run thermal area a bit further away from town. That was pretty interesting; in particular you can get up close to some very vigorous hot springs and a fantastic two-metre-high "mud volcano". At $25 per adult, though, it may be less appealing if you're not a thermophile like me.

Wednesday That's today ... again the weather was overcast but not raining ... quite surprising actually as the storm was supposed to sweep down from the north today. We drove to Hamilton in the morning and had lunch there. Then we continued north but stopped between Ngaruawahia and Huntly for a break, to do a walk that I'd located earlier --- the north end of the Hakarimata Scenic Reserve. It turned out to be really great! It was basically a direct ascent of the northernmost peak of a range of hills, with a bonus visit to a grove of regenerating kauri trees with a couple of huge old specimens that somehow survived logging. The track was a bit muddy in places but the kids handled it very well. The view from the top was excellent, over the Waikato to the north and also the south. Was just a bit too hazy to see all the way down to Ruapehu. Recommended.

After that we finally made it home for dinner and a chance to catch up on email for the first time in five days. Can't say I'm too excited about the latter!

Here's a photo of Emerald Lake, one of the Tarawera vent lakes in Waimangu valley. I'll probably post some more photos later.

Why do people call it Rotovegas anyway? It's almost, but not quite, entirely unlike Las Vegas.

Media Superstar

There's video of me and Chris speaking at the Auckland Web meetup online now.

There's also video of me speaking at mini-Webstock the following week.

I haven't watched the Webstock video yet. I forced myself to sit through the Auckland meetup video. Urgh.

Chris has just submitted the first iteration of a patch for <video> element support in Firefox. I'm really looking forward to the day when those videos can be provided in Theora format and we can just watch them in browsers with no plugins and no proprietary production tools required.

Things I Hate

Web 2.0 Startups ... Seriously, how many startups per week are sending out press releases like

Wallyport is a new social networking convergence service, currently in invitation-only alpha testing. This product aggregates Web 2.0 functionality, tools, creativity and a portal for converging business with user generated content. The premise is to offer diverse services where individuals, groups and businesses can take advantage of proximity and community.

Can anyone take this seriously? How many MySpace, Beebo, Facebook, Twitter clones does the world really need? How many Youtube clones? How many search engines? Why do thousands of "me too" startups get funded?* And who wants to work in one? "We're Youtube ... with voice chat!" Spare me.

There are so many interesting and challenging problems out there. For example, a hundred startups trying different approaches to parallel programming could actually be a useful contribution to the field. Or a few dozen Second Life clones. Personal life recorders. Omniscient debuggers. Web browsers. If I ever get involved in a startup, it had better not ever appear in a "Top 100" list of anything (unless it launched the bandwagon).

* Yes, I know the answer is that by and large VCs are sheep.

XML ... To quote Phil Wadler, "The problem XML solves is not very hard ... and XML doesn't solve it very well."

OK, so a simple generic extensible structured format is useful. Fortunately there's JSON.

Enterprise Software ... The "enterprise" breeds complex, bloated, and boring software. Hello DCE, CORBA, J2EE, WS-* and many more. It seems that if you create a niche for extremely expensive software that doesn't really work, someone will fill it.

(Interestingly, for a long time Microsoft was smart enough to stay out of these tar pits, but they've bought deeply into WS-*. Times change.)

IT ... Computing educators wonder why they have trouble attracting bright students to the field. Maybe it's because smart people aren't inspired by the software bureaucracy that is the domain of most corporate "IT departments". Every time I read about New Zealand's "IT industry" my stomach churns. We need to draw a clear line between IT and the cool stuff, and make sure that attention is focused on the cool side of the line.

Phew. That rant was building up for a while.

Holidays

I plan to be away Saturday to Wednesday on holiday. I also plan to be entirely offline during that period.

Text Status

Time for a quick update on textframe status... The new textframe was turned a couple of weeks ago. There have been some regressions, but I've fixed the big ones, I think. There was a small but significant Tp win (around 5% depending on platform).

Today I checked in a followup patch that fixes a few issues with soft hyphens (&shy;). They should basically work now. That's good news for us and for the Web.

Other than just fixing regressions and other bugs, there are a few followup issues I want to address before beta 1, or possibly after it:

• CVS remove nsTextFrame
  
• Remove nsTextTransformer, which is almost entirely unused now
  
• Remove other unused support code from nsLineLayout and elsewhere
  
• Reland changes to nsILineBreaker implementation that have it use Pango on Linux
  
• Work on my patch to get and cache accurate glyph bounds
  
• Figure out what our story is going to be on quality vs speed
  

The last item's a tricky one. Currently on Windows and Linux we have a "fast path" for Latin-1 text rendering that avoids Uniscribe/Pango, disabling ligatures (and kerning too, on Linux). It would be nice to have ligatures and kerning on, but going through Uniscribe and Pango for everything would be a significant slowdown. (On Mac we're currently going through ATSUI for everything; we could add a Latin-1 fast path if we want to, in which case these considerations would apply to Mac too...) Here's my current thinking:

• Text in chrome documents should default to optimizing for quality. We hardly ever need to layout massive quantities of text in chrome. Also some system fonts on Vista really want ligaturization.
  
• Text in content documents should default to optimizing for speed, for benchmarks and users with slow machines.
  
• But users, especially users with fast machines, should be able to opt for quality.
  
• Perhaps if we dectect the user has a fast machine, we could set them up for quality by default.
  
• We could also give authors control over quality, perhaps by using the SVG text-rendering CSS property.
  

This needs thought and wider discussion.

Mea Maxima Culpa

People have been complaining about poor stability in the latest Firefox 2.0.0 Mac update.

The good news is that I think Mats found the problem.

The bad news is it seems to be entirely my fault.

I guess it says a lot about how cool my job is that with a single mistake I can ruin the experience for millions of people.

Looking at it that way, "sorry" doesn't really cut it. But ... sorry!