Monday, 18 February 2019

Banning Huawei Is The Right Decision

If China's dictator-for-life Xi Jinping orders Huawei to support Chinese government spying, it's impossible to imagine Huawei resisting. The Chinese government flaunts its ability to detain anyone at any time for any reason.

The argument "no-one has caught Huawei doing anything wrong" (other than stealing technology) misses the point; the concern is about what they might do in the future.

The idea that you can buy equipment from Huawei today and protect it from future hijacking doesn't work. It will need to be maintained and upgraded by Huawei, which will let them add backdoors in the future even if there aren't any (accidental or deliberate) today.

Don't imagine you can inspect their systems to find backdoors. Skilled engineers can insert practically undetectable backdoors at many different levels of a computer system.

These same issues apply to other Chinese technology companies.

These same issues apply to technology companies from other countries, but New Zealand should worry less about technology companies from Western powers. Almost every developed country has much greater rule of law than China has; for example US spy agencies can force tech companies to cooperate using National Security Letters, but those can be challenged in court. We also have to weigh how much we fear the influence of different governments. I think New Zealand should worry a lot less about historically friendly democracies, flawed as they are, than about a ruthless tyranny like the Chinese government with a history of offensive cyberwarfare.

New Zealand and other countries may pay an economic price for such decisions, and I can see scenarios where the Chinese government decides to make an example of us to try to frighten other nations into line. Hopefully that won't happen and we won't be forced to choose between friendship with China and digital sovereignty — but if we have to pick one, we'd better pick digital sovereignty.

It would be easier for Western countries to take the right stand if the US President didn't fawn over dictators, spit on traditional US allies, and impose tariffs on us for no good reason.

Monday, 11 February 2019

Rust's Affine Types Catch An Interesting Bug

A function synchronously downloads a resource from Amazon S3 using a single GetObject request. I want it to automatically retry the download if there's a network error. A wrapper function aws_retry_sync based on futures-retry takes a closure and automatically reruns it if necessary, so the new code looks like this:

pub fn s3_download<W: Write>(
    client: S3Client,
    bucket: String,
    key: String,
    out: W,
) -> io::Result<()> {
    aws_retry_sync(move || {
        let response = client.get_object(...).sync()?;
        if let Some(body) = response.body {
            body.fold(out, |mut out, bytes: Vec| -> io::Result {
                out.write_all(&bytes)?;
                Ok(out)
            })
            .wait()?;
        }
    })
}
This fails to compile for an excellent reason:
error[E0507]: cannot move out of captured variable in an `FnMut` closure
   --> aws-utils/src/lib.rs:194:23
    |
185 |     out: W,
    |     --- captured outer variable
...
194 |             body.fold(out, |mut out, bytes: Vec| -> io::Result {
    |                       ^^^ cannot move out of captured variable in an `FnMut` closure
I.e., the closure can execute more than once, but each time it executes it wants to take ownership of out. Imagine if this compiled ... then if the closure runs once and writes N bytes to out, then the network connection fails and we retry successfully, we would write those N bytes to out again followed by the rest of the data. This would be a subtle and hard to reproduce error.

A retry closure should not have side effects for failed operations and should not, therefore, take ownership of out at all. Instead it should capture data to a buffer which we'll write to out if and only if the entire fetch succeeds. (For large S3 downloads you need parallel downloads of separate ranges, so that network errors only require refetching part of the object, and that approach deserves a separate implementation.)

Ownership types are for more than just memory and thread safety.

Mt Taranaki 2019

Last weekend I climbed Mt Taranaki again. Last time was just me and my kids, but this weekend I had a larger group of ten people — one of my kids and a number of friends from church and elsewhere. We had a range of ages and fitness levels but everyone else was younger than me and we had plans in place in case anyone needed to turn back.

We went this weekend because the weather forecast was excellent. We tried to start the walk at dawn on Saturday but were delayed because the North Egmont Visitor's Centre carpark apparently filled up at 4:30am; everyone arriving after that had to park at the nearest cafe and catch a shuttle to the visitor's centre, so we didn't start until 7:40am.

In short: we had a long hard day, as expected, but everyone made it to the crater, most of us by 12:30pm. Most of our group clambered up to the very summit, and we all made it back safely. Unfortunately clouds set in around the top not long before we go there so there wasn't much of a view, but we had good views much of the rest of the time. You could clearly see Ruapehu, Ngauruhoe and Tongariro to the east, 180km away. It was a really great day. The last of our group got back to the visitor's centre around 6pm.

My kid is six years older than last time and much more experienced at tramping, so this time he was actually the fastest of our entire group. I'm proud of him. I think I found it harder than last time — probably just age. As I got near the summit my knees started to twinge and cramp if I wasn't careful on the big steps up. I was also a bit shorter of breath than I remember from last time. I was faster at going down the scree slope though, definitely the trickiest part of the descent.

On the drive back from New Plymouth yesterday, the part of the group in our car stopped at the "Three Sisters", rock formations on the beach near Highway 3 along the coast. I just saw it on the map and we didn't know what was there, but it turned out to be brilliant. We had a relaxing walk and the beach, surf, rocks and sea-caves were beautiful. Highly recommended — but you need to be there around low tide to walk along the riverbank to the beach and through the caves.

Sunday, 27 January 2019

Experimental Data On Reproducing Intermittent MongoDB Test Failures With rr Chaos Mode

Max Hirschhorn from MongoDB has released some very interesting results from an experiment reproducing intermittent MongoDB test failures using rr chaos mode.

He collected 18 intermittent test failure issues and tried running them 1000 times under the test harness and rr with and without chaos mode. He noted that for 13 of these failures, MongoDB developers were able to make them reproducible on demand with manual study of the failure and trial-and-error insertion of "sleep" calls at relevant points in the code.

Unfortunately rr didn't reproduce any of his 5 not-manually-reproducible failures. However, it did reproduce 9 of the 13 manually reproduced failures. Doing many test runs under rr chaos mode is a lot less developer effort than the manual method, so it's probably a good idea to try running under rr first.

Of the 9 failures reproducible under rr, 3 also reproduced at least once in a 1000 runs without rr (with frequencies 1, 3 and 54). Of course with such low reproduction rates those failures would still be pretty hard to debug with a regular debugger or logging.

The data also shows that rr chaos mode is really effective: in almost all cases where he measured chaos mode vs rr non-chaos or running without rr, rr chaos mode dramatically increased the failure reproduction rate.

The data has some gaps but I think it's particularly valuable because it's been gathered on real-world test failures on an important real-world system, in an application domain where I think rr hasn't been used before. Max has no reason to favour rr, and I had no interaction with him between the start of the experiment and the end. As far as I know there's been no tweaking of rr and no cherry-picking of test cases.

I plan to look into the failures that rr was unable to reproduce to see if we can improve chaos mode to catch them and others like them in the future. He hit at least one rr bug as well.

I've collated the data for easier analysis here:

FailureReproduced manuallyrr-chaos reproductionsregular rr reproductionsno-rr reproductions
BF-9810--0 /1000??
BF-9958Yes71 /10002 /10000 /1000
BF-10932Yes191 /10000 /10000 /1000
BF-10742Yes97 /10000 /10000 /1000
BF-6346Yes0 /10000 /10000 /1000
BF-8424Yes1 /2321 /9730 /1000
BF-7114Yes0 /48??
BF-7588Yes193 /100096 /100054 /1000
BF-7888Yes0 /1000??
BF-8258--0 /636??
BF-8642Yes3 /1000?0 /1000
BF-9248Yes0 /1000??
BF-9426--0 /1000??
BF-9552Yes5 /563??
BF-9864--0 /687??
BF-10729Yes2 /1000?1 /1000
BF-11054Yes7 /1000?3 /1000

Monday, 31 December 2018

Vox On Nietzsche

When I was thinking of becoming a Christian I wanted to read some anti-Christian books. I'd heard Nietzsche was worth reading so I read The Anti-Christ and Twilight Of The Idols. If anything they pushed me towards Christ: rather than presenting arguments against Christianity, they assume it's false and then rant about the implications of that — implications which are wholly unattractive to anyone reluctant to give up on morality. So I can recommend those books to anyone :-).

I was reminded of that by this Vox piece. The author tries to put some distance between Nietzsche and the "alt-right" but only partially succeeds. It's certainly true that atheist alt-righters, in rejecting Jesus but idolizing secular Christendom, have it exactly the wrong way around (though I'm glad they understand Jesus is incompatible with their ideology). It's also correct that Nietzsche argued for demolishing the trappings of Christianity that people hold onto after rejecting Jesus. Unfortunately for the Vox thesis, as far as I read, Nietzsche focused his contempt not on the geopolitics of "Christendom", but (quoting Vox) "egalitarianism, community, humility, charity, and pity". In this, Nietzsche is on the side of Nazis and against progressives and other decent human beings.

The Vox author points out that Nietzsche himself was against racism and anti-Semitism, but those who embrace his philosophy, who "reckon with a world in which there is no foundation for our highest values", can end up anywhere. If you see "egalitarianism, community, humility, charity, and pity" as non-obligatory or contemptible, your prejudices are likely to blossom into racism and worse. Fortunately Nietzsche's philosophy is incompatible with human nature, our imago Dei; intellectuals (both actual and aspiring) pay lip service to "a world in which there is no foundation for our highest values", but they do not and cannot live that way.

Friday, 21 December 2018

Hollyford Track

Previously I recounted our Milford Track trip up to the point where the rest of our group departed, leaving my children and I in Milford. On the morning of December 12 we flew in a light plane from Milford up the coast to Martins Bay; from there we walked inland over the following four days up the Hollyford Valley until we reached the lower end of the Hollyford road.

The flight itself was a great experience. We flew down the Milford Sound to the ocean and turned north to fly up the coast to Martins Bay. We were flying pretty low and got a great view of the Sound, the rugged and relatively inaccessible Fiordland coast, and the bottom end of the Hollyford Valley. Our pilot didn't have other passengers that day, so he brought along his dive gear and went diving at Martins Bay after he dropped us off, leaving his plane parked beside the tiny gravel airstrip.

We walked for about an hour from the airstrip to Martins Bay Hut and spent the rest of the day based there. Probably my best moment of the trip happened nearly right away! I thought I'd try swimming across the Hollyford River to the sandspit, but as soon as I got into the water four dolphins appeared and swam around me for a couple of minutes until, presumably, they got bored. That was an amazing experience and completely unexpected. I felt blessed and privileged. Apparently dolphins and seals often swim from the ocean up the Hollyford River all the way to the head of Lake Mckerrow, which must be around 15km inland.

That day we also visited the Long Reef seal colony about 20 minutes walk from Martins Bay Hut. We were a bit nervous since December is calving time for the seals, and indeed we met a seal on the track who barked at us, sending us running the other way! I also saw, from a distance, a Fiordland crested penguin.

By the evening of that day five other trampers had arrived at Martins Bay Hut, but it's a large hut with plenty of room for up to 24 so it still felt very spacious.

The following day we walked to Hokuri Hut along the shore of Lake Mckerrow and had a relaxing afternoon. It rained, but only after we'd arrived at the hut. (In fact we didn't use our rain jackets at all on the Hollyford Track.) A couple of the trampers from Martins Bay Hut joined us, and we also had a couple coming south from Demon Hut. A group of four visited the hut; they had rafted down the Pyke River and the Hollyford River to Lake Mckerrow and were planning to fly out once they reached Martins Bay. Rather than stay in the hut they camped by the lake. Apparently they saw seals catching fish down there.

On the third day we walked the infamous Demon Trail along Lake Mckerrow to Mckerrow Island Hut. It's several hours of picking one's way over piles of large, slippery rocks. We took it slowly and it didn't bother us, but we were glad to reach the end. We crossed "3-wire bridges" for the first time and mostly enjoyed them.

We'd been warned that Mckerrow Island Hut was dirty and rodent-infested, but despite the hut being a bit old (built in the 1960s) it seemed fine and the location is wonderful — a very short track to a beach with great views down Lake Mckerrow. We saw no sign of rodents, though they may have been deterred because we had six people in the hut that night. Two of them were pack-rafting from the Hollyford road end, down the Hollyford River, out to Martins Bay, then carrying their rafts to Big Bay, over to the Pyke River, and back to the Hollyford confluence.

Our fourth day was pretty easy, about six hours of walking to get to the Hidden Falls Hut. On the fifth day we walked for just two and a half hours to reach the Hollyford Road end, a fine riverside spot to wait for a couple of hours for a shuttle to pick us up.

The Hollyford was a harder walk than a Great Walk, and would have been harder still with less perfect weather, but it was a bit quieter and the Hollyford Valley is just as stunning, so it was well worth doing. As you'd expect the trampers we met were, on average, a lot more hard-core. Apparently we just missed meeting a couple of Chileans who walked from the road to the ocean and back carrying surfboards, which sounds crazy. We met a few guys who had done the pack-rafting round trip from the Hollyford Road end to Martins Bay to Big Bay and back down the Pyke River in just over 24 hours, which is also crazy. We took it relatively easy and I'm happy with that.

Thursday, 20 December 2018

Milford Track 2018

Earlier this month I spent 11 days in the South Island walking the Milford Track and then, after a short break in Milford, the Hollyford Track.

It was my second time on the famous Milford Track. I took my kids again, and this time went with some friends from Auckland Chinese Presbyterian Church. We booked it back in June in the first hour or two after bookings opened for this summer; it's the most popular track in New Zealand and books up very fast. Note that despite being popular, because you have to book, it's not actually busy on the track. There are only 40 unguided walkers allowed per day on each section of track. There are another 40 or so guided walkers staying at the Ultimate Hikes lodges, but they start an hour or two behind the unguided walkers each day, so you seldom see many of them.

Once again we were lucky to have mostly good weather. Unlike last time, the weather on our first day (December 7) was excellent. The boat trip up to the end of Lake Te Anau to the trailhead is a wonderful start to the experience; you feel yourself leaving civilization behind as you enter the Fiordland mountains via the fjords of Lake Te Anau.

Our only rainy day was the third day (out of four), when we crossed Mckinnon Pass. Unfortunately this meant that once again I could not see the view at the pass, which is apparently spectacular on a good day. I guess I'll have to try again sometime! Next time, if the weather's good on day two, I should go as fast as possible up the Clinton Valley to Mintaro Hut, drop my gear there and carry on up to the pass for a look around before returning to Mintaro. I guess a reasonably fit person without a pack can probably get to the top from the hut in an hour and a half.

Bad weather days on these trips don't bother me that much since I will probably be able to go again if I really want to. I feel bad for foreign visitors who are much less likely have that chance!

I did get a chance to explore Lake Mintaro and its streams this time. It's very close to the hut and well worth a walk around.

I'm not very good at identifying wildlife but I think we saw a number of whio (blue ducks). They're still endangered but it appears their numbers are rebounding thanks to the intensive predator trapping going on in the Clinton and Arthur valleys and elsewhere. Apparently it is now quite rare for the trappers to catch stoats there. There is a beech mast this season which will probably mean large-scale aerial poison drops will be needed this winter to keep rats down.

Overall I really enjoyed the time with family and friends, met some interesting people, and thanked God for the beauty of Fiordland both in the sun and in the wet. We had a particularly good time stopping for over an hour at Giant's Gate Falls near the end of the track, where the warmth of the sun and the spray from the falls mostly keep the sandflies at bay.

After we got to Milford on the last day most of our group checked into Milford Lodge and cleaned up. The next day we did a Milford Sound cruise with some kayaking, which was lots of fun. Then the rest of our group bussed out to Te Anau while the kids and I stayed another night before starting the Hollyford Track on December 12. That deserves its own blog post.