On-Premises Pernosco Now Available; Reflecting On Application Confinement

In November we announced Pernosco availability for individual developers via our debugging-as-a-service platform. That product requires developers to share binary code, debug symbols and test data with us (but not necessarily source code), and we recognize that many potential customers are not comfortable with that. Therefore we are making Pernosco available to run on-premises. Contact us for a free trial! On-prem pricing is negotiable, but we prefer to charge a fixed amount per month for unlimited usage by a given team. Keep in mind Pernosco's current limitations: applications that work with rr (Linux, x86-64), C/C++/Rust/Ada/V8.

An on-premises customer says:

One of the key takeaways for me in our evaluation is that users keep coming back to pernosco without me pushing for it, and really like it — I have rarely seen such a high adoption rate in a new tool.

To deploy Pernosco on-premises we package it into two containers: the database builder and the application server. You collect an rr trace, run the database builder, then run the application server to export a Web interface to the database. Both steps, especially database building, require a reasonably powerful machine; our hosted service uses c5d.9xlarge instances for database building and a smaller shared i3.4xlarge instance for the application servers. If you want to run your own private shared service you are responsible for any authentication, authorization, public routing to the Web interface, wrangling database storage, etc.

To help our customers feel comfortable using Pernosco on-premises, all our closed-source code is bundled into those two containers, and we provide an open-source Python wrapper which runs those containers with sandboxing to confine it. Our goal is that you should not have to trust anything other than that easily-audited wrapper script (and rr of course, which is open source, but not so easily audited, though lots of people have looked into it). Our database builder container simply reads one subtree of the filesystem and drops a database into it, and the application server reads that subtree and exposes a Web server on a private subnet. Both containers should be incapable of leaking data to the outside world, even if the contents were malicious.

This seems like a natural approach to deploying closed-source software — no-one wants to be the next Solarwinds. In fact even if you receive the entire product's source code from your vendor, you still want confinement because you're not really going to audit it effectively. Therefore I'm surprised to find that this use-case doesn't seem to be well-supported by infrastructure!

For example, Docker provides an embedded DNS server to containers that cannot be disabled. We don't need it, and in theory crafted DNS queries could leak information to the outside world, so I'd like to disable it. But apparently our confinement goal is not considered a valid use-case by Docker. (I guess you can work around this by limiting the DNS service on the container host somehow, but that sucks more.)

Another interesting issue is preventing leaks through our Web application. For example when a user loads our application, it could send messages to public Internet hosts that leak information. We try to prevent this by having our application send CSP headers that deny all non-same-origin access. The problem is, if our application was malicious it could simply change the CSP it sends. Preventing that would seem to require a proxy outside the container that checks or modifies the CSP headers we send.

I'm surprised these problems haven't already been solved, or at least that the solutions aren't widely known and deployed.

What Would Jesus Do ... About Vaccination?

My thoughts about COVID19 vaccination as a Christian are pretty simple (assuming the Pfizer vaccine or something similar):

• Is it safe for me?
  Yes. I'm not known to be allergic to vaccine components or immunologically compromised, and the safety data is solid.
• If I get exposed to COVID19, will the vaccine stop me from getting infected and passing it on to people around me?
  Yes, almost certainly per the data.
• Knowing this, if I chose to not get vaccinated, caught COVID19, and infected people around me with COVID19, would I have disobeyed Jesus' command to love my neighbour?
  Yes.
• Is there any other way I can ensure I won't catch COVID19 and infect others?
  No.
• Are there any countervailing ethical issues with taking the vaccine?
  No. None of the vaccines on offer are closely connected to abortion.

Thus it is pretty clearly God's will for me to be vaccinated.

Mercer Bay

West of Auckland, between Karekare beach and Piha beach, there is a small bay called Mercer Bay. It's surrounded by cliffs and there is no marked track to get down to it, but I've known for a long time that there is an unmarked route down the cliff. Last year I met someone who knows the route and she kindly agreed to guide a group of us down it on Saturday morning.

We started from Karekare up the Cowan track to the turnoff. The cliffs look precipitious and I'm not good with heights, but the route is actually not very difficult, and we got down quite easily. Mercer Bay is very pretty. At dead low tide you can walk around the north end of the beach to an inlet with three large sea-caves. The largest goes a significant distance into the hill to a blowhole where the roof has collapsed. It's incredibly impressive.

Climbing back up the cliff is the sensible but strenuous way out, but at dead low tide you can walk south around the rocks back to Karekare beach, and that's what we did. There is some nontrivial (for me) climbing involved — a few narrow ledges, a few overhangs — but the barnacle-covered conglomerate rocks provide excellent footholds and handholds if you have gloves or don't mind a few scrapes. Unfortunately we did not time our walk well and the tide was coming in, so we had to hurry. In a few places we had to cross little inlets with waves surging in and out, and later on we just plunged in when it wasn't too deep. I don't know how close we came to being trapped, but I certainly would have preferred a larger safety margin. Lesson learned! (I was carrying my personal locator beacon and we may even have had cellphone coverage, so I think we would have been rescued had we taken refuge on the higher rocks, but how embarrassing!)

Anyway, now that I know the way I'm already looking forward to going back :-). If we go around the rocks again I'll make sure we start well before low tide.

Dehydration

Dehydrated food is great for tramping trips (saves weight and is less perishable) but the variety and cost in our local shops is not great, so although I don't like to accumulate many gadgets I bought a dehydrator over Christmas — a Biochef "Arizona" 6-tray unit. I've only used it a few times so far, but I'm very happy with it.

I've dehydrated sliced fruits: apples, pears, peaches, bananas and plums. Different people prefer different fruits but all of them have been well received. The pears are so sweet I feel guilty eating them. The unit can dehydrate at least 12 apples at a time, taking about 8 hours at 63C.

I've made beef and lamb jerky based on this recipe (using some Jack Daniels BBQ sauce I had around instead of "liquid smoke"). The unit can process about 2kg of meat in one run, taking about 6 hours at 70C with 6-7mm thick slices of meat. Longer dehydration times or smaller slices makes the jerky crunchy which is fine but not to everyone's taste. (The strips shown in the photo below are really too thin, because I bought a "stir fry" package from the supermarket with thinner strips than I expected, but they still taste good.)

I was surprised by how easy the process is. I thought it would take some practice to get good results but just about everything I've tried has turned out well.

There are a lot more experiments I want to do. In particular I want to investigate dehydrating vegetables for cooking meals while tramping. Fun!

Tongariro Northern Circuit 2021

Yesterday I got back from a walk on the Tongariro Northern Circuit. Unfortunately things didn't go quite according to plan!

We had intended to walk the circuit over three days, clockwise. On Monday we would walk from Mangatepopo over Mt Tongariro, via the Tongariro Crossing, to Emerald Lakes, where we would turn off and carry on down to Oturere Hut. On Tuesday we'd walk south to Waihohonu Hut, and on Wednesday walk west to Whakapapa to complete most of the circuit. However on Monday high winds were forecast at the top of Tongariro so the Department of Conservation issued a "bad weather" forecast, so the shuttle we had booked would not take us to Mangatepopo, so we couldn't walk from there even if we thought it was safe, which we did not. We talked to staff at the DoC visitor's centre in Whakapapa and eventually decided to walk from Whakapapa east across the Tama Saddle to Waihohonu Hut and then north to Oturere — i.e. do half the circuit anticlockwise from Whakapapa in our first day. On Tuesday we walked south to Waihohonu as planned. On Wednesday we walked out to the Desert Rd and caught a shuttle back to Whakapapa instead of walking back to Whakapapa as planned, because the weather forecast was still poor and there didn't seem much point in re-traversing the saddle into wind and rain.

I thought the first day might be a bit gruelling — about eight hours of walking, on paper, with significant wind and rain forecast. It actually turned out pretty well. Only a little rain fell on us — we seemed to be moving east just ahead of it — and the wind was mostly at our backs. The sun even broke out a few times. No-one complained about the length of the walk and I felt pretty good myself. We reached Oturere Hut after about eight hours but that included our lunch break, a lengthy stop at Waihohonu Hut for a rest and hot drinks, and a side trip to Lower Tama Lake, so we were actually quite fast. One upside of the weather was that Oturere Hut, which is rather small, would have been packed with twenty-six people in good weather but only our group of ten and two other women actually showed up, so it was very comfortable.

On Tuesday the weather was similar — westerly wind and rain — but some of us wanted to do a "side trip" up Oturere Valley to Emerald Lakes if possible, before moving on to Waihohonu Hut — an hour and a half each way. Six of us (out of ten) did it, but it was a bit brutal! It wasn't too bad in the valley — wind and some drizzle in our faces, the spectacular volcanic desert landscape obscured by drifting fog — but the track then climbs steeply up to the saddle with the lakes, and there it was colder and the wind was much stronger. Scrambling up the last, steep part of the path into strong winds and driving rain was no fun at all! (The strong smell of sulphurous gases from the volcano added an extra frisson!) We had a quick look around the lake and the Tongariro Crossing junction and then scrambled right back down again. The walk back down the valley with the wind at our backs was positively pleasant. One good thing about this side trip is that it confirmed we had made the right choice in not risking crossing Tongariro on Monday!

We were able to have lunch in the cosy Oturere Hut and then it was just an easy two and a half hour walk to Waihohonu. Actually it was only mostly easy; in a few especially exposed places we got some very big wind gusts, probably the windiest conditions I've ever walked in. I had to lean hard into the wind to not be blown over, and some in our group just had to squat down and wait for the gusts to pass. I guess it was probably blowing a hundred kilometres an hour.

Once again we had the hut mostly to ourselves — and I think Waihohonu is still the best hut in New Zealand! The two women from Oturere stayed there, and there was also a French woman who was in NZ to work for a few years (arriving eighteen months ago — that was good timing). We had great afternoon and evening — a fire, games, good food and fellowship — and then on Wednesday morning, an early start and a short seventy-five minute walk to the car park to get picked up and returned to Whakapapa.

The weather was certainly disappointing. Ruapehu and Ngauruhoe are beatiful mountains and were entirely covered by cloud the entire time we were there. Of course it was disappointing we couldn't cross Tongariro. On the other hand, I think most or even all people in our group of ten had a good time and have much to be thankful for. If you tramp regularly you have to accept that the weather won't always be good; if you enjoy yourself even in the bad weather, you've got the Right Stuff for tramping :-).

Rees-Dart Track

In the first part of our December tramping trip we did the Kepler Track "Great Walk" with a group of thirteen. Four of that group had a rest day in Queenstown and then, joined by two others, did the Rees-Dart Track December 10-14. This was the first time I've done this track. I chose it because it's the only major track in the Queenstown region that I haven't done. As expected, it's more difficult than any of the Great Walks, but it was amazing!

I originally planned to start in the Rees valley and end in the Dart valley because that's the direction described on the DoC site. DoC staff in Queenstown suggested the reverse direction might be a little better (better views looking up the Dart valley, going up instead of down the steep slope to Dart Hut), and also the weather forecast was for poor weather on December 11, when we'd be crossing the Rees Saddle from the Rees side. However the shuttle company wasn't willing to switch our booking at the last minute so we stuck with starting in the Rees valley. As we shall see this was probably for the best...

So in the morning of the 10th a van picked us up in Queenstown and took us to Glenorchy, with marvellous views along Lake Wakatipu, and there we switched to another van to take us all the way to Muddy Creek where the track officially starts. The driver offered to take us a couple of kilometres further on; a few of us got out so we would walk "the whole track", but the rest took the offer (and took our packs too!).

That first day's walk up the Rees valley was wonderful. The weather was warm and clear. The track through the open land off Rees Station is a bit boggy in places; we noticed some pack-rafters walking up the riverbed instead, but without local knowledge we thought it best to stick to the marked track. There were incredible views as we skirted the flanks of Mt Earnslaw. After a few hours we reached the park boundary and were into the bush, but the track was still easy going. After a while we emerged into clearings with marvellous views of snowcapped peaks and waterfalls all around. This was true for the entire tramp actually so I'm just going to stop mentioning them! We got to Shelter Rock Hut in about six hours, feeling good.

There were five other trampers at Shelter Rock Hut, all heading in the opposite direction to us, i.e. out to the car park the next day. I enjoyed talking to them; two were from Singapore, on working-holiday visas that were extended by several months due to COVID. One of the other three was a volunteer hut maintainer with lots of interesting information to share. As it turned out these were the last trampers we'd be staying with on the entire trip! Before we went to Queenstown I had expected the track to be a lot busier, because I'd met a couple of random people planning to do the Rees-Dart in December, but either that was a fluke or we were early enough in December to beat the rush.

It rained during the night but the next day dawned clear, which was good news as we planned to cross the Rees Saddle. However, just as we were about to leave the hut, it clouded over and snow started to fall! There was a forecast for morning snow to 1200m, but we were at 900m. It wasn't just flurries; the snow thickened and almost immediately started to accumulate as we headed off up the valley. It was lovely, especially for those in our group who hadn't been in snowfall before, but after an hour we were nearing the head of the valley, snow was still falling and there were a couple of inches on the ground. I was seriously wondering about crossing the saddle that day: certainly it would not be ideal to cross a mountain pass with snow falling, given we had no alpine gear and very little experience tramping in snow, and some people had neglected to bring some of the wet-weather gear I had asked for. On the other hand, the (day-old at this point) forecast was for snow to stop before midday and for the temperature to rise during the day, and in fact it was already reasonably warm and there was little wind, so we should not encounter ice. Plus of course we always had the option of turning back, and for worst-case scenarios I always carry an emergency locator beacon.

In fact as we got near the head of the valley the snow did stop falling and the sky cleared, leaving us in a snowy wonderland and making it a fairly easy decision to proceed at least to the top of the saddle and see the condition of the rest of the route. The last climb up to the saddle was a bit tricky but not really a problem. The view at the saddle was incredible. After a long lunch break we carried on down alongside Snowy Creek, and reached Dart Hut a bit later than anticipated but having had a truly exhilarating day.

At this point we discovered one of our group's boot was coming apart. Fortunately, there was a DoC hut warden staying at Dart Hut that night, and he did a great job patching the boot up with fencing wire! That repair lasted for over a day; later we had to resort to wrapping the boot in duct-tape to hold it together for the rest of the trip. A roll of duct-tape is another item I always bring on overnight tramps.

Our third day had brilliant fine weather, as indeed we had for the rest of the trip. As planned we used this day for a day trip up to the end of Dart Valley and then Cascade Saddle. It's a fairly long walk — it took us nine hours, leaving at 9:30am and returning at 6:30pm — but we were able to leave most of our gear at Dart Hut and the walk was truly outstanding. You walk up to where the Dart River emerges from the Dart Glacier, and get incredible views of the glacier and surrounding peaks as you climb up to the saddle. The view from the saddle itself across Matukituki Valley and Mt Aspiring National Park is awe-inspiring; our group was literally gasping "oh oh OH!" as we arrived.

This was the longest day of our trip and in some ways the hardest. The track is well-cairned except for one stretch crossing a slope of broken schist, but we got across that OK (and it was much easier to go back down than I feared). It was hot but water wasn't a problem since there are little streams everywhere (all drinkable) &madsh; even up on the saddle itself!

I think this was also the day when we finally got all our group members playing Bang. (And liking it!)

Day four was a relatively simple walk for six-ish hours down the Dart valley to Daley Flat Hut. I made it unnecessarily hard for myself by not packing my pack properly so it was top-heavy, and for that and a few other niggly reasons I was a little bit grumpy — sorry team! We passed a handful of trampers going the other way but Daley Flat Hut was unoccupied. There were some sandflies inside the hut but after we eliminated them we had a pleasant afternoon. Given it was a hot day we thought we'd try bathing in a pool by the Dart River ... unsurprisingly we could only stand the water very briefly, since it was literally glacial runoff.

That night some of our group got up at 2am to see the stars and the Geminid meteor shower. I wasn't one of them but apparently it was incredibly impressive — minimal light pollution out there!

Our last day was another relatively easy walk to the trail end at the Chinaman's Bluff car park. The track skirts the Sandy Flat area of Dart River which is an amazing lake. We started at 7am to be sure to make our pickup at 2pm, but only took a little over five hours in the end.

Given that the snow turned out to be a big win, I'd say in many ways this is the best tramp I've ever done. Getting all the way to Cascade Saddle is certainly a challenge but definitely a goal worth aiming for if you can get fit for it!

Kepler Track 2020

In what is becoming an annual tradition, I organised an early-December tramping trip in the South Island for friends and family, starting with a reasonably accessible tramp with a large group and followed by a more challenging tramp with a smaller group. This year our accessible tramp was the Kepler Track "Great Walk", December 5 to 8. It was the second Kepler trip for my kids and I, but new to the rest of the group.

This group was thirteen people this year, the largest group I've had to manage yet! There was a big range of ages and tramping experience, including a first-time overnight tramper. There was one person I'd never met before (a friend of a friend). We had seven men and six women. It was all a good mix and I thank God that, from my perspective at least, the group dynamics worked well.

In previous years with groups of ten I've found it difficult to keep track of who's carrying which supplies, especially food. This year I tried to mitigate those problems by splitting the large group into three subgroups and having the members of each subgroup carry supplies for their subgroup. This worked well. I had planned for the subgroups to actually cook independently but we ended up sharing cooking work across subgroups, which was a bit chaotic but still worked well with everyone eager to pitch in as needed. I mixed up the membership of the subgroups so that people got to know each other a bit more.

The weather forecast was looking pretty bad a week out but we ended up getting quite good weather, especially after the first day. That day was made more interesting because it also happened to be the day of the Kepler Challenge! That race starts at 6am and normally the runners run the whole Kepler Track the same direction we were going, thus would have been well ahead of us the whole way. However, due to the bad weather forecast they ran only as far as Luxmore Hut, then turned around and came back to the start at Lake Te Anau's control gates (followed by a run down to Moturau Hut and back). So, after we walked from town to the control gates (spotting the takahē at the sanctuary along the way) and got onto the Kepler Track proper, we were passed by hundreds of runners coming in the other direction. It was a little annoying but quite interesting, and I'm glad I wasn't running it myself!

It drizzled all morning and we had to stop for lunch in the rain, which was a slight downer, but after we got above the bushline the weather cleared up, we got some excellent views over Lake Te Anau, and everyone cheered right up. Pretty soon after that we got to Luxmore Hut and enjoyed a pleasant afternoon. We visited Luxmore Cave and were entertained by the antics of a kea on the deck. For dinner we had our current favourite first-night meal: sausages, fried onions and buttered bread.

On the second day we had our usual first breakfast of bacon and eggs. Carrying eggs in their cartons at the top of a few people's packs works surprisingly well. We had a marvellous clear day walking across the Kepler tops, and not much wind either, except up Mt Luxmore and later on the descent into Iris Burn valley. We encountered keas close up again at the Forest Burn and Hanging Valley shelters. Really we were exceptionally fortunate because the views were outstanding and the walk pleasant; on a windy, wet day, the tops could be a very unpleasant environment indeed. That night at Iris Burn Hut we had pasta with canned tuna, sundried tomato pesto and grated Parmesan cheese — our new second-night favourite.

The third day of the Kepler is an easy walk down to Moturau Hut and we got there early in the afternoon. It was a lovely sunny day and the whole group got to laze around at the fine beach next to the hut. A number of us braved the waters of Lake Manapouri — cold, but endurable to the point where after fifteen minutes or so it starts feeling OK! The swim was highly refreshing and made more enjoyable by the stunning views around the lake. Dinner was instant noodles and there was plenty of time for more relaxed socialising at the beach as the sun set around 9pm.

The last day was an even easier walk from Moturau back to the control gates and then into town to pick up our bags and take a shuttle back to Queenstown. Once again the weather was superb, sunny but shady in the bush and then windy to keep us cool in the open as we walked back to town.

It was a wonderful trip and I believe our whole group enjoyed it very much. A couple of people had some small fitness issues but were feeling much better by the end. Anyone who didn't enjoy it should assume tramping isn't for them! Coordinating such a large group was a bit stressful for me at times; part of the problem is that I'm not naturally good at socialising in large groups, especially when I don't know some of them very well, so occasionally I had to wander off on my own to pray and decompress a bit. Overall though I was very happy.

A couple of times it was appropriate to remind everyone what a privilege it is to be able to do a trip like this worry-free, while much of the rest of the world is suffering in various ways, and how grateful we should all be for that — we did not earn it!