Biotimylated

Pickle, slots, attrs, and inheritance

Tue, 02 Jan 2024 00:00:00 +0000

When slotted attrs classes are derived from a non-attrs base class, the attributes of the base class do not survive pickling.

What does this mean? Why would one do such a thing? What can one do about it?

For a contrived example, consider these two classes:

class MyMixin:
    def __init__(self):
        self.mixed = True

@attr.define
class A(MyMixin):
    spam: str = "eggs"

    def __attrs_post_init__(self):
        super().__init__()

Here, MyMixin is just some class that doesn’t depend on attrs.¹

This generally works the way you probably expect; instances of A all get decorated with a mixed attribute.

>>> a = A()
>>> a
A(spam='eggs')
>>> a.mixed
True

Wonderful.

The problems don’t begin until later, when one discovers that one has cause to use Python’s pickle module to serialize an instance of the class A.²

It seems to work:

>>> a_prime = pickle.loads(pickle.dumps(a))  # Round-trip `a` through pickle
>>> a_prime
A(spam='eggs')

But the integration tests break violently! Other code depends on the mixin’s attributes surviving serialization, but the mixin attributes don’t survive serialization:

>>> a_prime.mixed
AttributeError: 'A' object has no attribute 'mixed'

That’s funny. Where’d our attribute go?

The case of the missing member

If we start experimenting, we may notice a couple of things:

mixed gets dropped if we decorate A with @attr.define. But in code where we used the old-style @attr.s instead of @attr.define, instances round-trip through pickle just fine, preserving their mixed attribute. 🧐
If MyMixin is itself an attrs class, the mixed attribute always survives a round-trip, whether we use @attr.define or @attr.s.

If you’ve been following attrs development, you might have recalled that one of the key differences between the “cute” attr.s API and the “next generation” attr.define API is that the next-generation API defaults to creating slotted classes. The slotted data model for classes comes with some performance advantages and better hygiene: it prevents users from creating undeclared attributes on an instance at runtime. With attrs, it also comes with a long list of edge cases, mercifully summarized in the attrs docs. Could slots be to blame?

Indeed, redefining A with @attr.define(slots=False) restores our ability to round-trip the mixed attribute through pickling.

One of the caveats in the attrs docs might stand out on review:

Slotted classes must implement __getstate__ and __setstate__ to be serializable with pickle protocol 0 and 1. Therefore, attrs creates these methods automatically for slotted classes.

Peeking at the implementation of these methods explains our observations: the __getstate__ method that attrs writes for slotted classes works from a list of attrs properties belonging to the type to be serialized and its superclasses. It doesn’t try to include any properties of an instance that weren’t defined as attrs attribs.

How does this interact with inheritance? Mostly, it doesn’t. There is no established mechanism to invoke any __getstate__ or __setstate__ methods associated with a class’s superclasses, and the attrs-generated methods do not try.

These united states

So, what can we do to help pickle find our missing properties?

A few options are:

Make the base type (i.e. MyMixin) an attrs class. This may not be appropriate if the other children of the base type don’t want to use attrs, but it’s very easy and only requires a single line-of-code change.
Somehow customize __setstate__ and __getstate__ on every leaf class we want to pickle. It’s not sufficient to define these on the base type; even though attrs will avoid overwriting these methods if they are already defined on a class, it ignores inherited implementations. You could imagine implementing this with a class decorator or something similarly cute, but that would still require changes at each point of use of the base class, which would make the example mixin much more annoying to use.
Types can define a __reduce__ method which (empirically!) takes precedence over __getstate__ and __setstate__. We can implement __reduce__ on the base type and make it responsible for collecting and setting the right state about itself and any subtypes.

For both options 2 and 3, it’s awkward that MyMixin acquires responsibility for serializing subtypes that it doesn’t know or care about. One possibility is to delegate to any __(get|set)state__ methods that we find on the instance it receives, which must belong to subclasses, and can take responsibility for their own properties.

Writing a reduce method for the base type

__reduce__ is expected to return a weird tuple, of length between 2 and 6. Quoting the pickle docs for each member, with annotations:

1 . A callable object that will be called to create the initial version of the object.

This is usually object.__new__. That’ll work for us.

2 . A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument.

object.__new__ receives the type of the object to create. We can use (type(self),).

3 . Optionally, the object’s state, which will be passed to the object’s __setstate__() method as previously described.

This is effectively where we override __getstate__. The object’s state is held in self.__dict__, and in any slots on either this type or its children. If we assume that all slotted children are attrs classes, we’ll know that know that any subtypes that use slots will have a customized __getstate__() method.

What we can do is:

If __getstate__ exists, call it, and assume that it handles all of the subtype’s responsibilities.
Otherwise, grab the instance __dict__.
Add any state specific to the slots of the base class.

4 . Optionally, an iterator (and not a sequence) yielding successive items. These items will be appended to the object either using obj.append(item) or, in batch, using obj.extend(list_of_items).

We don’t care about this; we can set it to None.

5 . Optionally, an iterator (not a sequence) yielding successive key-value pairs. These items will be stored to the object using obj[key] = value.

We don’t care about this; we can set it to None.

6 . Optionally, a callable with a (obj, state) signature. This callable allows the user to programmatically control the state-updating behavior of a specific object, instead of using obj’s static __setstate__() method. If not None, this callable will have priority over obj’s __setstate__().

This is where we override __setstate__. We get to provide a callable that accepts a brand new instance of the class, and feeds it the state that the instance needs to contain.

If the instance has a __setstate__ method, we can accomplish most of our work by calling it, and then slide in our base-class state on top of it.

Otherwise, we’re responsible for the whole kit and caboodle. This is a little bit subtle in the usual attrs ways; we need to circumvent any property setters and frozen class behaviours. We can do this by using object.__setattr__ to assign each of the properties.

Denouement

A concrete implementation that rescues pickling round-trips for our MyMixin example is:

class MyMixin:
    def __init__(self):
        self.mixed = True

    def __reduce__(self) -> tuple:
        if hasattr(self, "__getstate__"):
            # Subtypes with slots should have customized their pickling behaviour.
            # Rely on the subtype to handle the things it's responsible for.
            state = self.__getstate__()
        else:
            # The subtype must not have slots. Grab its instance dictionary.
            state = self.__dict__
        # Sprinkle our own properties on top:
        if hasattr(self, "mixed"):
            state["mixed"] = self.mixed
        return (
            object.__new__,
            (type(self),),
            state,
            None,
            None,
            _setstate,
        )


def _setstate(obj: object, state: dict[str, Any]) -> None:
    """
    Inspired by https://github.com/python-attrs/attrs/blob/336ea8b691f148b81db62e24bb55c7ca11ddbc6c/src/attr/_make.py#L931
    """
    # Use object.__setattr__ to actually assign in state, to avoid any property setters
    # or frozen class behaviours.
    # I think the precise form of this (instead of repeatedly calling
    # object.__setattr__(obj, key, value)) is a performance microoptimization.
    _bound_setattr = object.__setattr__.__get__(obj)

    if hasattr(obj, "__setstate__"):
        # If the instance has a setstate method, rely on it to handle almost everything.
        obj.__setstate__(state)  # type: ignore
        # Restore our MyMixin state to the object
        if "mixed" in state:
            _bound_setattr("mixed", state["mixed"])
        return

    # The object doesn't know how to restore its state, so we're responsible for it all.
    for k, v in state.items():
        if k == "__weakref__":
            continue
        _bound_setattr(k, v)

@attr.define
class A(MyMixin):
    spam: str = "eggs"

    def __attrs_post_init__(self):
        super().__init__()

Finally, we can write:

>>> a = A()
>>> a_prime = pickle.loads(pickle.dumps(a))
>>> a_prime.mixed
True

Endnotes

1: A “mixin” class provides some functionality to child types, to which the mixin class may not be tightly coupled. It’s not important that the base class here is a mixin, instead of some other kind of base class, but it was the motivating example in the codebase where I encountered this strange circumstance.

2: To avoid slandering pickle, running into problems serializing this data layout isn’t a pickle-specific problem. A glance at the default cattrs representation of an instance of A is revealing:

>>> cattr.unstructure(a)
{'spam': 'eggs'}

There’s no attempt to represent the mixed attribute here, so it follows that you would have to similarly customize how cattrs structures and unstructures this type to preserve any interesting properties belonging to the mixin class.

Vancouver COVID wastewater plot bot

Sat, 16 Apr 2022 00:00:00 +0000

Wastewater sampling is one of the few available unbiased indicators of community prevalence of COVID-19 in Vancouver. @YVRCovidPlots is a community service that shares plots made from Metro Vancouver data to help present recent wastewater sampling in historical context.

Early in the pandemic, I was inspired by BC Public Health Officer Dr. Bonnie Henry’s observation that if you explain to people what you need them to do and give them the resources they need to do it, people will overwhelmingly do the right thing.

Lately, provincial officials have been striking a very different tone. In their last regular COVID-19 press conferences, Dr. Henry, health minister Adrian Dix, and BC premier John Horgan bandied about phrases like “personal responsibility.” Horgan, never known to keep his foot far from his mouth, suggested that people troubled by the withdrawal of mask mandates on public transit could simply use hand sanitizer. I’m not sure where he was going with that one.

I think a valuable way to assess government response to a crisis requiring collective action is to ask what government did to enable people to act. Telling the people of BC that they’re responsible for their own decisions, without using the resources of government to help people access and understand the data that could usefully inform those decisions, is hard to understand except as an abdication of responsibility brought on by an exhaustion of political will. I’m tired, too. COVID isn’t.

This Twitter bot is my small contribution to our collective self-defense. I hope it will help my neighbour Vancouverites make decisions about when it feels right to get out and reconnect, and when it’s time to pull out the good masks before a Skytrain trip. In my wildest dreams, this bot will help demonstrate to our leaders that there’s a demand for improving the quality of this data. If you like it, let them know!

How many people have COVID?

Right now, it’s very hard to understand how many people in BC currently have COVID and how quickly that’s changing.

That’s a problem because any given activity can be more or less safe depending on how many people around you have COVID. Getting a sense of whether COVID rates are high or low, and rising or falling, can help you make better decisions and safer plans.

Case counts are not reliable right now because people in low-risk groups in BC are not eligible for PCR tests, which is the only kind of test the government reports. That means the number of positive tests each day isn’t comparable to the numbers that have been reported in the past.

(If you’re in BC, you think you might have COVID, and you’re not absolutely certain you’re in a low-risk group, you should check the guidelines to find out if you’re eligible for PCR testing and treatment.)

In truth, getting a good estimate of COVID prevalence—the fraction of people in a population who have COVID at any given moment—is always hard. Counting confirmed positive tests always undercounts actual cases because not everybody with COVID gets a test, and sometimes tests give false negative results. Test positivity rates (number of positive tests divided by the total number of tests delivered) can help you understand whether a testing program is missing an unusually high number of cases, but you can’t interpret it as a population prevalence because it’s not a random sample of the population—people who have a reason to get a test are more likely to have COVID than people who don’t.

If you could test everyone, every day, you’d be able to measure prevalence more or less exactly. Of course, doing that with nasal swabs would be economically and politically untenable. You could try to assemble representative samples, using either a panel of volunteers you test regularly or recruiting people going about their daily lives for a quick swab, but most governments don’t seem to feel this is an effective use of public health staffing and laboratory resources.

Wastewater sampling

Another approach that comes close to our ideal for assessing infectious disease prevalence is wastewater sampling, which avoids sampling bias among the sewer-connected population because everyone poops. Taking samples from wastewater processing plant inflows requires relatively little dedicated labor, and running just a handful of laboratory tests gives you information about an entire community.

Wastewater sampling has its own challenges. For example, not all effluent passing through a treatment station came from a toilet. Industrial discharges and flows from stormwater, sinks, and showers can ebb and flow irregularly over the course of a day, which can unpredictably dilute waste in the sampling stream. There are more and less sophisticated ways of correcting for waste flows from other sources. A relatively fancy way is to use other viruses found in human waste as a control: pepper mild mottle virus is a plant RNA virus that passes right through us when we eat peppers. Expressing COVID concentration relative to PMMoV concentration lets you factor out other sources of effluent. A simpler method, used in Vancouver, is to assume that most variation comes from industrial and environmental sources and that treatment plants process about the same amount of human waste each day. Multiplying the measured concentration by the total volume of flows passing through the plant each day yields a daily COVID copy count, which removes the effect of dilution from other sources.

Metro Vancouver

Metro Vancouver is a regional district of BC which delivers services to the Lower Mainland, including water and sewerage. Metro Vancouver has been measuring and reporting COVID-19 levels in wastewater since summer 2020. I think Metro Vancouver and BCCDC deserve praise for providing this public service.

But there are a couple of problems:

Testing frequency is low. Wastewater is only sampled three days a week, on Monday, Wednesday, and Friday.
Testing latency is high. Wednesday, Friday, and Monday results are posted each Friday— so two of the new data points we get each week are already more than a week old.
It’s hard to use the plots on the Metro Vancouver website. Only a few data points from a single plant are visible at a time and the bar plots don’t make it easy to understand changes over time.

Slow and infrequent testing severely limits the usefulness of this data as a way to help people make better decisions. There are plenty of examples of governments doing better. For instance, Massachusetts Water Resources Authority, which serves greater Boston, takes samples daily and has a one-day turnaround. That’s news you can use!

There’s not much I can do about the first two points (contact your city government, Metro Vancouver, and your MLA to let them know you want better wastewater data from Metro Vancouver and BCCDC!) but I do know how to make a plot. Let’s see if we can make the data we do have easier to interpret.

Finding the plot

The first challenge: can we get the data in order to plot it? Pleasantly, Metro Vancouver’s plots page uses dynamic plots, which retrieve the data in the background from a public Sharepoint API. We can access the same API from a script to get machine-readable, readily plottable data.

The next challenge: what’s the right data to plot? Metro Vancouver provides both measured COVID concentration (copies/L) and an estimate of total daily COVID copies (copies/day). The latter is computed from the former by multiplying it by the volume of wastewater that passes through each water treatment plant each day. As discussed above, this is intended to help reduce the impact of dilution from environmental flows on the data. Whether it normalizes the data effectively is unclear; you can imagine a number of problems that could arise depending on the uncertainty of the volume data, the details of the sampling procedure, and the timing of the samples versus the timing of irregular flows. Metro Vancouver doesn’t share details on their sampling process.

In practice, they yield pretty similar looking curves if you put them on the same scales:

Normalization seems unnecessary here, and seems to suppress the signal from the 2nd and 3rd waves seen at Iona Island. I’ll prefer the rawer concentration values instead of the normalized copy values.

The next challenge: what should the plots look like? After playing with a few options, I settled on something similar to MWRA’s approach of presenting both data since the beginning of the pandemic, as well as a subset of recent data, to help give a sense of both global and local context for the latest numbers. I thought using small multiples to present each plant made it easier to appreciate the differences in the regional evolution of the pandemic than overplotting them in different colors.

An annoying problem is that the Omicron peak in January 2022 swamps the earlier waves; I tried applying a log transformation to the y axis but I found the result even harder to interpret.

This is what I ended up with:

The vertical lines indicate the current date, to show the gap since the last measurements.

Finally, how should I share these? I think one shouldn’t have to be a data scientist to be an informed citizen, so I want to get these out in the commons. I’ve been inspired by Vancouver CBC reporter and man-about-town Justin McElroy, who helpfully informed the COVID discussion in BC by making and posting accessible plots. I’m a sucker for a Twitter bot, so I decided to give that a try.

The bot tweets as @YVRCovidPlots, and the script lives on Github.

I declare! Personal infrastructure!

Wed, 22 Dec 2021 00:00:00 +0000

The time had come to rebuild my (circa 2014) VPS, which was running Ubuntu 18.04 and couldn’t be upgraded further. Inspired by glyph’s A Tired Raccoon’s Containerization Manifesto, I decided to try to push as much as I can into containers. Here are a few notes about things I learned on the way.

Declarative configuration management

Part of the ordeal of a migration is re-discovering all of the things I had to learn in order to configure the system the first time. This time, I’m capturing everything I’m changing by using a configuration management tool to set up the services on the host. Vim ist verboten!

I wanted a solution where I’d be able to edit the configuration on my laptop and just mash a button locally to synchronize the state of my VPS with the configuration.

I liked the idea of writing declarative configurations and I don’t like YAML, so between Ansible, Chef, Puppet, and Salt, I chose Puppet.

The onboarding documentation for these tools tends to focus on setting up the infrastructure you need to manage and monitor a whole fleet of servers.

Happily, Puppet’s Bolt tool lets you apply a Puppet manifest to any node you can ssh to, without any persistent management or agent services, which was exactly what I wanted.

Getting Docker services set up and working with Google Artifact Registry required installing the Google Cloud SDK and provisioning it with credentials from a service account. Once that’s in place, running containerized services works out of the box with the docker module.

This is what the manifest ends up looking like: Gist

Most of the arguments to the Docker resources end up being passed through to the docker command line one way or another.

Applying the manifest is as simple as bolt apply manifest.pp --targets myhosts after writing a Bolt inventory.yaml file.

Certificate management

Among the services I run on my server are static Web hosting for my partner and me, and a znc bouncer for libera.chat. I’d like to secure connections to these services with TLS.

The old server served HTTP with Apache and I managed TLS certificates using the Let’s Encrypt client, running a short shell script periodically to sync the refreshed certificates to znc. We can do better!

Caddy is an open-source, memory-safe web server with robust reverse proxy support and simple automatic TLS configuration that knows how to obtain certificates with ACME out of the box.

Using the level4 extension, it can happily provision certificates, terminate TLS, and reverse-proxy for arbitrary streams, including IRC.

Caddy ends up being the only exposed service, forwarding cleartext traffic to znc over the Docker bridge network.

A downside of using level4 is that you can’t use it with the friendly Caddyfile configuration DSL, so I had to figure out how to write configuration in Caddy’s JSON configuration language. This is partly mitigated by the caddy adapt utility, which emits the equivalent JSON for a Caddyfile, which is straightforward to graft into the proper JSON config, so I can continue to describe my web hosting in Caddyfile.

A couple of tricks are:

For the ZNC web interface to work, you have to keep Caddy from advertising http/2 availability in ALPN during the TLS handshake.
Not all IRC clients send SNI, so you need to define a default SNI so that caddy can choose a certificate to send.

The configuration for layer4 ends up looking like:

{
  "apps": {
    "layer4": {
      "servers": {
        "znc": {
          "listen": [
            ":1337"
          ],
          "routes": [
            {
              "handle": [
                {
                  // Terminate TLS
                  "handler": "tls",
                  "connection_policies": [
                    {
                      "alpn": [
                        "http/1.1",
                        "http/1.0"
                      ],
                      "default_sni": "kvm.tds.xyz"
                    }
                  ]
                },
                {
                  // Forward cleartext to ZNC
                  "handler": "proxy",
                  "upstreams": [
                    {
                      "dial": [
                        "znc:2337" // Network address to proxy to
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    }
  }
}

What have we learned?

It’s possible to rub some declarative configuration onto a VPS to make administering personal infrastructure a little more predictable.

Throwing some services into a container and putting Caddy in front is a convenient way to get automatic certificate renewal without too many hijinks.

Death still comes for us all, but at least migrating to a new VPS provider won’t suck as much.

A minimal Gmail API example

Thu, 27 Dec 2018 00:00:00 +0000

Once upon a time, I created a whole lot of gmail labels that I don’t use anymore. I wanted to clean those up in anticipation of Inbox going away. There are a lot of outdated examples for creating a Google API client application, so I wanted to capture the form of the “modern” solution I found.

The solution is captured in a notebook. It uses the google_auth_oauthlib library to create an authorized requests session, and then just does REST-y things instead of using a client library wrapper.

To use it, run the first cell, visit the link, copy-paste the authorization code into the text widget, and then run the rest of the notebook.

Certificate spelunking for fun and profit

Sat, 31 Mar 2018 00:00:00 +0000

Google and others contribute TLS certificates to Certificate Transparency logs as they crawl the web, but there is less systematic effort devoted to discovering certificates from secure, non-HTTPS protocols. I searched the Rapid7 “More SSL” scan log for novel certificates and contributed them to CT logs.

Certificate transparency is an effort to publicly log all publicly trusted TLS certificates. Publishing certificates has a number of benefits, including allowing site owners to detect fraudulent certificates issued in their name, and allowing more comprehensive oversight of CA operations.

Google Chrome will begin requiring CT records to trust newly-issued certificates, beginning in the next month, but historical certificates may not have been logged, and site operators may prefer not to log certificates which are not intended for web browsers.

TLS certificates protecting HTTPS connections are collected by Google’s spider and added to CT logs. I hypothesized that TLS-protected, non-HTTPS services might be an additional source of novel certificates for logging, and could reveal unidentified misissuance.

Rapid7’s Project Sonar scans selected services on the entire IPv4 internet at regular intervals and produces reports containing, among other things, any TLS certificates provided by the scanned hosts. This work is based on the moressl dataset, which contains certificates discovered while crawling services including SMTP, IMAP, and FTP in both TLS-wrapped and STARTSSL modes.

Discovered certificates

This data set did indeed reveal many certificates which had not been previously logged.

The moressl dataset contains 10,341,552 certificates as of Mar 22, 2018
2,172,408 certificates (21%) had valid signatures and chained to roots included in the Mozilla trust store
56,249 trusted certificates (3.6% of trusted certificates) were novel to crt.sh, which has views of many public CT logs
22,893 trusted certificates (40% of novel certificates) were unexpired
2,279 unexpired, trusted certificates were novel but contained a SCT proof, meaning that a precertificate had already been disclosed to CT, but the final, issued certificate had not been logged

This validates the idea that non-HTTPS TLS services are an important source of undisclosed certificates, and confirms the novelty of this analysis. :)

Although CT signature embedding will become more common, which requires that CAs submit precertificates to logs before certificate issuance, some CAs do not have a practice of subsequently logging the final issued certificate containing the embedded CT signature proofs. Although the precertificates are sufficient to identify most cases of misissuance, understanding which proofs are actually embedded in certificates helps assess the impact of decertifying logs, so collecting these issued certificates is still worthwhile even in the presence of a logged precertificate.

cablint results

Running cablint, which audits certificates to the CA/Browser Forum Baseline Requirements, over the unexpired certificates reveals a variety of badly encoded or otherwise non-compliant certificates, presented here in approximate order of interest.

Noncompliance possibly revealed by this work

These certificates were unrevoked at the time of discovery. I believe these issues with these issuers have not been previously discussed, or were considered resolved, or minimally that they were the only results I saw for these CAs on the misissued.com cablint page. Please let me know if you think I shouldn’t take credit for them. :)

I have not applied cablint to all certificates in CT; it is completely possible that other certificates with these problems, from these issuers or others, have already been publicly disclosed. But these were the ones that you couldn’t have seen before yesterday!

Issuer	Last issued	Certificate	Details
GoDaddy	2013	1	≥ 60 month validity period
Digicert	2016	1	IP address encoded as a DNS SAN
ABB (Digicert)	2016	1	CN not in SAN *
NetLock	2017	1, 2, 3	Invalid EKU (KeyAgreement for RSA key)
KPN (Logius/PKIoverheid)	2016	1, 2, 3	Invalid EKU (KeyAgreement for RSA key)
ECCE 001 (Digicert)	2018	1, 2	Invalid string encoding
Tenera (Digicert)	2017	1, 2	Invalid string lengths
Entrust	2015	1	Invalid string encoding

* A certificate with the same problem from the same issuer was revoked after it was disclosed last August.

Noncompliance already visible from previously logged certificates

The HydrantID SSL ICA G2 CA is trusted by Mozilla (via QuoVadis) for TLS authentication, but issues certs intended for IPSEC and which lack serverAuth and clientAuth EKU values, which are not BR-compliant (7.1.2.3.f). These certificates are compliant; zlint was reporting them incorrectly. I regret the error.
Digicert has issued certificates with underscores in DNS names
Microsoft’s CA software inserts trailing NULs in the CPSuri field and fails to flag the KeyUsage extension as critical.

Otherwise notable

Comodo issued certificates in 2017 and 2014 where the CN is a U-label and only the matching A-label is present as a SAN; this situation, but not this certificate, was discussed on m.d.s.p
Thawte very recently revoked two certificates containing only metadata in the subject OU identifier; 1, 2

Discussion

This work shows that non-HTTPS servers present a meaningful volume of undisclosed browser-trusted TLS certificates, and that these certificates contain interesting examples of BR violations.

It is not obvious to me that the rate or distribution of problems in this data set is different than they would be from a random sample of certificates already disclosed to CT, which describes an interesting follow-on experiment. The misissued.com cablint page monitors recent, but not historical, disclosures, and unknown horrors may lurk in CT’s merkled depths. Cthulhu fhtagn!

Happily, systematic evaluations of the quality of historical logged certificates have been undertaken recently, including work presented a few days ago by Oliver Gasser at TU Munich, and an upcoming conference paper by Deepak Kumar at UIUC and colleagues. Their work on logged certificates provides a useful baseline for future comparisons to discovered certificates.

The source code for retrieving and processing scan data is available at https://github.com/tdsmith/tattle.

Other outcomes of this work included a bug report against pyca/cryptography.

Acknowledgements

Thanks to:

Alex Gaynor for helpful discussion and advice. I also used Alex’s ct-tools to submit certificates to CT logs, and his x509-validator informed my chain-building strategy
Alex Gaynor and Paul Kehrer for their work maintaining pyca/cryptography
Comodo and Rob Stradling for operating crt.sh
Amazon and Peter Bowen for cablint
Rapid7 and their collaborators for publishing scan data

Any errors are, of course, mine alone.

Vegan Key Lime Pie-lets

Wed, 14 Mar 2018 00:00:00 +0000

I participate in a vegan cook-off with some friends once a month and I was happy with how my recipe came together for this month’s Pie-Off, so I thought I’d share!

I made key lime pie-lets, consisting of individual firm custards each served on their own graham cracker. There are three components:

Graham crackers, adapted from Nancy Silverton by way of Smitten Kitchen
A simple but delicious raspberry sauce, from Lizzy T
A coconut-milk agar “custard,” by way of my own experimentation :)

The graham cracker adaptations were straightforward:

Earth Balance replaced butter,
salt was omitted, since Earth Balance is quite salty,
coconut milk replaced cow’s milk,
and golden syrup replaced honey.

I took the option to sub in some whole wheat flour and I thought it worked quite well. The dough was quite easy to work, despite Deb’s warnings, particularly as it warmed up. I might skip the chilling step if I made it again. Perhaps I improved the recipe?

The raspberry sauce was made as-is, except that I strained the hot raspberry-sugar mixture through a wire mesh sieve before blending in the cornstarch, to remove most of the seeds. Crushing the raspberries against the sieve with the back of a spoon helps liberate the most raspberry goodness.

The custard was my own invention! Instead of using acid to thicken sweetened condensed milk, like in a traditional key lime pie, the thickener in this recipe is agar. Agar is derived from seaweed, colorless, flavorless, and sets to a firm gel.

At my local Asian-food supermarket, I noticed they stocked both pure agar-agar (Telephone brand), and an agar-agar/sugar mix (Golden Coin brand). The Golden Coin brand is about 10% agar-agar by weight, and is easier to work with when making small quantities of the custard.

Arrowroot is decidedly non-traditional in agar desserts; I added some to see it would help give the custards a creamier texture. I think it helped; see what you think!

I made the custard in individual ramekins. You could also make a large batch by setting the agar in a pan and using a cookie cutter to cut out individual custards.

For each 50 ml custard, you’ll want:

EITHER 6 g Golden Coin agar-agar mix (10% agar/sugar),
OR (600 mg Telephone-brand agar-agar and 5.4 g table sugar)
an additional 4 g sugar
1 g arrowroot powder
25 ml coconut milk
15 ml water
10 ml lime juice
a pinch of lime zest

You can make several at once by scaling up each ingredient, as long as you have enough ramekins. The custard will solidify at room temperature if you leave it for more than a few minutes. (If this happens, you can just reheat it, but heated lime juice may lose some of its pep.)

Combine all ingredients except the lime juice and zest in a microwave-safe measuring cup, and microwave until just boiling and the agar is completely dissolved. Watch it carefully; it will want to boil over!

Stir the lime juice and zest into the hot agar and quickly pour 50 ml into each 2” ramekin. Refrigerate for 10-15 minutes or until solid.

Unmold the agar by carefully running a paring knife around the inner circumference of the ramekin, and then firmly tapping the open end of the ramekin against your palm.

If you like, use a fluted cookie cutter to give each custard a cute edge.

You can store the custards in a closed container in the fridge for a few days; separate them with layers of wax paper.

To serve, just pop a custard onto a graham cracker, and top with the raspberry sauce!

On Vancouver business license data

Sun, 04 Mar 2018 00:00:00 +0000

Yesterday was Open Data BC’s Open Data Day Hackathon, which was a great chance to learn about civic data sets and spend some hacking on them with data nerds and subject matter experts from the city and province.

One interesting data set is the city’s business license registry, which contains data all the way back to 1997.

Vancouver residents are technically required to hold a business license to pursue almost any sort of non-employment income. That includes renting property.

I didn’t quite find a story I wanted to tell with the data, but one surprising trend I noticed was that the number of licenses associated with single-family housing rentals (teal line) has been declining year-over-year.

The trend is repeated in almost every neighborhood except for very built-up neighborhoods with very few single-family structures like the CBD, West End, and Fairview.

Some explanations for this trend could include:

Poor awareness of the responsibility to hold a permit: the number of licenses issued appears much smaller than the number of housing units, so maybe all we’re measuring is decreasing compliance. (I think this is probably the most likely interpretation.)
Densification or multiple-unit conversions reducing the number of single-family structures available to rent. I think this is a factor in some neighborhoods, but I wouldn’t expect the effect to be so uniform across the city.
Increased occupancy of single-family structures by owners, or an increasing number of empty structures, as the expected investment return on the property swamps the expected profit on a rental.

Some more reliable data sets about housing in Vancouver include city building permits, which is only available for CY 2017 forward but promises to be a valuable resource, and CMHC’s information portal, best accessed through Jens von Bergmann’s R package.

Do you have another explanation? What kind of stories would you like to be able to tell about how and where businesses open and close in the city?

Implementing a small language in RPython

Tue, 25 Oct 2016 00:00:00 +0000

Inspired by Alex Gaynor’s “So You Want To Write An Interpreter?” PyCon talk, I decided I wanted to try to write an interpreter! In this post I’ll talk about writing Baby’s First Interpreter, porting my interpreter to RPython, and performing the translation.

My final code lives in a Github repository if you want to follow along.

Like Alex in his talk, I’ll use rply to generate the lexer and parser for my interpreter. Alex’s documentation for rply gives a walkthrough for building a simple calculator parser that generates an abstract syntax tree which can be evaluated by the interpreter.

Before continuing with this blog post, I recommend implementing and playing with the calculator demo to get a sense of how the lexer and parser interact and to get a sense for the feel of an AST class heirarchy.

Looking for a slightly taller hill to climb, I found an old paper [source] by Lawrence A. Coon with iteratively more complex sample languages designed for a compiler design lab class. This post will be about my experience with his eenie language.

NB: Josh Sharp wrote a very helpful introduction that covers some of the same ground!

Lexing eenie

I started writing the front-end for eenie by building the lexer. The eenie lexer is morally similar to the calculator lexer with a couple of key differences:

eenie has identifiers: we need the lexer to generate tokens for the names of variables
eenie has reserved words we need to identify, like program

Handling reserved words is a little tricky with rply’s lexer generator because you need to be able to distinguish a reserved word like has from a legal identifier name like hasty. If you create a rule like lg.add("HAS", "has"), the regex has will match hasty, which is not what you want. The pattern "has$" doesn’t ever match.

rply is based on PLY, so I went looking at PLY’s documentation; PLY essentially lets you define a callback function that’s invoked each time a certain type of token is identified that lets you modify the lexer output. You can solve this problem with PLY by creating an identifier rule and then, in the callback, identify the identifiers that are actually reserved words, and then emit a new token type for them.

rply doesn’t let us define callbacks but we can do something morally similar by intercepting the lexer output after it’s produced and operating on the output a token at a time before feeding it to the parser:

reserved = ["program", "has", "decls", ...]
lg.add("ID", r"[a-zA-Z][a-zA-Z0-9]*")
lexer = lg.build()

...

def lex(corpus):
	for token in lexer.lex(corpus):
		if token.name == "ID" and token.value.lower() in reserved:
			yield Token(token.value.upper(), token.value)
		else:
			yield token

Note that it’s possible to test the lexer without implementing a parser!

Parsing eenie to an AST

The grammar in Appendix A of the eenie paper contains the production rules that we can feed the parser generator to produce our parser. The calculator parser is, again, a good example to begin with.

BNF notation

One notation note is that the BNF for eenie contains rules like:

a ::= a, b | b

which produces an a from either a b, or an existing a plus a b. While translating to rply, in order to generate a product from more than one rule, define an additional generator rule instead of using an or-pipe, like:

@pg.production("a : a COMMA b")
def a_from_ab(p):
	pass

@pg.production("a : b")
def a_from_b(p):
	pass

You can also apply more than one production rule to a single method, like in Alex’s calculator example:

@pg.production('expression : expression PLUS expression')
@pg.production('expression : expression MINUS expression')
@pg.production('expression : expression MUL expression')
@pg.production('expression : expression DIV expression')
def expression_binop(p):
	pass

Note that the canonical eenie grammar specifies separate term, factor, and exp products in order to handle operator precedence. You can use exp for all of these if you add predence rules for the arithmetic operators to the ParserGenerator constructor.

Building up a parser

I found it helpful to begin by implementing the parser in stages.

Declare the production rules and methods but just return the input p from all of them. Now, you can make sure that a) parser generation (parser = pg.build()) works, and b) the parser runs on your example code.
Implement the AST classes and modify the production methods to return instances of those classes.
Implement .eval() on the AST classes.

Evaluating the AST

One feature eenie has, which the calculator language is missing, is variable assignment and reference. A simple way to keep track of these variables is to pass a Python dictionary around as you evaluate the AST which contains the current values of all defined variables.

Therefore, the signature of the eval function is ASTNode.eval(self, context) where context is a dict mapping strings (identifier names) to Numbers (variable values). Variable assignment stores a value in context and identifier references evaluate to the matching value in context.

High five!

My “finished” Python interpreter is here: https://github.com/tdsmith/dummylang/tree/master/eenie.

Porting eenie to RPython

I quickly discovered a number of incompatibilities between RPython and my Python coding style. :) RPython is a subset of Python upon which type inference is possible. The advantage of writing an interpreter in RPython is that the RPython translation toolchain can compile your interpreter and make it Very Fast. The disadvantage is that RPython is much stricter than standard Python. Some of the restrictions are:

Type stability

Functions should always be called with arguments of the same type (except that functions that accept complex types may also accept None)

This means, for example, that you can’t lazily cast something to an int after it’s been passed into a constructor. You always need to give the constructor an int!
Functions should always return a value of the same type, except that a) functions that return complex types may also return None, and b) instances of different classes that inherit from the same superclass are allowed.
Names must hold variables of a single type in any given scope; i.e. a = "foo"; a = Bar() is not allowed.

For the purpose of this paragraph, a “complex type” is any type that is not an integer, float, or boolean.

The things I needed to change about my naïve Python implementation were:

Stop using attrs for my AST classes. :(
Make sure that all production functions returned a descendent of the same superclass; mine was called ASTNode. In particular, they cannot return lists; I had to jury-rig a ASTList class. (While you implement this, note that RPython doesn’t support the __add__ dunder on user-defined classes.)
Make sure that ASTNode.eval() always returns an ASTNode or None, and never a primitive.

Adding and checking static type annotations with mypy is a great intermediate step before trying to translate your interpreter with RPython; the mypy error messages are considerably more cromulent.

Standard library support

The only stdlib modules that have RPython support are math, os, and time. This means that operations on sys.stdin are right out; raw_input is also unimplemented. Instead, use os.read(0, ...) and os.write(1, ...) to interact with stdin and stdout.

Defining a translation target

The RPython translation framework needs some hints about how to invoke your interpreter. This blog post has a useful sample that I stole wholesale.

Perform the translation

Download a pypy source tarball or a hg checkout somewhere and run:

python ~/pypy/rpython/bin/rpython target.py

If all goes well, you will have a fancy compiled interpreter in ./target-c in about 30 seconds!

All did not go well for me. I began getting UnionErrors which reflected that I had called methods using different type signatures. Tracking all of these down took longer than I want to admit in such a small codebase. I think earlier use of mypy would have saved me a lot of angst. You can also just run python target.py prog1.eenie to make sure you should be expecting sane behavior.

Profit???

My RPython eenie interpreter lives here.

I hope this story about my experience was helpful! The RPython toolchain is a powerful tool for language implementors and I look forward to playing with it in greater detail.

Voting in the 2016 General Election: Candidates

Sat, 15 Oct 2016 00:00:00 +0000

As usual, some attempt to share my efforts making an even slightly informed decision about voting. More blather about propositions later.

President: You probably don’t need my help here but um definitely Hillary?

Senate: Kamala Harris; she has almost literally everyone’s endorsement and seems less erratic than Loretta Sanchez, which is good, because this is a Senate seat in California, which means the incumbent’s going to be there until they feel like leaving.

US House, 45th District: Ron Varaseth; Mimi Walters is a terrifying right-winger.

CA Senate, 37th District: I’m voting for Ari Grayson because he’s a Democrat with good things to say about climate change etc and has the left’s endorsements, but I’m doing it with relatively warm feelings about Moorlach; he spends a lot of time beating the fiscal discipline drum and seems genuinely passionate about the work he does. I don’t always agree with Moorlach but I think he’s serving us well!

CA Assembly, 74th District: I’m voting for Democrat and perennial candidate Karina Onofre because I’ve never heard anything good about Matt Harper.

South Orange County Community College District, Governing Board, Area 3

Dana Point Times roundup of the SOCCCD trustee races

Peter J. Espinosa; impressive CV. Emeritus professor at Saddleback College and adjunct at CSU Fullerton. Seems to have lots of experience in university and school administration. Has a campaign website and everything. Decent ratings at RateMyProfessors.
Kimberly Clark; not the paper product conglomerate. “I have served as a school-site administrator and counselor, coordinated programs, and volunteered as a court appointed CASA advocate, education foundation board member, and community volunteer. Currently, I am the director of a 501 C(3) educational non-profit organization.”
Gary V. Miller; retired teacher? affiliated with the nutso American Independent Party. Hard pass.
Barbara “Bobbe” Jay; appointed incumbent, took her deceased husband’s seat. The Saddleback College Lariat has an article about the appointment. Official bio. Allegedly promised not to run for re-election.

Between Clark and Espinosa I’m leaning towards Espinosa based on his extensive experience with the community college system.

South Orange County Community College District, Governing Board, Area 6

James “Jim” R. Wright; incumbent. Official bio; long-experienced professor, dean, and administrator.
Mike Dalati; perpetual candidate, Karina Onofre’s husband. Also running for Irvine Ranch Water District board. No relevant experience?

What the hell is Mike Dalati doing? I’m happy to vote for Wright.

South Orange County Community College District, Governing Board, Area 4

Terri Whitt; appointed incumbent. Official bio; promised not to run in 2016. Award-winning nursing faculty.
John Alpay; corporate lawyer and Capistrano USD board member; OC Political says he’s a Republican.
Jim Leach; government relations exec currently with the Rancho San Margarita water board. Served on the Saddleback College Foundation Board. Apparently associated with OC GOP.

It seems like Whitt and Leach both have relevant experience. I’ll probably vote for Whitt since I think inside experience is good; I’d rather see faculty here than lawyers.

Irvine USD Board

Vote for no more than 3 of:

Mark Newgent; had a fun Twitter beef with him about his awful ISIS-comin’-for-your-kids-gotta-elect-a-warrior messaging. Anyway, don’t vote for him.
Naz Hamid; website, endorsements from Irvine Teachers, OC Democrats, Planned Parenthood, Tom Torlakson, incumbent Bokota
Geri Zollinger; website, Harvey Mudd grad, Duke BME MS, lawyer. Weirdly mad about curriculum.
Betty Carroll; website, Irvine Public Schools Foundation chair, endorsed by incumbent Brooks, California School Employees Association
Paul Bokota; incumbent, Irvine Teachers endorsement
Lauren Brooks; incumbent, Irvine Teachers endorsement

The OC Register very briefly covered a candidate’s forum but it’s pretty redundant with the candidate statements.

Nobody seems to have a beef with the incumbents. Has controversy ever visited this board? Hamid and Carroll both have convincing stories about classroom participation; Zollinger and Newgent don’t. Hamid seems aligned with Democrats and Carroll seems aligned with Republicans. I’ll vote for Bokota, Brooks, and Hamid.

Irvine Mayor

Don Wagner; former Republican assemblyman. Has a committee open to explore an Attorney General run in 2018. Doesn’t even have a website? OC Register endorsement.
Katherine Daigle; perennial candidate, no website, LinkedIn still says she’s running for Assembly. (Daigle was running for 74th AD as a Republican until she lost the primary.)
Gang Chen; some skinny nerd, no government experience, has lots of yard signs. Questionable 1st Amendment credentials. Doing lots of outreach to the Chinese community. Running as a center-right candidate. Wildly attacking Gaido as “too radical” and Wagner as “a career politician”. Did a bunch of stupid saber-rattling about a recall effort in the spring.
David Chey; oh my god, how did he even get on the ballot
Mary Ann Gaido; website, council member 1976 - 1984, Democrat. Lost narrowly to Steven Choi in 2014. Really awful logo. Sierra Club endorsement.

Update: The Irvine candidate’s forum was posted to YouTube.

Update: Check out these interviews at Ask A Leader from KUCI!

I’m skeptical about all of these candidates. I do not understand the hysteria about growth and traffic that seems to be sweeping Irvine. The traffic’s fine. Growth can be healthy.

Understand while evaluating candidates that “Irvine Community News & Views” is a fake newspaper run by Larry Agran lieutenants. “Irvine City News” is likewise a threadbare Republican organ. You might, like me, find this deception upsetting.

The OC Register covered a candidates forum at UCI late last month. I like Daigle’s talk about affordable housing but I don’t think she’s a credible candidate.

Gaido is the only Democrat. Ironically, lots of dark money is being spent to attack Gaido, accusing her of taking too much money. ¯\_(ツ)_/¯ She touts an “endorsement” from Larry Agran’s fake-ass newspaper on her website, which is unacceptably deceptive. I disagree with Gaido’s stance that voters should approve development projects, but it may not matter with an opposed Council.

I’ve historically disagreed with Wagner as a conservative in the Assembly. Wagner doesn’t have a website, which is insulting.

I think Chen, who is campaigning amateurishly but vigorously, is probably drawing votes from both Gaido and Wagner. I expect Daigle and Chey to be non-factors. The OC Reg noted that Gaido has $72k in fundraising, compared to $29k for Chen and $12k for Wagner. Wagner has also been the benefactor of $110k of supporting money and $243k of anti-Gaido money from PACs. Daigle has not solicited funds.

I’d like to see Chen, Gaido, and Wagner at a forum and I’d like to see them answer questions about Irvine’s role in climate change, transit-oriented development with respect to the Master Plan, and what Irvine police should learn from Black Lives Matter. Municipal power is really important and here we are, awash in banalities about traffic and who’s in whose pockets. So far, I’ve found out about two forums after they happened, with no recordings available. I don’t think that should be happening.

I’ll probably vote for Gaido unless I learn something surprising.

City of Irvine Council

Vote for no more than 2 of:

Farrah N. Khan; Democratic Foundation endorsement.
Anthony Kuo; worked for Christina Shea.
Chistina L. Shea; incumbent, Republican endorsement.
Courtney Santos
Ian Daelucian; on the Gang Chen “slate”
Matthew Ehorn; literally some UCI undergrad.
Hyunjoung “Genii” Ahn
Anila Ali
Dale Cheema
Melissa Fox; website, Democratic Foundation endorsement. I’ve spoken to her when she hosted an event in Uni Hills a couple years ago and I was impressed by her knowledge of Irvine’s history and planning. I’m enthusiastic about her opinions about bikes and development. She narrowly missed a seat in the last Council race by just a couple hundred votes. I think she’ll do a great job on the Council.
Shiva Farivar

Update: The Irvine candidate’s forum was posted to YouTube.

Update: Check out these interviews at Ask A Leader from KUCI!

I really want to see Melissa Fox on the Council. I think Christina Shea is likely to be re-elected. Therefore, I’m planning to vote only for Fox, to avoid helping any of the other candidates edge her out. I haven’t done any other research for this race. Khan was recently the target of some unreasonable, illegal, and anonymous yard signs and I’d support her on that basis alone.

Irvine Ranch Water District Directors

Incumbent bios

Vote for no more than 2 of:

John B. Withers; incumbent. Has an empty campaign Facebook page.
Mike Dalati; perennial candidate, also running for South Orange County Community College District governing board, no relevant experience
Mary Aileen Matheis; incumbent President. No Internet presence.
Margaret Brown; official site and Facebook page, currently Director of District Facilities at Garden Grove USD. Running on highlighting board compensation practices. Targeting Matheis. Observing that non-incumbents haven’t been elected to the board since the 70s; typically when directors leave, they’re replaced by an appointee who’s then re-elected.

I think IRWD is doing a good job of managing our water resources. I’ll vote for Brown and Withers, on the theory that a little bit of qualified new blood is rarely a bad idea, especially when incumbents aren’t putting any effort into their campaigns. Don’t vote for Dalati.

Integrating Google Forms and Slack

Wed, 09 Mar 2016 00:00:00 +0000

Getting submissions to a Google Form to appear in a Slack chat is straightforward. You can use services like Zapier to do this, but it turns out to be easy to use Google’s scripting interface to get instant message-pushing without a third-party tool.

This approach is lifted directly from an Obake Labs blog post, but the code didn’t quite work off the shelf for me.

To implement:

Set up a Google Form and link it to a spreadsheet.
Set up a Bot User in Slack and /invite it to the channel you want it to speak in. Make a note of the key.
Go to the response spreadsheet and open the Script Editor (Tools -> Script editor).
Paste in the following code, replacing the values as necessary:

function SendSlackMessage(e){
  // Only include form values that are not blank
  var user_info="";
  for (var key in e.namedValues) {
    var val = e.namedValues[key][0].toString();
    if (val !== "") {
      user_info += key + ': ' + val + '\n';
    }
  }
  var url = "https://slack.com/api/chat.postMessage";
  var payload =
  {
    "token" : "BOT_TOKEN_HERE",
    "as_user" :"true",
    "text" : "New response received:\n" + user_info,
    "channel" : "#general", // or whatever
    "type" : "post",
  };
  var options =
  {
    "method" : "POST",
    "payload" : payload,
    "followRedirects" : false,
    "muteHttpExceptions": true
  };
   
  //Hit the Slack API with the request
  var result = UrlFetchApp.fetch(url, options);
   
  //Check the request went through and log errors if necessary
  if (result.getResponseCode() == 200) {
   var params = JSON.parse(result.getContentText());
   Logger.log(params);
  }
}

Finally, go to Resources -> Current project’s triggers, and add a trigger to run SendSlackMessage on a form submit response from a spreadsheet. Save your project and you should be in business!

I’m looking for ways to get responses in to Google Forms from Slack in a reasonable manner. Botkit’s approach to conversations is the only thing I’ve come across so far; let me know if you have a better way.

Installing botbot.me on Scaleway

Mon, 07 Mar 2016 00:00:00 +0000

BotBot.me is a logging IRC bot. Scaleway provides bare-metal ARM servers for €3/mo. Can we combine them? We can!

Cribbed from the real install instructions, with extensions as necessary.

Instantiate a new server based on the Ubuntu Wily (15.10) image
Install botbot.me dependencies: apt-get install postgresql-contrib-9.4 postgresql-server-dev-9.4 python-dev virtualenv golang-go redis-server git build-essential
Install node.js: curl -sL https://deb.nodesource.com/setup_5.x | bash - && apt-get install -y nodejs
Create a botbot user with adduser botbot --disabled-password
Create a botbot postgres user with sudo -iu postgres createuser -dP botbot and assign a secure password (consider pwgen 24 1). You do need to configure password authentication because of how botbot interacts with lib/pg.
Switch to the botbot user with su botbot, change to the home directory with cd
virtualenv botbot && source botbot/bin/activate
pip install -e git+https://github.com/BotBotMe/botbot-web.git#egg=botbot
cd $VIRTUAL_ENV/src/botbot
make -j4 dependencies NPM_BIN=/usr/bin/npm
Edit .env; set SECRET_KEY and WEB_SECRET_KEY to good random strings (consider pwgen 24 2). Set STORAGE_URL to STORAGE_URL='postgres://botbot:YOUR_PSQL_PASSWORD@localhost/botbot'. Set WEB_PORT='0.0.0.0:8000' or set up appropriate SSH forwarding. Comment out PUSH_STREAM_URL.
Export .env with export $(cat .env | grep -v ^# | xargs)
createdb botbot
As root: echo "create extension hstore" | sudo -iu postgres psql botbot
As botbot again: manage.py migrate
manage.py createsuperuser
honcho start

It’s alive! Proceed to the setup instructions. Note that your bot will not connect to Freenode until you reload the configuration in “add a channel” step 7.

To use a real web server, set up nginx and uwsgi. Install uwsgi into your virtualenv and apt-get install nginx. It will fail; this is fine. Edit /etc/nginx/sites-available/default and comment out listen [::]:80 default_server;. Run apt-get install -f to make dpkg happy.

Add to sites-enabled:

upstream botbot {
	server localhost:8000;
}

server {
	listen 80;
	server_name your_website.com;
	charset utf-8;
	
	client_max_body_size 100M;
	
	location /static {
		alias /home/botbot/botbot/var/static;
	}
	
	location / {
		uwsgi_pass botbot;
		include /etc/nginx/uwsgi_params;
	}
}

Edit the Procfile to launch the web job like uwsgi --socket localhost:8000 --module botbot.wsgi. Run manage.py collectstatic to collect the static files. Run honcho start again and you should be in business!

Ideally, this is where you write systemd configuration files to launch the bot at startup… or you could run it in tmux like me. #devops Someone tell me how to do this and I’ll thank you.

Liberating Irvine campaign finance data

Sat, 26 Sep 2015 00:00:00 +0000

Comrades of the Data Liberation Front rejoice! I come to report a successful action against the campaign finance records of the City of Irvine.

There is a shocking amount of money in Irvine politics. I received, without exaggeration, several pounds of direct mail in the run up to the November 2014 election, and I have very few guesses about who paid for it all. Irvine politicians are required to report donations, loans, and campaign expenditures. Unfortunately and unaccountably, they have not been required to file electronically. Several candidates have, in the past, chosen to file on paper; those filings are not data-entered by the City Clerk’s office, and are posted online as useless PDF page images.

Since the City Clerk didn’t do their job, it’s up to us. I contracted a freelancer to enter the data from the November 2014 election. It took her about 30 hours. I have posted those records on Github. I assert that copyright cannot be held on this data and that these records exist in the public domain. I haven’t done any systematic quality control but the spot checks I’ve done look reasonable. Counterintuitively, donor and vendor mailing addresses are redacted (presumably: by hand, laboriously) from the scanned PDFs, but not from the e-file data.

If you’re passionate about public access and campaign transparency, I’d love to have your help cleaning up and analyzing these records; please get in touch! There’s a lot of work left to do: these should be merged with the e-file data into a single coherent data set, and then it’ll be time to try and generate some analyses of the data. It would be exciting to come up with some newsworthy pitches for one of the woefully under-resourced Orange County news organizations.

Building Python extension modules on OS X with cmake or autotools

Tue, 01 Sep 2015 00:00:00 +0000

tl;dr: Please use -undefined dynamic_lookup instead of -lpython or -framework Python to build Python extension modules on OS X, no matter what python-config says. Only use -lpython or -framework Python if you intend to embed an interpreter.

When a project with a cmake or autotools-based build system decides to start shipping Python bindings, developers often notice that the default behavior of the Apple and Linux linkers is different. On Linux, building with -shared tells the linker to ignore undefined symbols, assuming that the dynamic linker will be able to resolve them at runtime. A consequence is that it’s unnecessary to pass -lpython to the linker when building extension modules.

On OS X, using -shared (or, more correctly, -bundle) on its own will still result in the linker demanding that all symbols are resolved at link time. Some developers work around this by adding -lpython or -framework Python to their linker invocations. (Sometimes, this is done implicitly by trusting the output of python-config --ldflags.) This appears to work but it is not the best solution. The best solution is to provide -bundle -undefined dynamic_lookup, which has the same behavior as the Linux linker.

Why -lpython is wrong

The effect of -lpython or -framework Python is to unnecessarily force the extension module to be used with the Python interpreter against which it was built. Building a module against one Python interpreter and then attempting to import it from a Python interpreter installed to a different location will cause the interpreter to segfault even if the build and import interpreters are ABI compatible. This often appears in practice if someone builds bindings with system Python and then later installs a compatible version of Python from another source (Homebrew, Python.org, pyenv) and tries to import the already-built bindings using their new python. It may also manifest if the first python in $PATH is not what a user thinks it should be during their build. It is a common source of crash reports to projects and to Homebrew, often leading to user and developer frustration and hairy workarounds.

Another consequence is that the compiled module cannot be redistributed in a wheel or bdist – or in Homebrew’s bottles – because those must be portable to different installation locations.

You can detect if a module is explicitly linked to a Python library by running otool -L my_module.so, which will reveal any links to a python library or framework binary. For example, you may see /System/Library/Frameworks/Python.framework/Versions/2.7/Python for a module linked explicitly to system Python. If your module is built correctly, you should see no reference to Python at all.

What to do instead

If you are building an extension module, pass -undefined dynamic_lookup instead of -lpython or -framework Python. If you are worried that the linker will stop complaining about other symbols which may be missing, you can optionally use the -bundle_loader flag, which asks the linker to check that the specified library provides any missing symbols, but does not link to it.

Note that you do need to pass -lpython or -framework Python when linking something that embeds the Python interpreter, but those things are probably not importable.

You should not trust the output of python-config --ldflags or --libs, which give the right answer for embedding an interpreter, but not for building extension modules. (In fact, the respective intentions of python-config and python’s pkgconfig files are not well-defined.)

Indeed, -undefined dynamic_lookup is how Python’s distutils builds extensions (run pip install -vU --force-reinstall --no-use-wheel markupsafe for proof). I’ve helped several projects, including Boost.Python, opencv, pycairo, openimageio, and ledger, change their build systems to match this behavior. Afterwards, these projects and Homebrew see many fewer crash reports from users on OS X.

If you have any questions about this easy way to safely improve the portability of your bindings on OS X, please reach out. I’m tdsmith in #machomebrew and #pypa on Freenode.

Homebrew Python 3.5 transition

Mon, 31 Aug 2015 00:00:00 +0000

Python 3.5 is scheduled to be released on September 13, 2015. Since Homebrew carries the newest stable version of its packages, the Homebrew python3 formula will adopt Python 3.5 very quickly upon its release. This post explains how migrating to Python 3.5 will affect Homebrew users who have set up a Python 3 development environment using Homebrew packages.

Packages installed with pip will not be found and scripts will break

Python packages which were installed against Python 3.4 with pip or easy_install will need to be reinstalled for Python 3.5. Packages installed against Python 3.4 will not be visible from Python 3.5 because Python uses versioned site-packages directories. Scripts (i.e. ipython3) which were installed with pip or easy_install using Python 3.4 will stop working until they are reinstalled.

To reconstitute your Python environment, consider running pip3 freeze > site-requirements.txt before you upgrade to save a list of your installed packages. You should edit this list by hand to remove any packages which are managed by Homebrew. Then, you can run pip3 install -r site-requirements.txt after you upgrade to reinstall your environment.

(If you forget to do this before you upgrade, you can run $(brew --cellar)/python3/3.4.3_2/bin/python3 -m pip freeze to generate the list of packages 3.4 knows about, even after 3.5 is installed.)

Homebrew packages built with `--with-python3` will break

Homebrew will take care of rebuilding the relatively few Homebrew packages which have mandatory or recommended dependencies on "python3" or :python3. If you are only using these packages, you won’t need to do anything. (The complete list of those packages is: circlator, gubbins, iva, keepassc, lensfun, libsigrokdecode, mypy, pastebinit, ponysay, pulseview, py3cairo, pymummer, pyqt5, retext, and xonsh.)

Packages which may be optionally built against python3 by passing --with-python3 will break and will need to be reinstalled by you. You can get a list of packages which were installed using --with-python3 using brew and jq like:

Running brew reinstall for each of those packages will bring your system up to date after the transition. (You can equivalently pipe the output into xargs brew reinstall.)

Continuing to use Python 3.4

Life on the bleeding edge isn’t for everyone. If you would prefer to continue using Python 3.4, please consider uninstalling the python3 formula and using pyenv instead. You can install pyenv with Homebrew with brew install pyenv. Pyenv is a great tool for managing multiple parallel Python installations.

I prefer not to accept Python versions into homebrew-versions because of the relatively high maintenance load, but starting a tap (Homebrew’s term for an alternative formula repository) is easy to do if you’d like to maintain a python34 formula yourself (or a suite of similar formulas, like Felix Krull’s deadsnakes PPA for Ubuntu). I would owe you a beer. :)

Migrating early

If you would like to transition to Python 3.5 before its official release, you can run brew install python3 --devel to install the release candidate. After 3.5 final is released, a normal brew update && brew upgrade will upgrade you to the release version. Anything that works with a pre-release version of 3.5 should continue working with a release version of 3.5.

Questions

If you need help, please feel free to reach out on Homebrew’s Github issue tracker or the mailing list (homebrew@librelist.com). Thanks for your patience!

SDSC Summer Institute Wrapup

Sat, 15 Aug 2015 00:00:00 +0000

What would you do with 2 petaflops of processing power? It’s not an idle question: NSF’s XSEDE infrastructure offers researchers access to large-scale computing resources. I spent the last week at the San Diego Supercomputer Center, which hosts an annual Summer Institute to introduce scientists from many disciplines to high-performance computing technologies. Tissue engineering doesn’t have a lot of petascale problems but I did learn a lot about tools that can help me write more performant code on single and multi-processor systems—even in Python, my favorite programming language. ~And~ I got to take a selfie with a supercomputer.

Faster Python code

We touched on a couple of technologies that can make code faster even on a single core on my laptop. Numba is a neat library that provides just-in-time compilation of regular ol’ numpy-using Python code using LLVM; all you have to do is label your function with a @jit decorator. It’s a great way to speed up code that can’t take advantage of numpy’s fast vectorization.

Here’s a quick demo – note that the slow and jitted implementations are exactly the same, except for the @jit decorator:

#!/usr/bin/env ipython
from numba import jit
import numpy as np

def slow(x):
    sum = 0
    for i in xrange(len(x)-1):
        sum += x[i] * x[i+1]
    return sum

def vectorized(x):
    return (x[1:] * x[0:-1]).sum()

@jit
def jitted(x):
    sum = 0
    for i in xrange(len(x)-1):
        sum += x[i] * x[i+1]
    return sum

x = np.random.random(5000)

%timeit slow(x)
%timeit vectorized(x)
%timeit jitted(x)

Here are the results:

Implementation	Time per loop
slow	1.44 ms
vectorized	7.79 µs
jitted	4.83 µs

The vectorized numpy implementation is 180x faster than the naïve slow implementation, but the jitted version of the naïve implementation is even slightly faster than the numpy version, with no extra work!

To use numba you need to have llvm and llvmlite installed; on Homebrew, all you have to do is brew install homebrew/python/numba. Continuum Analytics also provides a conda package.

GPU processing

GPUs offer massive parallelism. The AMD Radeon M370X card in my laptop provides 10 compute units with 64 stream processors each and GPU architectures can efficiently handle multiple threads. Cluster GPU nodes are even more powerful. While tools and interfaces like OpenCL and Nvidia’s CUDA have eliminated the need to write GPU code in VHDL, leveraging GPUs fully requires understanding their architecture. In the examples we saw, GPU kernels each perform a small calculation on a single location in memory; the outer layer of a for loop is essentially parallelized over all of the GPU cores, each pushing a result into a single cell of GPU memory. CUDA exposes parameters for allocating jobs to GPU cores and getting them right can have important effects on performance. Main memory management on the device is also handled by the user.

Higher-level APIs like OpenACC, which uses OpenMP-like #pragma syntax, promise to make GPU computing more straightforward but vendor and toolchain support are still works in progress. I’m optimistic about ArrayFire, which is a high-level API with Python bindings that abstracts over the GPU’s API, so it works with my AMD card on OS X over an OpenCL interface as well as with Nvidia cluster compute nodes.

A straightforward Arrayfire translation of the same problem looks like:

import arrayfire as af
# use my discrete AMD GPU, not the integrated Isis GPU
af.set_device(1)
def af_vec(x):
    return af.sum(x[:-1] * x[1:])
x = af.randu(500000, 1)
% timeit af_vec(x)

Running on the GPU is slower than the numba version for n < about 500,000, but is faster after that. I’ll look forward to learning which problems can be solved faster on a GPU and how to implement them optimally.

Scaling up, scaling out

I attended a session that touched on OpenMP and MPI, which I’ve always gotten confused before. OpenMP is a classic solution for multi-threaded shared-memory computation on a single node. MPI lets you run multiple independent processes with independent memory spaces on one to n nodes, which can communicate and synchronize with each other through the MPI API. They are often used together. (For the record: OpenMPI is a particular MPI implementation, and has nothing to do with OpenMP!) I don’t think either is right for my purposes right now: they’re powerful and widely deployed but they work most naturally in lower-level languages like C and Fortran, and it’s easy to get burned without a deep appreciation of the challenges and idioms of multiprocessor programming.

While IPython is maybe best known for its notebook view, it also delivers a powerful set of tools for parallel processing, expanding well beyond the functionality of the python multiprocessing module. IPython has built-in support for spinning up cluster jobs on queue managers like Grid Engine as well as over AWS EC2. I think using IPython’s parallel tools will be my first foray into real cluster computing since it provides the infrastructure for moving the single-node multiprocessing I’ve been doing across a HPC cluster in a straightforward way.

Apache Spark is an interesting tool for friendly parallel computing over large data sets. I was worried about Spark because it’s part of the Hadoop ecosystem and my experiences with Hadoop weren’t very pleasant but Spark is much easier to use. Spark leverages the Hadoop distributed file system to enable fast, multi-node access to big data and implements a terrifyingly smart abstraction layer over Hadoop’s map/reduce architecture to make problem-solving more straightforward and performant, including a suite of ML algorithms. I’ll definitely turn to Spark next time I want to deal with machine learning at scale. Andrea Zonca gave a great talk; his slide deck is worth a perusal if you’re even slightly curious.

I should also plug GNU Parallel, which is basically a multiprocessing xargs; it’s the easiest way to spin up a job queue of embarrassingly parallel tasks on a single node.

Resources at SDSC and UCI

SDSC hosts two XSEDE facilities, both of which are architecturally interesting. Comet embodies the “computing for the 99%” strategy; while it provides 2 Pflop/s over nearly 48,000 computing cores, Comet is optimized for jobs that run on 72 or fewer nodes (1,728 or fewer cores) simultaneously. While some models can only be executed by scaling out to a huge number of nodes, most jobs are much smaller; Amdahl’s law suggests that many practical problems encounter sharply decreasing returns beyond a certain degree of parallelism. Scheduling an ensemble of these “mid-sized” jobs can be a very effective way to compute. Comet also hosts 36 GPU nodes, each of which boasts 4 Nvidia compute GPUs.

Gordon, the other facility, provides 341 Tflop/s and vast amounts of on-node fast flash storage for data-intensive processing.

Comet, Gordon, and other XSEDE facilities are available through a (somewhat) competitive allocation process, but small grants for exploration and collecting scalability data for competitive grants are available very quickly.

UC Irvine’s facilities, where I will actually be working, are modest in comparison but still very powerful. Irvine has a small dedicated HPC cluster available to the campus, with 576 cores in 9 nodes available for general use, and another 5,952 cores in 93 condo nodes which are, generously, available to the community for free when they aren’t being used by their owners.

The Summer Institute was a great opportunity to learn from world-class HPC experts and get my hands dirty with some multiprocessing code. If you’re curious about anything I touched on, the slide decks and example code from this year’s session are available on Github.