Justice and the Conscience

I do not pretend to understand the moral universe, the arc is a long one, my eye reaches but little ways.

I cannot calculate the curve and complete the figure by the experience of sight; I can divine it by conscience.

But from what I see I am sure it bends towards justice.

Things refuse to be mismanaged long.

Theodore Parker, 1853 calling for abolition of American slavery

“Let us realize the arc of the moral universe is long, but it bends toward justice.”

Martin Luther King Jr. 1964 on struggle against economic and racial injustice

Our planet is a lonely speck in the great enveloping cosmic dark. In our obscurity, in all this vastness, there is no hint that help will come from elsewhere to save us from ourselves.

Carl Sagan, Pale Blue Dot on the preservation of Earth

Running Django Tests in Github Actions

I’ve bought into Github Actions as a CI/CD and deployment workflow for Django.

But since Github Actions is still young there are not as many guides on setting using it with Django, especially for the testing step.

If you are just getting started using Github Actions with Django, I suggest reviewing Michael Herman’s recent epic, “Continuously Deploying Django to DigitalOcean with Docker and Github Actions.” The repo for Michael’s tutorial is here.

What about Testing?

Test Driven Development or TDD or really just writing tests as a matter of habit while programming allows you to build bigger things faster.

If you’ve been building for a while, you know this, and maybe you’d enjoy Brian Okken’s Test & Code podcast.

I realize many folks are migrating over to Pytest, but I have not yet. So, this example focuses on the normal Django TestCase. It also presumes your use of postgres.

Why not use SQLite instead of Postgres for CI/CD Testing?

When I first looked at setting this up, I was tempted to use SQLite instead. After all, it wouldn’t require spinning up a container!

However, if you have read Speed Up Your Django Tests by Adam Johnson, you’d know that it is not a good idea to swap in the use of SQLite for Postgres when running your tests.

Thus, we shall not be tempted to use SQLite even though it might be easier! Instead we will seek shortcuts to make use of postgres in your Github Actions CI for Django as-fast-as-possible.

Example Django Test Job for Github Actions using Postgres

Here is what I came up with to perform a Django testing stage using a project setup similar to Michael’s:

jobs:
  run_tests:
    if: github.ref == 'refs/heads/develop'
    name: Run Django Tests
    runs-on: ubuntu-latest
    services:
      db:
        image: postgres:12.3-alpine
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: github_actions
        ports:
          - 5432:5432
        options: --mount type=tmpfs,destination=/var/lib/postgresql/data --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
    steps:
      - name: Checkout
        uses: actions/checkout@v1
        with:
          ref: 'refs/heads/develop'
      - name: Set up Python 3.8.5
        uses: actions/setup-python@v1
        with:
          python-version: 3.8.5
      - name: Install dependencies
        run: pip install -r app/requirements.txt
      - name: Run tests
        run: python app/manage.py test app/
        env:
          SYSTEM_ENV: GITHUB_WORKFLOW

Explanation:

Line 2: This job sits at the same indentation level as the build and deploy jobs used in Michaels’ example Github Actions workflow file.

Line 3: Higher up in your workflow, your action should be on: [push]. Line 3 tells Github Actions to only run the testing if the push was to a branch called develop. You can change this to suit your project, though note it will also need to be updated below.

Line 8: Note that the postgres service is from Alpine. Most services use the general postgres container, but that’s not necessary. Alpine is smallest docker container version of postgres I’m aware of, and you should choose it to help make this job as fast as possible.

Lines 9-12: These environment variables are used by the postgres image in setting up your database that will be used for test. These must match what your project’s settings.py file uses when it is run by Github Actions

Line 13: It is simplest to explicitly declare 5432 as the port for this job.

Line 15: These options ensure the database is ready before allowing the job to continue to run the tests. In addition, a temporary filesystem (tmpfs) is mounted. h/t @AdamChainz

Line 20: This specifies where the code that contains the tests you’re running will be. This line should match Line 3 above: you want to test the branch you’ve just pushed to.

Lines 21 and 25: Presuming your requirements.txt file contains the expected postgres dependencies, these are the only two steps you need to perform in order to get to running your django tests. Some guides suggest you need to run migrations or install postgres dependencies manually. I did not find this to be the case.

Line 28: Note that you must specify the project path twice in the test command. Normally, in pycharm for example, you don’t notice this is needed but try running ./manage.py test from the parent directory to /app without the second app/ and you’ll see zero tests are run.

Line 30: This line is crucial! It is passed means that the SYSTEM_ENV environment variable is set to GITHUB_WORKFLOW at the time your django project is run by Github Actions.

This is important because you will need to customize the DATABASE portion of your settings.py so that it will be set appropriately for this stage of Github Actions.

Configuring your Django Settings for Github Actions Django Testing Job

SYSTEM_ENV = os.environ.get('SYSTEM_ENV', None)
if SYSTEM_ENV == 'PRODUCTION' or SYSTEM_ENV == 'STAGING':
    DEBUG = False
    SECRET_KEY = os.environ.get('SECRET_KEY', None)
    ...
    CSRF_COOKIE_SECURE = True
    ...
    # Set Sentry and Error Debug
    import sentry_sdk
    ...
elif SYSTEM_ENV == 'GITHUB_WORKFLOW':
    DEBUG = True
    SECRET_KEY = 'TESTING_KEY'
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': 'github_actions',
            'USER': 'postgres',
            'PASSWORD': 'postgres',
            'HOST': '127.0.0.1',
            'PORT': '5432',
        }
    }
elif SYSTEM_ENV == 'DEVELOPMENT':
    DEBUG = True

The above shows the settings override section of a django project that allows you to detect for the SYSTEM_ENV set in an environment variable.

For the example in this blog entry, you only need to pay attention to Lines 11-23. However, I included the declaration of the SYSTEM_ENV (Line 1) and the conditional detection of production and staging environments to help show how SYSTEM_ENV can be used to effectively switch Django settings overrides for dynamic CI like Github Actions can allow.

Pay attention to lines 17-21. These need to match the settings used for the db service in the job’s postgres service declaration above. If you don’t get the match right, or something is funky with your settings overrides, you may get an error like:

django.db.utils.OperationalError: could not translate host name "db" to address: Temporary failure in name resolution

I give an extended explanation of how to resolve this in this Stack Overflow answer.

Notes on Debugging Github Actions

As I mention in the above SO answer, one of the problems with Github Actions is the delay between trying something and waiting for feedback from Github.

The disconnect a change and the result can disguise things you might otherwise catch in local development. For example, you may think you’re working on relevant code / pushing to the right place / setting the correct environment variables when you are not.

Part of the reason this might happen is because you started browsing the web or were otherwise distracted waiting to see if your fix worked. I am guilty of this.

Another pain is that in writing Github Actions that act on a push, you may find yourself pushing to a branch that you don’t want to bunch of clumsy changes in.

If your commit message quality adherence is strong, it can become a bit annoying to have to keep explaining changes and reverts of those changes as you test things out.

Improvements to this example

This is just one job out of what can be an awesome CI/CD workflow for Django with Github Actions. For example, you would probably not want to deploy a copy of your Django project that fails the testing job!

With a complete setup you could add a

needs: [build_develop, run_tests]

to your deploy_staging job, which would allow your build and test jobs to run in parallel! I won’t get into that here, but it really is magic when it is all working.

Specific to this guide, I have no doubt that this can be improved upon as I’m still learning the ways of Github Actions myself. Please feel free to suggest them in the comments, hit me on twitter or email me.

Additional Links on This Topic

Here are some additional links and resources I found while learning about this topic:

Monstera Deliciosa Timelapse with Wyzecam

One of my favorite houseplants is my is a Monstera I picked up from Portland Nursery last year. I normally give the plant a bunch of artificial light to help it out. Every now and then a big new leaf grows, and this time I captured the last five days of its unfolding in a time lapse video

The music for this video comes from Kora’s sunrise set at Maxa Xaman, Burning Man 2018.

To make this HD time lapse, I recorded the video on a Wyze cam, which I physically removed the internal hardware microphone from. This cheap cam has a handy auto night mode which shows how much leaf development is happening at night.

I chose not to use wyzecam’s own timelapse behavior because I wanted to build it from the raw HD video myself and get the best time compression possible.

The data was stored on a 32GB microSDHC UHS-I memory card. I read the card using this handy pink card reader.

Wyzecam stores many mp4 files per hour in individual folders per day. I used MP4tools to perform the joins which took the most time. Some of the mp4 files created by the wyzecam were corrupted. I didn’t bother to try to repair them, but this resulted in some of the jumpiness at times as they were not compiled in to the day-long clips.

Finally, I dropped each compiled day of leaf growth (~8GB each) into iMovie and added some tunes. I used the simple timeline editor to drop the length of each day. Not a precise operation but handled it quickly. The timeline looks like this

If I were to improve on this timelapse creation process, I would:

  • Consider more consistent lighting for daytime
  • Get a 64GB micro sd card to get 10 days of video
  • Write some kind of automation for the video segment assembly
  • Spend more time on the music / video production

Ongoing Project: EasyALPR – Parking Enforcement App

I’ve been working on series of parking enforcement products using license plate recognition technology called EasyALPR.

Several years ago I was working on a privacy and crypto-currency application, Gliph. Apparently, this was the first venture-funded crypto-startup because we raised our first round prior to Coinbase.

That company did not make it, so with funding from some of my previous investors, I started another one.

This time I flipped privacy on its head by building a live video streaming application. Any old smartphone becomes a webcam.

That product concept, Perch, was real art: see-all-the-things type stuff. The world in a fishbowl. Police action, puppies, perpetrators, you name it all live and archived in high definition. I go into more details in this EasyALPR company blog entry.

The tech was sweet, beat Facebook Live. Beat Twitch Clips by months. But the concept was too artsy, and too expensive to operate. I shuttered that product but kept the company.

When a user caught some perpetrators at a gas station in Oakland on Perch I stumbled again on the power of LPR. Ya, it only took me five years to come around worrying about it to working on it.

I did consider building some more scan-all-the-things type stuff as I explored five years ago on this blog. But I am not into that whole vibe really.

I am trying to focus with this product and so far it has been terrific for patrolling parking lots. So I’ve become interested in parking even though I don’t own a car.

It is really about automation and putting away manual processes. It has a license plate recognition app you can download free from the App Store, and a web application that does a lot of cool stuff too.

I explore that some in this entry on ALPR databases and more.

Not the most artistic idea in the world, but I built most of the product entirely myself, and I’m pretty happy with it so far. I have two product releases out so far, Parking Hero and Parking Defender. I have another, new one, in beta right now.

Hello Again

I haven’t posted an entry for over four years.

I was active for most of that time micro-blogging. I have a lot of content on Twitter but set it all to private and stopped posting not long after the 2016 election.

I’m still into music and technology.

Not so much cryptocurrency. I’ve been getting into house plants recently. I’ve enjoyed Burning Man for several years and growing new capabilities to apply out there.

Maybe I’ll post more often again. Maybe not. There’s going to be at least one to drain some SEO toward a project.

Want to Spread an Idea Fast? Describe it with Software.

I’ve been thinking about ideas recently. What it takes to move from neurons in one person’s head to changing the lives of many.

The essence of an idea can be documented in a software’s backend system. Hidden behind the buttons you click on and input boxes you type into is a backend describes complex “business rules” or logic that describe the idea.

The focus of my work very recently is expression of an idea in software at its various user interface (UI) endpoints. A goal of this work is to build interfaces that communicate the fundamental idea to people while solving specific existing problems.

Even unknowingly, users are influenced by core concepts that drive a software system’s behavior. When people engage with a system via a software UI, it can establish new social norms and behavior.

For example, if you have used Wikipedia, you are immediately learning that it is possible for anyone to share important information using writing and pictures and that this can be reviewed by and edited by peers for free.

The idea that people could collaborate in such a way was not widely accepted as a good one to understand until a backend system and desktop web interface was created to express the idea.

In Facebook’s S-1 filing, Mark Zuckerberg said Facebook was created “to accomplish a social mission–to make the world open and more connected.” To express such an idea in software at the time meant reliving past ideas like MySpace’s wall and bringing new taste to the expression of the idea. Interestingly, part of Facebook’s success was in limiting the idea’s early availability to students.

Compared to reading a white paper or listening to a lecture, average folks will probably understand the meaning of an idea more quickly by interacting with it via software. That is, if the software is fun to use.

Software can now spread to individuals extremely fast.  This is exciting because when a sufficiently advanced new idea is described for the first time in software, the idea may be spread nearly as fast.

This suggests that if you have a big new idea and want the idea to influence how people think and behave, perhaps you should consider how it would be described using software.

Mobile Automated Fluid Dispenser that Accepts Bitcoin as Payment Mechanism

Farmer, thermal dynamics researcher, and DIY pro, Andy Schroder created a “mobile, automated fluid dispenser that accepts Bitcoin as a payment mechanism.”

This is an amazing look at the future of P2P service delivery. I’m particularly impressed with depth of thought Andy gave to the capabilities of his demonstration. If you want to skip the explanation, the action really starts about 8 minutes into the video:

This is so futuristic and awesome that it could only be improved only by the presence of a node to a mesh network with an uplink to the Bitcoin network, rather than reliance on commercial telecom. Bravo, Andy.

Swartz vs Snowden OR Resist Restrictions that Seem Arbitrary or Capricious

MIT released its report on Aaron Swartz (.pdf) this morning. It is interesting to read about the amount of energy and concerted effort around Swartz, in contrast to Edward Snowden. Swartz’ prosecution and MIT’s “neutrality” to that effort seem sad when taken in the context of President Obama’s comment on June 27th where he said “No, I’m not going to be scrambling jets to get a 29-year-old hacker.”

In the report the authors pose questions to the MIT community, including: “How can MIT draw lessons for its hacker culture from this experience?” The answer includes:

MIT celebrates hacker culture. Our admissions tours and first-year orientation salute a culture of creative disobedience where students are encouraged to explore secret corners of the campus, commit good-spirited acts of vandalism within informal but broadly— although not fully—understood rules, and resist restrictions that seem arbitrary or capricious. We attract students who are driven not just to be creative, but also to explore in ways that test boundaries and challenge positions of power.

This raises the question of whether the MIT community is sufficiently aware of what the hacker culture is meant to be about, of the risks inherent in crossing lines as part of hacking, and the roles of faculty, staff and administration in responding to what might or might not be a hack.

Yet in the computer context, unlike as in the physical world, “unauthorized access”—ill defined as it may be—can be grounds for a major federal felony prosecution. For Swartz the end result was calamitous. The entire episode may create a chilling effect for those students contemplating exploits that may push the bounds of their and society’s knowledge, but will also take them to places where conventional rules say they are not supposed to be—“coloring outside the lines” so to speak, punishable by criminal records rather than mere forfeiture of crayons. [emphasis added]

I can’t help but think of the recently exposed removal of President Obama’s promise to “Protect Whistleblowers” from Change.gov. Which read:

Often the best source of information about waste, fraud, and abuse in government is an existing government employee committed to public integrity and willing to speak out. Such acts of courage and patriotism, which can sometimes save lives and often save taxpayer dollars, should be encouraged rather than stifled. We need to empower federal employees as watchdogs of wrongdoing and partners in performance.

Barack Obama will strengthen whistleblower laws to protect federal workers who expose waste, fraud, and abuse of authority in government. Obama will ensure that federal agencies expedite the process for reviewing whistleblower claims and whistleblowers have full access to courts and due process. [emphasis added]

Later, in the answer to the above question, the MIT report asks “Are we misleading students and community members by advertising one kind of community and enforcing rules more appropriate to a different kind of community?”

This seems to be the question not just for the MIT community, but for all Americans.

Steve Jobs on Loyalty

This isn’t exactly Steve Jobs on loyalty, but rather comments from Joshua Michael Stern, director of the new Steve Jobs film JOBS:

“… I think that he was loyal to his vision and he was loyal to the one thing he wanted to bring to the world … I think he was loyal to what he was trying to achieve. If you were loyal to that too then he loved you and if you didn’t then you didn’t understand him.”

I think this comes from feeling that Steve Jobs’ execution toward his vision for how things could and should be was the greatest single contribution he can make to society. If you are following a vision of similar grandeur, you can not see another way to behave that would make the world a better place.

When you see only one vision for how you can help the world, you must have people around you who support this. Otherwise, those people are blocking a better world. When the vision is big enough, there is no room for half-hearted allegiance or simple cooperation. It is ride or die.

Rockonomics

I’ve greatly enjoyed Alan Krueger’s “Land of Hope and Dreams: Rock and Roll, Economics and Rebuilding the Middle Class” speech given at the Rock and Roll Hall of Fame this past June.

In the speech, Krueger compares the economics of the music industry to what is happening in the U.S. economy as a whole. It puts the major shift in the average cost of concert tickets right next to changes in the US income distribution.

It features details from a clever study (.pdf) that measured the impact of luck in combination with merit on the success of artists in a system for downloading music. It also puts forth proposals from President Obama’s economic agenda for addressing the dangerous consequences of a winner-take-all economy.

I’m an avid concert-goer and watched ticket prices climb in the late 90s and 2000s. Later I ran a Dave Matthews Band fan website where we were so regularly bombarded by requests to run advertisements from scalpers we had to take a stand.

Not surprisingly, my favorite portion of the speech focuses on the concept of fairness:

If artists charge too much for their tickets, they risk losing their appeal. In this sense, the market for rock ‘n roll music is different from the market for commodities, or stocks and bonds.

Considerations of fair treatment exert pressure on how much musicians can charge, even superstars.

Along these lines, one of my favorite performers, Tom Petty, once said, “I don’t see how carving out the best seats and charging a lot more for them has anything to do with rock & roll.”

This is a major reason why there is a market for scalped tickets.
But many artists have been reluctant to raise prices to what the market will bear for fear of garnering a reputation of gouging their fans.

They also protest when tickets sell for a higher price on the secondary market, and often try to prevent the secondary market entirely. And it is considered scandalous when performers sell tickets on the secondary market themselves.

This behavior can only be explained in light of fairness considerations. Singers want to be viewed as treating their fans fairly, rather than charging them what supply and demand dictate.

Indeed, you can think of market demand as depending on the perception of fairness.

Read The Economist’s digest of the speech or Download the text of the speech (.pdf)