It’s sad to see so many people going along with or cheering on Trump’s inhumane and cruel policies. I like to imagine that someday they’ll realize what he is and feel shame for their support, but who am I kidding? Even those who were on the side of racism during the civil rights movement never felt a bit of remorse.

Also going through my head this morning:

I hope you’re proud how you would grovel in submission
To feed your own ambition
So though I can’t imagine how
I hope you’re happy now

I’m trying a couple of new things on this site.

One is to be a little less guarded about myself. I’m sometimes afraid to overshare and so end up overly reserved. I’ve started with a more comprehensive and personal about me page.

I’ve been posting some photos using the Glass app over the last couple of years, and I’ve now copied them here as well, along with an RSS feed.

Since I last analyzed my electricity usage two years ago, several things have changed:

1. I had a 240V charger installed, which speeds up charge time and draws more power.
2. My electric company changed the XML data slightly, with an entry every 15 minutes instead of 60, and with entries for power returned to the grid. For me, the latter are always zero because my house does not generate electricity.
3. We replaced our family-hauling gas minivan with an electric vehicle, so now there are two cars to charge. This happened at the end of the year, so you don’t really see it in the data yet.

To extract the data in Python, using the built-in xml.etree module to convert to a Pandas series, I reused most of the code from last time:

from xml.etree import ElementTree
import datetime
import pandas as pd

ns = {'atom': 'http://www.w3.org/2005/Atom', 'espi': 'http://naesb.org/espi'}
root = ElementTree.parse('/path/to/file.xml').getroot()
prefix = "./atom:entry/atom:content/espi:IntervalBlock[@rel='Delivered']/espi:IntervalReading"
times = [datetime.datetime.fromtimestamp(int(x.text))
         for x in root.findall(prefix + "/espi:timePeriod/espi:start", ns)]
values = [float(x.text)
          for x in root.findall(prefix + "/espi:value", ns)]
ts = pd.Series(values, index=times).sort_index()

The main difference is the addition of [@rel='Delivered'] to filter to only power delivered to me and not the other way around. I also added the sort_index command, because for some reason the dates are not entirely in order in the XML.

At this point, I wanted to determine when I was charging a car. Charge loads are probably pretty easy to isolate, because they last for a long time and are relatively constant. If I were trying to be robust, I would probably figure out what expected power draw for a given time of year and time of day, and then find out periods where the draw is significantly higher than that. Using something simple like the median of the surrounding 14 days of a given time would probably work, since I charge less than half of the days.

But in my case, the 7.5 kW of power that our electric Mini draws is more than our entire house uses over any 30-minute period. There are five 15-minute periods that reach that level, but these are relatively easy to filter out.

I wrote this code to compute the charge state. I wanted to separate it into cycles of “off”, “start”, “on”, and “stop”. My thinking was that these “start” and “stop” periods are probably times where I was charging the car for some but not all of the 15-minute period. I used a threshold of 1800 Wh, which is 7.2 kW over a 15-minute period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

THRESHOLD = 1800
charge_state = pd.Series('off', index=ts.index)
charging = False
for i in range(len(ts)):
    if charging:
        if ts.iloc[i] >= THRESHOLD:
            charge_state.iloc[i] = 'on'
        else:
            charge_state.iloc[i] = 'stop'
            charging = False
    # Look at the two entries after this one to see if
    # they are both above the threshold.
    elif all(ts.iloc[i+1:i+3] >= THRESHOLD) and i+1 != len(ts):
        charge_state.iloc[i] = 'start'
        charging = True

Line 2 creates a new series with the same index as our time series. We then look at the entries one by one and determine when to transition to “start” (Line 13, if we are not already charging and we see two upcoming entries above the threshold), when to stay “on” (Line 6, as long as we stay above the threshold), when to transition to “stop” (Line 8, as soon as we first go below the threshold). Note that Pandas uses iloc to look up an entry by integer offset, rather than by time.

With this charge_state series, it is easy to play around with the data. For example, to count how many charge sessions:

sum(charge_state == 'start')

To look at the entries where usage is high but you aren’t charging. This means “filter ts to points where it is above the threshold but charge_state is off.”

ts[(ts > THRESHOLD) & (charge_state == 'off')]

Finally, a good visualization is always helpful to understand the data. I don’t usually use much ChatGPT while programming, because at work I am usually dealing with complicated code that I don’t want to mess up. But it is impossible to remember how to do anything in Matplotlib, and I confess that I asked ChatGPT for a lot of help, and it did a pretty good job most of the time.

Here my goal is to draw dots at start and stop time for each charge, and connect them with a line. I really just have three arrays here:

1. start_time is the full date and time when I started charging. This is used as the x axis.
2. start_hour is the time of day when I started charging.
3. charge_hours is the number of hours that I charged.

Note that since charging often happens overnight, I’m using the end time as start_hour + charge_hours, which might be greater than 24, but I think that makes a better visualization than wrapping around to the bottom.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import pyplot as plt

start_time = ts[charge_state == 'start'].index
start_hour = start_time.hour + start_time.minute / 60
charge_hours = (
    ts[charge_state == 'stop'].index -
    ts[charge_state == 'start'].index).total_seconds() / 3600

fig, ax = plt.subplots(figsize=(15, 8))
# The bars
bars = ax.bar(start_time, charge_hours, bottom=start_hour, color='b')
# Draw the number on top of the bar
ax.bar_label(bars, labels=[f'{h:.0f}' for h in charge_hours], padding=4)
# Bottom dot
ax.scatter(start_time, start_hour, marker='.', color='b')
# Top dot
ax.scatter(start_time, start_hour+charge_hours, marker='.', color='b')
# Add a grid and make it denser than it wants to be.
plt.grid(True)
plt.yticks(np.arange(0, 29, 2));

And here is the final result. The Mini only has a 32 kWh battery, so can always fill in four hours or so. The longer lines from December are for the new car, which has triple the battery size, but also can max out the 50-amp circuit that my charger is on by pulling 9.5 kW. (If you do the math, that is only 40 amps, because code require that a continuous load uses only 80% of the rated amperage.)

The Mini used to be scheduled to start at 11 p.m., because it draws 30 amps on our 100 amp service, and I was afraid that if I charged it while everyone was still awake it might trip the breaker. In November, I decided to stop babying our house, and scheduled the charge to start at 9:15 instead. Cheap electricity starts at 9:00.

Also, a quirk of the Mini app is that if you plug it in on a Saturday night (the way my schedule is set), it won’t start charging until Sunday morning at 3:00. That is a long story for another time.

The back side of a clock from inside Musée d’Orsay. In the distance, the Louvre.

Welcoming courtyard

Ancient Roman Arena, Arles, France

The Camargues, France

Cloître Saint-Trophime, Arles, France

View of the river from Pont du Gard, France

Pont Neuf and Square du Vert-Galant, Paris

Boats along the Seine

New growth

Scooter in the rain

I had the “pleasure” of visiting the DMV this week to apply for a driving permit for my oldest kid. By a miraculous sequence of events, we got the permit in a single visit, but it was a close call.

Because the permit will eventually turn into a driver’s license and therefore a REAL ID, the application required two different documents to provide proof of address. There is a long list of valid documents, such as utility bills and property tax bills, and furthermore the DMV recognizes that not everyone living at an address receives such bills:

What if I do not have one of the above residency documents?

You can use a relative’s (parent, child, spouse/domestic partner) residency document if you live at the same address and provide a document (such as a birth or marriage certifcate [sic]) that shows that relationship.

It all seems reasonable enough, but the rules are implemented like a poorly-written computer program.

The question is, can I use my driver’s license (together with a birth certificate) as proof of my teen’s residency? In theory, this should count as definitive proof of address, since they required me to show two address documents in order to receive the license in the first place. At the very least, it should count as one of the two factors, at least as valid as a SoCalGas bill that anyone with a basic PDF editor could easily doctor.

In practice, as you have probably guessed, it counts as nothing. Why? Because the main list of documents is written assuming that they are in the name of the applicant, and this “relative’s residency document” special case is tacked on at the end. And of course, it would be silly to say that you could use your current REAL ID as proof of address to get a REAL ID, so thus you cannot use a relative’s REAL ID as proof of address to get your REAL ID.

Being a paranoid person, I brought two documents in addition to my driver’s license, but even that was almost not enough. See, my address can be written as either 221B Baker St or 221 Baker St #B. The two bills that I brought didn’t match, which (1) was apparently a problem and (2) my driver’s license wasn’t going to get me out of it. The only thing that saved me (this is the miraculous part) was that one of the two bills had the address written both ways.

(For completeness, two other miracles. One, that my kid passed the ridiculous written exam on the first try. A test that did have a question about NEVs without explaining the acronym, and is known for questions like “In which of these locations is it illegal to park? (a) blocking an unmarked crosswalk (b) in a bicycle lane or (c) within three feet of a driveway.” The answer is (a). Nobody knows why. The second miracle is that my teen even got to take the test in the first place, because the DMV shut down the testing center at 4:30 on the dot, sending away everyone who was in line at the time. Credit for this miracle goes to the employee who processed our application, because she shut down her station and went over to the photo station to clear out the queue, getting us through and into the testing center with less than a minute to spare. At the time, we had no idea that we were up against a clock, but I’m pretty sure that she knew and intervened.)

Anyway, now it is time for 50 hours of supervised (by me) driving practice. Wish us luck!

I have a 5-year-old Brother Scanner (the ADS-1700W) that serves me pretty well. It is small enough to keep on my desk, fast, and has a Wi-Fi connection. But getting scans from the scanner to the computer is sometimes bothersome. My primary way of using it is to scan to a network folder on my home server, which is great for archiving things, but not so great for family members or when I need something right away.

My preferred way to get a quick scan is by email. You immediately have it on whatever device you are using, and you have it saved for later if you need it. The Brother Scanner has a scan-to-email function, but it is buggy. Specifically, it sends slightly malformed emails that Fastmail accepts but Gmail returns to sender.

But since network scanning is rock solid, last year I wrote a program to watch a set of folders and send by email whatever files it finds. I was on a bit of a Go kick at the time, and I think Go works pretty well for the task.

Here is the relatively short program that I wrote. It is meant to be running as a daemon, and as long as it is running, it will email you all the files that it finds. Since the interesting parts are at the end and I don’t think anyone will read that far, I’ll show you the program in reverse order, starting with the main function.

 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

func main() {
    fmt.Println("Starting up")
    changes := make(chan int)
    go waitForChange(changes)

    ticks := time.Tick(5 * time.Minute)
    for {
        err := filepath.Walk("/home/scanner/incoming", sendFile)
        if err != nil {
            fmt.Println(err)
        }
        select {
        case <-changes:
            fmt.Println("File changed, looking for emails to send")
        case <-ticks:
            fmt.Println("5m has passed, looking for emails to send")
        }
    }
}

We start up, print a nice message and make a channel, which is a Go structure that allows us to pass data, in this case integers, between different threads of the program. We call it changes because it will notify us every time there has been a filesystem change.

The go keyword on line 99 starts waitForChange on a separate thread, which is a good thing for us, because you will later see that it runs an infinite loop. We pass it the channel so that it can notify us when it sees a change.

On line 101 we get a Ticker channel, which will receive a signal every five minutes. Since I don’t completely trust that I will be notified every time a file changes, every once in a while we want to look through the directories to see if we find anything.

Starting at line 102, we have an infinite loop here in the main thread. This starts by mailing out any files that are waiting. Then we get to the select statement, which pauses and listens to both the changes channel, and the ticks channel. The somewhat strange arrow syntax means that we are attempting to read values from the channel. If we wanted, we could assign the values we read to a variable and do something with them, but we don’t care what is on the channels. As soon as another thread writes to one of these channels (whichever channel comes first), we write the corresponding log statement and then continue back to the top of the for loop, which mails out any files we find, and then goes back to waiting for action on the channels.

By having the two channels, we have programmed the logic to walk the filesystem every time we see a change and every 5 minutes, but, crucially, never more than once at a time. In reality, the change watcher is very reliable, and the email generally arrives seconds after the paper comes out of the scanner.

83
84
85
86
87
88
89
90
91
92
93
94

func waitForChange(c chan<- int) {
    defer close(c)
    for {
        cmd := exec.Command("/usr/bin/inotifywait", "-r", "-e", "CLOSE_WRITE", "/home/scanner/incoming")
        err := cmd.Run()
        if err != nil {
            fmt.Printf("%v\n", err)
            time.Sleep(5 * time.Minute)
        }
        c <- 0
    }
}

Here is the waitForChange function, which just calls inotifywait. This in turn runs until someone writes a file and then exits. At this point, our function writes 0 into the channel, which kicks the main thread into action. Meanwhile this thread calls inotifywait again, to begin waiting for the next change.

56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

func sendFile(path string, info os.FileInfo, err error) error {
    if err != nil {
        return err
    }
    if info.IsDir() {
        return nil
    }
    to, err := getToAddress(path)
    if err != nil {
        return err
    }
    isOpen, err := fileIsOpen(path)
    if err != nil {
        return err
    }
    if isOpen {
        return fmt.Errorf("Skipping %v because it is opened by another process\n", path)
    }
    if err := sendEmail(to, path); err != nil {
        return err
    }
    if err := os.Remove(path); err != nil {
        return err
    }
    return nil
}

This sendFile function is called on every file in the directory tree. This is where Go gets annoyingly verbose. So much error handling! But fairly straightforward. As we walk the tree, we skip directories, send out emails if we have a file, and then delete the file after we send it.

44
45
46
47
48
49
50
51
52
53
54

func fileIsOpen(path string) (bool, error) {
    cmd := exec.Command("/usr/bin/lsof", "-t", path)
    err := cmd.Run()
    if err == nil {
        return true, nil
    }
    if exitError, ok := err.(*exec.ExitError); ok && exitError.ExitCode() == 1 {
        return false, nil
    }
    return true, err
}

This fileIsOpen function wasn’t there at first, but my early tries sent out files that were still being uploaded. Live and learn.

26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

func sendEmail(to string, doc string) error {
    msg := gomail.NewMessage()
    msg.SetHeader("From", "scanner@xxxx")
    msg.SetHeader("To", to)
    msg.SetHeader("Subject", "Here is your scanned document")
    msg.SetBody("text/plain", "")
    msg.Attach(doc)

    n := gomail.NewDialer("smtp.example.com", 465, "user", "pass")

    if err := n.DialAndSend(msg); err != nil {
        return err
    }

    fmt.Printf("Sent %v to %v\n", doc, to)
    return nil
}

It is relatively simple to send an email using this third-party gomail package. And it isn’t malformed like the scanner’s attempts to send email!

12
13
14
15
16
17
18
19
20
21
22
23
24

func getToAddress(path string) (string, error) {
    var addresses = map[string]string{
        "amy":        "xxx@yyyy",
        "nathan":     "xxx@wwww",
    }

    _, lastDirName := filepath.Split(filepath.Dir(path))
    to, ok := addresses[lastDirName]
    if !ok {
        return "", fmt.Errorf("No email address available for %v", lastDirName)
    }
    return to, nil
}

This is a relatively simple function that decides who to email to based on the folder that the file is in. This is the abbreviated version; my older kids also have emails configured here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

package main

import (
    "fmt"
    gomail "gopkg.in/gomail.v2"
    "os"
    "os/exec"
    "path/filepath"
    "time"
)

Finally, the most boring part of all, the import section.