Today I did some buffering tests in 10 languages where I wrote the equivalent of this program and then ran the program with ./program | grep hello

while True:
    print("hello")
    sleep(1)

• In 4 of the languages (Python, Ruby, C, Perl) nothing at at all gets printed (because they buffer the output when stdout isn’t a TTY, the buffer size is probably 4k or something)
• in the others (Go, C++, JS, Rust, Java, Lua, bash) you see “hello” printed out once per second. PHP is in this category too.

I knew there was some variation here I didn’t have a list of which languages did which and now I do. (If you want to not buffer the output you can flush stdout)

Here’s the code I ran to test this

I have my local vim set up to integrate with my system clipboard (so that p pastes from my system clipboard etc), but I thought it was impossible to do the same thing over SSH. It turns out that actually it is possible with an ANSI escape sequence called OSC52! A few links:

• It’s a core Neovim feature: the neovim OSC 52 docs
• a vim-osc52 plugin
• a vim github issue about adding OSC 52 support

I haven’t tried this yet but it seems very cool

Dmitry Mazin also told me you can create this script on a remote host, call it pbcopy, and piping into it will copy to your clipboard! I tested it and it works.

#!/bin/bash
printf "\033]52;c;%s\007" "$(base64 | tr -d '\n')"

Someone pointed me to The TTY demystified on Mastodon and I appreciated the list of every signal that the kernel’s terminal driver sends processes (SIGWINCH, SIGPIPE, SIGCHLD, SIGINT, SIGHUP, SIGTSTP, SIGQUIT). I thought SIGWINCH and SIGHUP were sent by the terminal emulator but they aren’t.

I learned recently that the terminal has two different kinds of modes that can be set:

1. “termios” modes, which
   • are managed by the operating system
   • you can view with stty -a
   • are set with the ioctl system call
2. “ANSI” modes (not sure if this is the right name), which
   • are managed by the terminal emulator
   • you can’t normally view, except in GhostTTY’s inspector
   • are set by sending ANSI escape codes (like "\033[?25l" for “make cursor invisible”)

I tried cross compiling in Go for the first time and it’s really amazing how GOARCH=amd64 GOOS=linux go build just works immediately.

Sounds like the one caveat is that things aren’t so magical if if you use CGO, someone linked to https://github.com/messense/homebrew-macos-cross-toolchains which apparently makes that easier on Mac OS.

Just discovered litecli, it feels like a big improvement over the default SQLite REPL. (autocompletion! support for multiline queries! syntax highlighting!)

Apparently there’s also mycli and pgcli by the same author for MySQL and Postgres.

Yesterday a program made my terminal’s cursor disappear! I was really surprised because programs so rarely mess up my terminal these days.

Normally my prompt looks like this:

But after running echo -e "\033[?25l" (the escape code for “make cursor invisible”), the cursor is gone, like this.

I just dealt with it by opening a new terminal tab, but you can also run reset or echo -e "\033[?25h". I don’t really like using reset because it sleeps for 1 second and I find it kind of annoying, I wish it ran instantly.

I don’t have much reason to use git bisect usually, but today I noticed that a bug in an open source project had been fixed and I wanted to figure out when.

The main gotcha was that git bisect usually assumes that you’re trying to figure out when a bug was introduced (not fixed), so I had to reverse everything.

step 1: Write a script test.sh that fails if the bug is fixed (so I had to reverse the exit codes):

pip3 install .
if unbuffer sqlite-utils insert test.db test --csv test.csv | grep -o '100.......' | grep 25h
then
    # this means the bug is fixed, so fail (reverse the output)
    exit 1
else
    exit 0
fi

step 2: run git bisect

git bisect start
# we have to mark the current state as "bad" even though it's technically the
# old state that was "bad"
git bisect bad main
git bisect good 0d45ee11
git bisect run test.sh

It worked perfectly and after 10 seconds gave me the exact commit that had fixed the bug.

I’ve been doing a lot of SQLite migrations, and I’ve learned that this doesn’t work in SQLite.

CREATE TABLE newspapers (
    id INTEGER PRIMARY KEY,
    editor_id INTEGER,
    FOREIGN KEY (editor_id) REFERENCES users(id)
);
ALTER TABLE newspapers DROP COLUMN editor_id;

It fails with this error:

error in table newspapers after drop column:
unknown column "editor_id" in foreign key definition

Instead you have to create a new table, copy over the data, and rename it. The SQLite docs have a 12-step process for recreating tables like this, and there’s a great blog post by Simon Willison.

Learned via Jacob Kaplan-Moss today that now Python has a way to specify dependencies inline in scripts?! For example I can create this script:

# /// script
# requires-python = ">=3.9"
# dependencies = [
#   "requests<3",
# ]
# ///
import requests
requests.get("http://example.com")

and then run it with uv run blah.py. I haven’t totally understood what’s going on with uv yet but it seems interesting. The syntax is documented here, some notes on uv from @simonw here

I’ve never used the GitHub gh CLI at all really but I learned recently that you can use it to set secrets to use them in GitHub Actions. SO much easier than setting them from the web interface.

gh secret set FLY_API_TOKEN

Also gh auth token is a great way to get a token to authenticate to the GitHub API, like this:

export GITHUB_TOKEN=$(gh auth token)
curl -H "Authorization: Bearer $GITHUB_TOKEN" \
        https://api.github.com/repos/gohugoio/hugo/releases/tags/v0.136.0 | jq -r '.body'

I’ve been reading Let’s Go by Alex Edwards and one of my favourite tips from the book is to write a HTTP middleware to recover from panics, something like this:

func (app *application) recoverPanic(next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    defer func() {
      if err := recover(); err != nil {
        w.Header().Set("Connection", "close")
        fmt.Println(err)
        debug.PrintStack() // from "runtime/debug"
        app.serverError(w, fmt.Errorf("%s", err))
      }
    }()
    next.ServeHTTP(w, r)
  })
}

Previously my webserver would just crash if there was a panic (not even returning an error!) and I think this is better.

Apparently now you can run Jupyter notebooks in a browser without a server with WebAssembly using JupyterLite?? I learned about this because @paddymul made a pull request to my pandas-cookbook repo to add support for it.

So now you can try out the cookbook in your browser. It seems to work!

One of the missing pieces about databases for me has been what to use for backups. I recently decided to go with restic and it seems to be working okay so far.

My backup script looks somethnig like this. It pings healthchecks.io when it’s done.

sqlite3 /data/requests.sqlite "VACUUM INTO '/tmp/requests.sqlite'"
gzip /tmp/requests.sqlite

restic -r s3://s3.amazonaws.com/MY_BUCKET/ backup /tmp/requests.sqlite.gz 
restic -r s3://s3.amazonaws.com/MY_BUCKET/ forget -l 7 -H 12 -d 2 -w 2 -m 2 -y 2
restic -r s3://s3.amazonaws.com/MY_BUCKET/ prune
curl https://hc-ping.com/ID

I’ve been making a web application where I haven’t ironed out my database schema yet so I wanted a migration tool to tweak it as I change my mind.

Someone on Mastodon recommended dbmate recently and it’s been working well. Here’s how I use it:

1. Run dbmate new my_migration
2. Edit the migration file to
3. Run dbmate up to apply it

Then I can run dbmate up in production too. I’ve only been doing “up” migrations, no “down” migrations.

The #1 thing I miss from Linux while using a Mac is strace. Recently I found out about fs_usage that does part of what strace does and I was SO happy.

(I’ve also tried dtruss/dtrace etc and a couple of other things which I have failed to get to work, I’m not willing to turn off SIP)

For example here’s me looking at what config files fish opens:

(for some reason I can’t grep fs_usage’s output directly, not sure why)

Just found this inclusive design checklist by Heydon Pickering. I found it because I wanted to implement a skip link for this blog, and it linked to this very clear guide from WebAIM on implementing skip links.

I’ve been reading advice to use semantic HTML for years but never really found an explanation that clicked. This article by Heydon Pickering really helped me understand how to do it in practice, specifically this “owl selector” tip for how to make everything evenly spaced:

.stack > * + * {
  margin-top: 1rem;
}

I also found his website’s CSS a really interesting read, especially how most of the selectors are for base HTML tags and classes but there’s occasionally something like this:

.margin-top\:0 {
  margin-top: 0;
}

I put a lot of screenshots in my blog posts and it’s always a little annoying to take the screenshot, put it in the right folder, and insert the <img> tag.

So I finally wrote the world’s smallest vim plugin which lets me just run :Screenshot NAME and it’ll save the image from my clipboard to the right place and write the <img> tag. It uses pngpaste.

diffdiff.net is a really nice side by side online diff tool, by Joe Kaptur. I’ve actually known about it for a long time but I’m always forgetting what it’s called.

Here’s what it looks like diffing some tcpdump output – used this to see that the MSS in two SYN packets didn’t match.

I wanted to make a cryptographic hash for a fun side project, didn’t know how to do it, then Kamal pointed me at Cryptographic Right Answers from Latacora. Specifically I learned that I should use HMAC for signatures.

Seems like a really great resource and there’s also a post-quantum version

Apparently you can download a public Google Sheet as a CSV, just tried it and it works great.

Sometimes I run some_program | grep blah, but libc buffers the output because the program isn’t writing to a terminal (in libc, writing to a terminal is line-buffered but writing to a pipe/file isn’t). So the grep doesn’t work. It’s annoying and I’ve never known what to do about it.

I finally found a solution to this thanks to folks on Mastodon: unbuffer from the expect package. unbuffer program will open a new TTY, set the program’s stdout to that tty, and then write to the pipe itself and flush the output.

You can also use unbuffer to trick the program into thinking it’s writing to a TTY for some other reason.