Lunduke
News • Science & Tech
The Unlikely Story of UTF-8: The Text Encoding of the Web
Plan 9, Placemats, New Jersey Diners, and last minute ideas
June 22, 2023
post photo preview

If you are reading this on a computer -- of any kind -- odds are good that the words on the screen are all encoded using something called "UTF-8".

UTF-8 (or "Unicode Transformation Format - 8 bit") is, put simply, a format for encoding and storing text -- one which allows for far more text characters than the older "ASCII" encoding (which could only show a total of 95 printable characters).

And UTF-8 is, quite simply, everywhere.

Nearly every major computer operating system heavily uses UTF-8 for handling text... likewise it is the standard for websites, with close to 100% of all webpages explicitly using UTF-8 for the text on the page.

test
The source for Wikipedia.  Like most of the web, using UTF-8.

An argument could be made that UTF-8 is one of the most successful and widely adopted standards in all of computer history.

But this almost wasn't the case.

In fact, UTF-8 was created -- at the very last possible moment -- and it was first implemented in a computer system that most people don't even know existed.

X/Open's search for better text encoding

In the early 1990s, text encoding was... an issue.

While solutions for extended character sets (beyond simple ASCII characters) existed, they were less than ideal.  To put it mildly.  The most popular solution, known as UTF-1 (aka "ISO 10646"), suffered from serious performance issues... and often caused significant problems with software which used plain "ASCII" text (including UNIX file system paths).

Having a character encoding on UNIX systems that could cause problems with UNIX file systems?  Not good.

Obviously a new type of text encoding was needed.

So, in 1992, X/Open (originally known as the "Open Group for UNIX Systems", a consortium of UNIX vendors, including: Sun, HP, AT&T, IBM, and several others) set about the task of selecting a proper text encoding standard to be used across all of the UNIX world.

The proposal that gained the most traction was known as FSS/UTF (aka "File System Safe Universal Character Set Transformation Format").  Roills off the toungue, right?

This proposal was both faster than the old text encoding standard... and, as the name suggests, it was "File System Safe".  Which was a big win.

Enter: The Plan 9 Nerds

Which brings us to September 2nd, 1992.  Sometime in the early evening.

The X/Open group was meeting, in Austin, Texas, to formally decide on the file encoding standard.

Looking to get some feedback on the proposal, some members of X/Open made a call to two legendary programmers -- Ken Thompson and Rob Pike -- who were working on the Plan 9 Operating System project at Bell Labs in New Jersey.


A little background...

Ken Thompson was one of the creators of MULTICS, UNIX, the B programming language (the predecessor to C), among many other accomplishments.

Rob Pike, also a UNIX programmer, was the co-creator of Blit, writer of multiple UNIX and programming books, and the creator of the first UNIX windowing system.

To call these two "absolute legends" in the world of computing would be, perhaps, a bit of an understatement.  The two were currently working together on a research operating system, at Bell Labs, called Plan 9.  An attempt to fix some of the perceived shortcomings of UNIX... by the creators of UNIX, itself.


What happened next... after Ken Thompson and Rob Pike received that phone call?  Luckily, we have a detailed accounting... written by Rob Pike, himself.

"We had used the original UTF from ISO 10646 to make Plan 9 support 16-bit characters, but we hated it.  We were close to shipping the system when, late one afternoon, I received a call from some folks, I think at IBM - I remember them being in Austin - who were in an X/Open committee meeting.  They wanted Ken and me to vet their FSS/UTF design."

Asking two legendary engineers for their input?  You can probably guess what happened next...

"Ken and I suddenly realized there was an opportunity to use our experience to design a really good standard and get the X/Open guys to push it out.  We suggested this and the deal was, if we could do it fast, OK."

That's right.  Ken and Rob had some ideas.  And the X/Open folks agreedd to listen to those ideas... if they could get them something fast.

And, by fast, they really meant "immediately... like... right now."  Because the X/Open team were, quite literally, all gathered in Austin to decide on this... right then.

"Yeah.  I could eat."

Ken and Rob did what any good programmers would do when placed on an almost impossibly tight deadline -- and needed to come up with an amazing idea that could change the course of computing for decades to come... they went out to grab some grub.

"So we went to dinner, Ken figured out the bit-packing, and when we came back to the lab after dinner we called the X/Open guys and explained our scheme.  We mailed them an outline of our spec, and they replied saying that it was better than theirs (I don't believe I ever actually saw their proposal; I know I don't remember it) and how fast could we implement it?  I think this was a Wednesday night and we promised a complete running system by Monday, which I think was when their big vote was."

Remember.  This was 1992.

Which means, while laptops and such certainly existed, most people (even legendary programmers) did not have any sort of mobile, portable computers.  Certainly not the kind you could take out to a restaurant.

So what, pray tell, did they write their new text encoding design on?

A placemat from a New Jersey diner.

This is not the placemat that UTF-8 was designed on.

Seriously.

"UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner."

The boys, Ken and Rob, now had just a few days to get all of this done -- before the big vote on the new text encoding standard.  And they sure as heck didn't waste any time.

They got back from dinner, placemat in hand, and got to work.

"So that night Ken wrote packing and unpacking code and I started tearing into the C and graphics libraries.  The next day all the code was done and we started converting the text files on the system itself.  By Friday some time Plan 9 was running, and only running, what would be called UTF-8.  We called X/Open and the rest, as they say, is slightly rewritten history."

They converted an entire operating system over to a brand new -- just designed on a placemat -- text encoding format... in less than two days.

Here's the rough time-line:

  • Wednesday (Sep 2) evening: Dinner at a New Jersey Diner.  Ken Sketches out the idea on a placemat.
  • Wednesday night: Coding begins.
  • Thursday: Coding complete.
  • Friday: Entire Plan 9 operating system is now using "UTF-8".
  • Monday (Sep 7): X/Open group votes to use the Ken/Rob encoding design.

On Tuesday, September 8th, 1992 (at 3:22am), mere hours after the official vote to accept their text encoding design, Ken Thompson sends out the following email regarding Plan 9 now using UTF-8:

"The code has been tested to some degree and should be pretty good shape.  We have converted Plan 9 to use this encoding and are about to issue a distribution to an initial set of university users."

That's right.

Ken and Rob got a call asking for feeback on a Wednesday.  By the next Tuesday (at 3am) they were ready to ship a version of their Plan 9 OS with all the changes, and their designs had been voted on by the largest UNIX companies in the world.

Like I said.

A recent picture of the two legends, themselves.

These guys are legends.

What about that placemat?

Considering the vast impact of UTF-8 on the world of computing... whatever happened to that original "design document" (aka "the placemat")?  It would certainly be of historic significance.

"I very clearly remember Ken writing on the placemat and wished we had kept it!"

Let this be a lesson to all of the programmers out there:

Keep all of you doodles, notes, and sketches you make for your projects... you never know when one of those projects will become critical to the entire world... making your quick sketch worthy of being in a museum.

Especially if it's on a placemat.  From a diner.  In New Jersey.


Copyright © 2023 by Bryan Lunduke.  All rights reserved.  The contents of this article are licensed under the terms of The Lunduke Content Usage License.

community logo
Join the Lunduke Community
To read more articles like this, sign up and join my community today
11
What else you may like…
Videos
Podcasts
Posts
Articles
Rust Changes from "Master" to "Main"

Two weeks ago, The Lunduke Journal pointed out how many Leftist Open Source organizations (Rust, NixOS, Linux Kernel, etc.) still used the "Master" branch naming (against their "inclusive naming" rules). Yesterday, Rust changed to "Main".

More from The Lunduke Journal:
https://lunduke.com/

00:05:11
October 17, 2025
GNOME Foundation Discusses Refusing Funds from Framework Computer

Framework Computer is offering monthly financial support to the GNOME Foundation. Framework also supports Omarchy Linux & Hyprland, which GNOME activists call "Fascist", so GNOME may refuse those funds.

More from The Lunduke Journal:
https://lunduke.com/

00:09:00
October 17, 2025
The Confidential Communication from UK's Ofcom to 4Chan

The lawyer representing 4chan has provided documents sent by the UK's Office of Communications (marked CONFIDENTIAL). And they are absolutely wild.

More from The Lunduke Journal:
https://lunduke.com/

00:19:31
November 22, 2023
The futility of Ad-Blockers

Ads are filling the entirety of the Web -- websites, podcasts, YouTube videos, etc. -- at an increasing rate. Prices for those ad placements are plummeting. Consumers are desperate to use ad-blockers to make the web palatable. Google (and others) are desperate to break and block ad-blockers. All of which results in... more ads and lower pay for creators.

It's a fascinatingly annoying cycle. And there's only one viable way out of it.

Looking for the Podcast RSS feed or other links? Check here:
https://lunduke.locals.com/post/4619051/lunduke-journal-link-central-tm

Give the gift of The Lunduke Journal:
https://lunduke.locals.com/post/4898317/give-the-gift-of-the-lunduke-journal

The futility of Ad-Blockers
November 21, 2023
openSUSE says "No Lunduke allowed!"

Those in power with openSUSE make it clear they will not allow me anywhere near anything related to the openSUSE project. Ever. For any reason.

Well, that settles that, then! Guess I won't be contributing to openSUSE! 🤣

Looking for the Podcast RSS feed or other links?
https://lunduke.locals.com/post/4619051/lunduke-journal-link-central-tm

Give the gift of The Lunduke Journal:
https://lunduke.locals.com/post/4898317/give-the-gift-of-the-lunduke-journal

openSUSE says "No Lunduke allowed!"
September 13, 2023
"Andreas Kling creator of Serenity OS & Ladybird Web Browser" - Lunduke’s Big Tech Show - September 13th, 2023 - Ep 044

This episode is free for all to enjoy and share.

Be sure to subscribe here at Lunduke.Locals.com to get all shows & articles (including interviews with other amazing nerds).

"Andreas Kling creator of Serenity OS & Ladybird Web Browser" - Lunduke’s Big Tech Show - September 13th, 2023 - Ep 044

I need to vent and i figure you all can have a good laugh at my expense...
I've got this HP Vectra 486/33VL I've been working on. On my bench, I got everything working but the CDROM. So I decided to put everything back together. It looked good but the rear panel was loose which I knew I need to deal with.
That's when I started having issues...
The computer was no longer talking to the Sound Blaster 16. I tried everything. I spent hours troubleshooting. Was pulling parts in and out and the whole time, the rear of the case was getting looser and looser. I knew I needed to repair the clips on it, but what's the point if I can't get the stupid computer working?
I was inspecting traces and going over every inch with a microscope. I was checking voltage rails and pinouts.
After over 3 hours and attempting a fresh install of DOS, which was a pain itself since I couldn't get the CDROM or Floppy drive working, I got seriously frustrated with the back panel and just removed it.
That's when I realized ...

4 hours ago

Are you sure this is AI??

October 15, 2025
post photo preview
The Unpublished Anti-Lunduke Hit-Piece
A Tech Journalist interviewed me for a hit-piece article. But the questions made them look bad, and they shelved the story. So I'm publishing their hit-piece for them.

Back in September, shortly after the assassination of Charlie Kirk, I was contacted by a Tech Journalist writing for FossForce.com (a smaller, Open Source focused publication) who was working on an article around Open Source, Antifa, and the Lunduke Journal’s coverage of those topics.

This particular outlet had, several months prior, run an “anti-Lunduke” hit piece without first reaching out for comment — which resulted in their most popular article (at least on social media) in quite some time.

With that in mind, it seemed reasonable that they’d want to repeat that success with another “anti-Lunduke” story.

This time they were doing the responsible thing. They reached out to the subject of the hit-piece article with questions. I like encouraging Tech Journalists when they do actual journalism, so I answered each and every query with easy-to-quote responses.

But, it would appear that the answers they received were not conducive to creating the hit-piece they were hoping for — my guess is they realized their questions made them look like the villain in the story. The villain they, clearly, hoped to portray me as.

They opted to not publish the piece.

So I’m publishing their hit-piece for them.

Below is every question — and every answer (with no edits) — which I was asked, on September 19th, by a Tech Journalist by the name of Christine Hall, writing for FossForce.

Fair warning: This is very, very politically charged.

Enjoy.


September 19th

Hall:

The last time I mentioned you in an article, you castigated me for not reaching out to you beforehand. Well, I’m reaching out now. We’ll see what comes of this.

You do recognize that the vast majority of organizations using the term antifa as a descriptor are not in the least bit terrorist and pose no threat to society -- and indeed, the only threats they might pose to fascist groups are not physical or life-harming?

Lunduke:

Hello Christine! Nice to hear from you!

Many, if not most, of those proclaiming support for Antifa (within Open Source) have also made statements encouraging or supporting violence and discrimination.

Regardless of that fact -- which I have documented extensively in Lunduke Journal coverage -- when violent acts are committed (such as murder, riots, and lynchings) in the name of “Antifa”, to turn around and immediately declare yourself to be “Antifa” is a clear declaration of support of that violence.

Hall:

And why did you feel it necessary to call out Danielle Foré’s [the founder of the elementary OS Linux Distribution] trans status in such an ugly manner?

Lunduke:

There is a noteworthy overlap between “Trans activism” and support for political violence -- including in the recent murder of Charlie Kirk (the murderer’s boyfriend was “Trans”).

In the case of Daniel Fore, he, a leader of an Open Source project, regularly calls for discrimination (and violence) against people he disagrees with -- often in conjunction with his self-declaration as “Trans”.

Thus, his declaration of being “Trans” becomes a part of the overall story.

It is worth noting here that The Lunduke Journal has never -- and would never -- call for discrimination or violence against someone because of how they identify or who they may (or may not) vote for.

This is in stark contrast those, such as Mr. Fore, who consider themselves “Trans” or “Antifa” -- who actively advocate for both discrimination and violence.

Hall:

Mentioning a person’s trans status in ways that are pertinent to your argument necessates rudeness such as calling her a “dude who likes to wear dresses”?

Lunduke:

Dan Fore is, in fact, a dude who likes to wear dresses.

The only reason to view that as a negative is if you view dudes wearing dresses as a negative.

Hall:

I’ll quote you on that, which I’m pretty sure won’t bother you in the least.

Lunduke:

Absolutely! Quote anything I say here. In fact, I suggest quoting absolutely everything I’ve written to you here, today.

Hall:

You also understand, don’t you, that voicing disagreement with an assessment made by POTUS is not only legal but a healthy part of the national dialog.

Lunduke:

Absolutely! Did I say somewhere that it was illegal to disagree with a politician? It seems unlikely that I have ever said that.

Hall:

Also, how would you reply to this:

There have been very few murders linked to individuals associated with Antifa, some incidents of rioting attributed to Antifa supporters, and no credible evidence of lynchings conducted in the name of Antifa. Compared to far-right groups, violence attributed to Antifa is much less frequent and lethal, with only one suspected kill—Aaron Danielson in Portland, by an anti-fascist activist—officially confirmed in recent U.S. history.

Lunduke:

Murder is bad. I am opposed to all murder.

In the context of these discussions, bearing in mind the Kirk murder is important (as many statements were made in response to it). The murderer of Kirk appears to have been pro-Trans and pro-Antifa (based on all available information).

Hall:

Is there any evidence that the suspect was part of an antifa group? I haven’t seen any.

Lunduke:

I have seen some reporting to this effect (including statements from family and messages he wrote).

But, far more important to this story, is the response to the murder among Antifa supporters (including those within Open Source). A large portion of Antifa supporters have celebrated the murder as justified because it killed someone they considered to be a “fascist”.

Hall:

Also, no group should be held responsible for what some deranged person who identifies with the group has done.

Lunduke:

I agree that a broader group should not be held responsible for the actions of a small number of individuals.

However, and this is critically important, it is entirely appropriate to hold people responsible for their own statements and actions.

With that in mind: The overall messaging of Antifa (and Antifa supporters) tends heavily towards violence. Punching, killing, molotov cocktails, etc. are all common messaging used by Antifa (including by those I quote within the Open Source world -- many of whom have advocated violence against myself).

Advocating for violence, then celebrating when violence is committed, are not good things.

Yet we see a great deal of that among Open Source supporters of Antifa.

Read full Article
October 13, 2025
Sale ends in a few hours, Lifetime Subs set up.

Holy moly, you guys are amazing.

A few days ago I published a “50% off” sale for Lunduke Journal subscriptions… and all of you showed up. In a big way.

To everyone who grabbed a Lifetime Subscription over the last few days: All of you are set to full Lifetime access. You should have a confirmation email in your inbox. If not, email me and I’ll make sure you’re setup properly.

That “50% off” sale ends tonight at midnight. So you have a few hours to snag a discounted subscription, if you haven’t already.

A huge thank you to everyone who supports this work. Couldn’t do it without you.

-Lunduke

Read full Article
October 12, 2025
50% Off Lunduke Journal Extended Through Monday (Oct 13th)

Just a quick heads up:

The “50% off every kind of Subscription to The Lunduke Journal” sale has been extended through Monday (October 13th).

So. You know. Grab one at 50% off between now and end of the day on Monday.

To all of you amazing nerds who have picked up a Lifetime Subscription already this weekend: You are awesome. You’ll be receiving a confirmation email, with all of the Lifetime Subscription details, by tomorrow (if you haven’t already).

Oh, and remember how we hit 11 Million views last month? Yeah. We’re well on our way to blowing past those numbers in October.

Wild.

See you all on Monday!

-Lunduke

Read full Article
See More
Available on mobile and TV devices
google store google store app store app store
google store google store app tv store app tv store amazon store amazon store roku store roku store
Powered by Locals