Lunduke
News • Science & Tech
The Unlikely Story of UTF-8: The Text Encoding of the Web
Plan 9, Placemats, New Jersey Diners, and last minute ideas
June 22, 2023
post photo preview

If you are reading this on a computer -- of any kind -- odds are good that the words on the screen are all encoded using something called "UTF-8".

UTF-8 (or "Unicode Transformation Format - 8 bit") is, put simply, a format for encoding and storing text -- one which allows for far more text characters than the older "ASCII" encoding (which could only show a total of 95 printable characters).

And UTF-8 is, quite simply, everywhere.

Nearly every major computer operating system heavily uses UTF-8 for handling text... likewise it is the standard for websites, with close to 100% of all webpages explicitly using UTF-8 for the text on the page.

test
The source for Wikipedia.  Like most of the web, using UTF-8.

An argument could be made that UTF-8 is one of the most successful and widely adopted standards in all of computer history.

But this almost wasn't the case.

In fact, UTF-8 was created -- at the very last possible moment -- and it was first implemented in a computer system that most people don't even know existed.

X/Open's search for better text encoding

In the early 1990s, text encoding was... an issue.

While solutions for extended character sets (beyond simple ASCII characters) existed, they were less than ideal.  To put it mildly.  The most popular solution, known as UTF-1 (aka "ISO 10646"), suffered from serious performance issues... and often caused significant problems with software which used plain "ASCII" text (including UNIX file system paths).

Having a character encoding on UNIX systems that could cause problems with UNIX file systems?  Not good.

Obviously a new type of text encoding was needed.

So, in 1992, X/Open (originally known as the "Open Group for UNIX Systems", a consortium of UNIX vendors, including: Sun, HP, AT&T, IBM, and several others) set about the task of selecting a proper text encoding standard to be used across all of the UNIX world.

The proposal that gained the most traction was known as FSS/UTF (aka "File System Safe Universal Character Set Transformation Format").  Roills off the toungue, right?

This proposal was both faster than the old text encoding standard... and, as the name suggests, it was "File System Safe".  Which was a big win.

Enter: The Plan 9 Nerds

Which brings us to September 2nd, 1992.  Sometime in the early evening.

The X/Open group was meeting, in Austin, Texas, to formally decide on the file encoding standard.

Looking to get some feedback on the proposal, some members of X/Open made a call to two legendary programmers -- Ken Thompson and Rob Pike -- who were working on the Plan 9 Operating System project at Bell Labs in New Jersey.


A little background...

Ken Thompson was one of the creators of MULTICS, UNIX, the B programming language (the predecessor to C), among many other accomplishments.

Rob Pike, also a UNIX programmer, was the co-creator of Blit, writer of multiple UNIX and programming books, and the creator of the first UNIX windowing system.

To call these two "absolute legends" in the world of computing would be, perhaps, a bit of an understatement.  The two were currently working together on a research operating system, at Bell Labs, called Plan 9.  An attempt to fix some of the perceived shortcomings of UNIX... by the creators of UNIX, itself.


What happened next... after Ken Thompson and Rob Pike received that phone call?  Luckily, we have a detailed accounting... written by Rob Pike, himself.

"We had used the original UTF from ISO 10646 to make Plan 9 support 16-bit characters, but we hated it.  We were close to shipping the system when, late one afternoon, I received a call from some folks, I think at IBM - I remember them being in Austin - who were in an X/Open committee meeting.  They wanted Ken and me to vet their FSS/UTF design."

Asking two legendary engineers for their input?  You can probably guess what happened next...

"Ken and I suddenly realized there was an opportunity to use our experience to design a really good standard and get the X/Open guys to push it out.  We suggested this and the deal was, if we could do it fast, OK."

That's right.  Ken and Rob had some ideas.  And the X/Open folks agreedd to listen to those ideas... if they could get them something fast.

And, by fast, they really meant "immediately... like... right now."  Because the X/Open team were, quite literally, all gathered in Austin to decide on this... right then.

"Yeah.  I could eat."

Ken and Rob did what any good programmers would do when placed on an almost impossibly tight deadline -- and needed to come up with an amazing idea that could change the course of computing for decades to come... they went out to grab some grub.

"So we went to dinner, Ken figured out the bit-packing, and when we came back to the lab after dinner we called the X/Open guys and explained our scheme.  We mailed them an outline of our spec, and they replied saying that it was better than theirs (I don't believe I ever actually saw their proposal; I know I don't remember it) and how fast could we implement it?  I think this was a Wednesday night and we promised a complete running system by Monday, which I think was when their big vote was."

Remember.  This was 1992.

Which means, while laptops and such certainly existed, most people (even legendary programmers) did not have any sort of mobile, portable computers.  Certainly not the kind you could take out to a restaurant.

So what, pray tell, did they write their new text encoding design on?

A placemat from a New Jersey diner.

This is not the placemat that UTF-8 was designed on.

Seriously.

"UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner."

The boys, Ken and Rob, now had just a few days to get all of this done -- before the big vote on the new text encoding standard.  And they sure as heck didn't waste any time.

They got back from dinner, placemat in hand, and got to work.

"So that night Ken wrote packing and unpacking code and I started tearing into the C and graphics libraries.  The next day all the code was done and we started converting the text files on the system itself.  By Friday some time Plan 9 was running, and only running, what would be called UTF-8.  We called X/Open and the rest, as they say, is slightly rewritten history."

They converted an entire operating system over to a brand new -- just designed on a placemat -- text encoding format... in less than two days.

Here's the rough time-line:

  • Wednesday (Sep 2) evening: Dinner at a New Jersey Diner.  Ken Sketches out the idea on a placemat.
  • Wednesday night: Coding begins.
  • Thursday: Coding complete.
  • Friday: Entire Plan 9 operating system is now using "UTF-8".
  • Monday (Sep 7): X/Open group votes to use the Ken/Rob encoding design.

On Tuesday, September 8th, 1992 (at 3:22am), mere hours after the official vote to accept their text encoding design, Ken Thompson sends out the following email regarding Plan 9 now using UTF-8:

"The code has been tested to some degree and should be pretty good shape.  We have converted Plan 9 to use this encoding and are about to issue a distribution to an initial set of university users."

That's right.

Ken and Rob got a call asking for feeback on a Wednesday.  By the next Tuesday (at 3am) they were ready to ship a version of their Plan 9 OS with all the changes, and their designs had been voted on by the largest UNIX companies in the world.

Like I said.

A recent picture of the two legends, themselves.

These guys are legends.

What about that placemat?

Considering the vast impact of UTF-8 on the world of computing... whatever happened to that original "design document" (aka "the placemat")?  It would certainly be of historic significance.

"I very clearly remember Ken writing on the placemat and wished we had kept it!"

Let this be a lesson to all of the programmers out there:

Keep all of you doodles, notes, and sketches you make for your projects... you never know when one of those projects will become critical to the entire world... making your quick sketch worthy of being in a museum.

Especially if it's on a placemat.  From a diner.  In New Jersey.


Copyright © 2023 by Bryan Lunduke.  All rights reserved.  The contents of this article are licensed under the terms of The Lunduke Content Usage License.

community logo
Join the Lunduke Community
To read more articles like this, sign up and join my community today
11
What else you may like…
Videos
Podcasts
Posts
Articles
GNOME Foundation Executive Director Out After 4 Months

His 4 month reign highlighted by GNOME attacks on Jews and visually impaired users, an expanded "Pride Month", pledging fealty to the UN, and defense of a registered sex offender.

The Article:
https://lunduke.substack.com/p/gnome-foundation-executive-director

More from The Lunduke Journal:
https://lunduke.com/

00:25:19
4Chan and Kiwi Farms File Lawsuit Against UK

It is both an important legal case... and a brilliant trolling of the British government.

The article:
https://lunduke.substack.com/p/4chan-and-kiwi-farms-file-lawsuit

More from The Lunduke Journal:
https://lunduke.com/

00:21:00
Microsoft Fires "Intifada" Employees

This last week, a group of anti-Jewish Microsoft employees got rowdy. Microsoft fired some of them and sent The Lunduke Journal a statement. Then held a media briefing. Let's watch it together.

More from The Lunduke Journal:
https://lunduke.com/

00:27:39
November 22, 2023
The futility of Ad-Blockers

Ads are filling the entirety of the Web -- websites, podcasts, YouTube videos, etc. -- at an increasing rate. Prices for those ad placements are plummeting. Consumers are desperate to use ad-blockers to make the web palatable. Google (and others) are desperate to break and block ad-blockers. All of which results in... more ads and lower pay for creators.

It's a fascinatingly annoying cycle. And there's only one viable way out of it.

Looking for the Podcast RSS feed or other links? Check here:
https://lunduke.locals.com/post/4619051/lunduke-journal-link-central-tm

Give the gift of The Lunduke Journal:
https://lunduke.locals.com/post/4898317/give-the-gift-of-the-lunduke-journal

The futility of Ad-Blockers
November 21, 2023
openSUSE says "No Lunduke allowed!"

Those in power with openSUSE make it clear they will not allow me anywhere near anything related to the openSUSE project. Ever. For any reason.

Well, that settles that, then! Guess I won't be contributing to openSUSE! 🤣

Looking for the Podcast RSS feed or other links?
https://lunduke.locals.com/post/4619051/lunduke-journal-link-central-tm

Give the gift of The Lunduke Journal:
https://lunduke.locals.com/post/4898317/give-the-gift-of-the-lunduke-journal

openSUSE says "No Lunduke allowed!"
September 13, 2023
"Andreas Kling creator of Serenity OS & Ladybird Web Browser" - Lunduke’s Big Tech Show - September 13th, 2023 - Ep 044

This episode is free for all to enjoy and share.

Be sure to subscribe here at Lunduke.Locals.com to get all shows & articles (including interviews with other amazing nerds).

"Andreas Kling creator of Serenity OS & Ladybird Web Browser" - Lunduke’s Big Tech Show - September 13th, 2023 - Ep 044

I wasn't able to cover many ads in August, so I uploaded a big IBM interview: https://computeradsfromthepast.substack.com/p/byte-interviews-ibms-philip-d-estridge

Enjoy.

August 31, 2025

I have to admit to having been a bit prejudiced against MorphOS, more or less since the beginning. Back then it just seemed as if it was sowing division in an Amiga community that needed, more than anything else, to hang together. That, and it only running on PowerPC, meant I'd never really taken a close look at it before.

But now I have a G4 Mac Mini, and there's not much else you can do with one of those except install MorphOS on it.

I am seriously impressed. It's way more slick and polished than I expected - much more so than AROS - and the integration of 68k Amiga applications is utterly seamless. Personal Paint, my go-to program for pixel art, just works. And it's faster than it is in UAE on my PC, which is amazing given that this is 20-year-old hardware. (I assume it's taking advantage of PPC-native libraries where possible, while UAE has to emulate everything. Even then, it's surprising how much difference that makes. In UAE, running as fast as possible, it loads and remaps ...

I have Mint 22.1 using xlibre on a X230. It is running VERY well and seems noticeably smoother than the previous X11 it was using.

post photo preview
Fired Microsoft Employee Encouraged Corporate Sabotage
Leaked screenshots from a Microsoft whistleblower reveals efforts, by a "Worker Intifada" organizer, to disrupt Microsoft business and send spam propaganda email to thousands.

This last week, members of the Microsoft “Worker Intifada” claimed that some of their members were fired by Microsoft in an attempt to silence the speech of pro-Palestinian employees.

Thanks to an internal Microsoft whistleblower, we have gained additional insight into the real reasons at least one of the “Intifada” organizers was fired.

Including mass emailing thousands of Microsoft employees with anti-Microsoft messages, and building internal, corporate websites dedicated to disrupting company business. Over the course of months.

And, of course, we’ve got screenshots.

The Microsoft “Intifada”

The Microsoft’s “Worker Intifada” held a press conference, on August 28th, at the office of CAIR — a response to a Microsoft’s Media briefing held 2 days earlier.

The topic of both press events being the “Intifada” actions of the previous week — including vandalism, breaking and entering, and public demonstrations against Jewish people (with chants of “Go away, Jews!”).

A key topic being the firing of 4 Microsoft employees related to the “Intifada”.

One of those fired employees is Nisreen Jaradat, seen here speaking at the press conference at CAIR on August 28th:

 

Thanks to leaked Microsoft material, we can get a glimpse into the anti-Microsoft activity of Nisreen Jaradat… going back several months.

Anti-Microsoft Microsoft Employee

Nisreen created an internal website — on the Microsoft corporate Intranet — entitled “PledgeForPalestine”.

 

On that website, Nisreen encourages employees to take a pledge declaring that employees will “not support genocide”, specifically demanding that Microsoft no longer interact with Israel in any way.

It should be noted that, as of just a few days ago, this website was up within Microsoft.

Nisreen’s website also includes a list of ways where employees could sabotage internal corporate work.

 

Nisreen encouraged employees to:

  1. Refuse to work on support tickets that involve Israel (and sabotage any tickets worked on by others).

  2. Refuse to work with other employees on tasks which may involve Israel — or any Israeli Microsoft employees or customers.

In July, Nisreen sent internal Microsoft Teams messages to thousands of employees encouraging them to sign that anti-Israel and anti-Microsoft pledge.

 

But this wasn’t the first time Nisreen spammed Microsoft employees with pro-Palestinian propaganda.

Back in May of this year, Nisreen sent an email to over 8,000 Microsoft employees entitled “You can’t get rid of us.”

 

Within that email, she declared that her employers have “shown their true face, brutalizing, detaining, firing, pepper spraying, threatening and insulting workers and former workers protesting.”

Surprise, She Got Fired

Protesting your bosses and accusing them of “genocide” while encouraging thousands of other employees, in spam emails, to sabotage corporate work?

Now, think what you will about Microsoft… but I can’t imagine any company not firing someone who takes those actions internally.

In fact, I would suggest that it is wild that Microsoft put up with the actions of this “Intifada” organizer for so many months.


If you would like to be a Tech whistleblower — or would like to support the work of The Lunduke Journal — head to Lunduke.com. All of the information is there.

Read full Article
post photo preview
Lunduke's Week in Tech - Aug 30, 2025
GNOME Loses Another Executive, Microsoft's Intifada, & 4Chan + KiwiFarms Sue the UK

Yet another absolutely insane week of Tech News has wrapped up.

From more Microsoft troubles with their “Intifada”… to the 4Chan & KiwiFarms lawsuit against the UK, it’s been one seriously nutty week.

Oh! And the GNOME Foundation lost yet another Executive Director! This one only lasted 4 months on the job! Crazy! All of the stories are linked below (and free for all to enjoy).

And, once again, it is amazing how many of these stories were almost completely ignored by Brand X Tech News outlets.

One story we did not see this week was the final ruling on the remedy for USA v. Google. The case that could determine wether Mozilla Corporation can continue to receive 80% of their income from Google.

That ruling was expected by yesterday… but, so far, all is quiet. I checked in with sources at both Google and Mozilla… and neither has heard a peep about it. Will we see the ruling from Judge Mehta this coming week? Who knows!

Stories This Week

Note: All articles, videos, & podcasts are free for all (subscribers and non-subscribers alike).

Note 2: Links for the stories are all to Substack. You can also find links to all other platforms — X, Rumble, Locals, YouTube, etc. — which The Lunduke Journal publishes to, at Lunduke.com.

As always, thank you to all of the subscribers to The Lunduke Journal. Thanks to you, we can remain 100% ad-free and Big Tech free. Couldn’t do it without your support.

-Lunduke

Read full Article
post photo preview
GNOME Foundation Executive Director Out After 4 Months
His 4 month reign highlighted by GNOME attacks on Jews and visually impaired users, an expanded "Pride Month", pledging fealty to the UN, and defense of a registered sex offender.

As of today, the GNOME Foundation has lost yet another Executive Director — Steven Deobald, who lasted a short 4 months.

The previous executive director, Holly Million, lasted a grand total of 9 months in the role.

To say things have been chaotic within the GNOME Foundation, would be a wild understatement — with last year seeing GNOME elections overturned in secret meetings and massive cashflow issues.

The last 4 months, since the new Executive Director took the position, hasn’t exactly been smooth sailing. Here’s a quick timeline of just some of the strange activity since then.

And now today, August 29th, their new Executive Director is out. After only 4 months on the job. Did any of those events contribute to the departure? GNOME isn’t saying.

In fact, the statement from Allan Day, the new Acting GNOME Executive Director, is incredibly vague regarding the reason for the change.

“Steven Deobald has been in the post of GNOME Foundation Executive Director for the past four months, during which time he has made major contributions to both the Foundation and the wider GNOME project. Sadly, Steven will be leaving the Foundation this week. The Foundation Board is extremely grateful to Steven and wish him the very best for his future endeavors.”

According to the outgoing Executive Directors own statement — entitled “So short, and thanks for all the flinch”:

“As the board announced earlier today, I will be stepping down from the Executive Director role this week. It’s been an interesting four months.”

That reads to me like a firing (or a forced layoff due to lack of funds). But, unless someone speaks up, it’s unlikely we’ll know for sure.

The following was posted by Deobald (outgoing Executive) and Allan Day (new acting Executive) on the GNOME Foundation’s Matrix chat channel.

 

The statement, “I feel like I’ve been glued to an Emacs buffer for 3 straight days” suggests 3 days of discussions (or waiting) leading up to this change.

What will happen next for the GNOME Foundation?

With multiple years of absolute chaos — including financial troubles and an inability to have leadership last more than a few months (not to mention an obsession with attacking The Lunduke Journal) — it’s hard to imagine the next few months being anything other than a continuation of the insanity at GNOME.

Read full Article
See More
Available on mobile and TV devices
google store google store app store app store
google store google store app tv store app tv store amazon store amazon store roku store roku store
Powered by Locals