Lunduke
News • Science & Tech
The Unlikely Story of UTF-8: The Text Encoding of the Web
Plan 9, Placemats, New Jersey Diners, and last minute ideas
June 22, 2023
post photo preview

If you are reading this on a computer -- of any kind -- odds are good that the words on the screen are all encoded using something called "UTF-8".

UTF-8 (or "Unicode Transformation Format - 8 bit") is, put simply, a format for encoding and storing text -- one which allows for far more text characters than the older "ASCII" encoding (which could only show a total of 95 printable characters).

And UTF-8 is, quite simply, everywhere.

Nearly every major computer operating system heavily uses UTF-8 for handling text... likewise it is the standard for websites, with close to 100% of all webpages explicitly using UTF-8 for the text on the page.

test
The source for Wikipedia.  Like most of the web, using UTF-8.

An argument could be made that UTF-8 is one of the most successful and widely adopted standards in all of computer history.

But this almost wasn't the case.

In fact, UTF-8 was created -- at the very last possible moment -- and it was first implemented in a computer system that most people don't even know existed.

X/Open's search for better text encoding

In the early 1990s, text encoding was... an issue.

While solutions for extended character sets (beyond simple ASCII characters) existed, they were less than ideal.  To put it mildly.  The most popular solution, known as UTF-1 (aka "ISO 10646"), suffered from serious performance issues... and often caused significant problems with software which used plain "ASCII" text (including UNIX file system paths).

Having a character encoding on UNIX systems that could cause problems with UNIX file systems?  Not good.

Obviously a new type of text encoding was needed.

So, in 1992, X/Open (originally known as the "Open Group for UNIX Systems", a consortium of UNIX vendors, including: Sun, HP, AT&T, IBM, and several others) set about the task of selecting a proper text encoding standard to be used across all of the UNIX world.

The proposal that gained the most traction was known as FSS/UTF (aka "File System Safe Universal Character Set Transformation Format").  Roills off the toungue, right?

This proposal was both faster than the old text encoding standard... and, as the name suggests, it was "File System Safe".  Which was a big win.

Enter: The Plan 9 Nerds

Which brings us to September 2nd, 1992.  Sometime in the early evening.

The X/Open group was meeting, in Austin, Texas, to formally decide on the file encoding standard.

Looking to get some feedback on the proposal, some members of X/Open made a call to two legendary programmers -- Ken Thompson and Rob Pike -- who were working on the Plan 9 Operating System project at Bell Labs in New Jersey.


A little background...

Ken Thompson was one of the creators of MULTICS, UNIX, the B programming language (the predecessor to C), among many other accomplishments.

Rob Pike, also a UNIX programmer, was the co-creator of Blit, writer of multiple UNIX and programming books, and the creator of the first UNIX windowing system.

To call these two "absolute legends" in the world of computing would be, perhaps, a bit of an understatement.  The two were currently working together on a research operating system, at Bell Labs, called Plan 9.  An attempt to fix some of the perceived shortcomings of UNIX... by the creators of UNIX, itself.


What happened next... after Ken Thompson and Rob Pike received that phone call?  Luckily, we have a detailed accounting... written by Rob Pike, himself.

"We had used the original UTF from ISO 10646 to make Plan 9 support 16-bit characters, but we hated it.  We were close to shipping the system when, late one afternoon, I received a call from some folks, I think at IBM - I remember them being in Austin - who were in an X/Open committee meeting.  They wanted Ken and me to vet their FSS/UTF design."

Asking two legendary engineers for their input?  You can probably guess what happened next...

"Ken and I suddenly realized there was an opportunity to use our experience to design a really good standard and get the X/Open guys to push it out.  We suggested this and the deal was, if we could do it fast, OK."

That's right.  Ken and Rob had some ideas.  And the X/Open folks agreedd to listen to those ideas... if they could get them something fast.

And, by fast, they really meant "immediately... like... right now."  Because the X/Open team were, quite literally, all gathered in Austin to decide on this... right then.

"Yeah.  I could eat."

Ken and Rob did what any good programmers would do when placed on an almost impossibly tight deadline -- and needed to come up with an amazing idea that could change the course of computing for decades to come... they went out to grab some grub.

"So we went to dinner, Ken figured out the bit-packing, and when we came back to the lab after dinner we called the X/Open guys and explained our scheme.  We mailed them an outline of our spec, and they replied saying that it was better than theirs (I don't believe I ever actually saw their proposal; I know I don't remember it) and how fast could we implement it?  I think this was a Wednesday night and we promised a complete running system by Monday, which I think was when their big vote was."

Remember.  This was 1992.

Which means, while laptops and such certainly existed, most people (even legendary programmers) did not have any sort of mobile, portable computers.  Certainly not the kind you could take out to a restaurant.

So what, pray tell, did they write their new text encoding design on?

A placemat from a New Jersey diner.

This is not the placemat that UTF-8 was designed on.

Seriously.

"UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner."

The boys, Ken and Rob, now had just a few days to get all of this done -- before the big vote on the new text encoding standard.  And they sure as heck didn't waste any time.

They got back from dinner, placemat in hand, and got to work.

"So that night Ken wrote packing and unpacking code and I started tearing into the C and graphics libraries.  The next day all the code was done and we started converting the text files on the system itself.  By Friday some time Plan 9 was running, and only running, what would be called UTF-8.  We called X/Open and the rest, as they say, is slightly rewritten history."

They converted an entire operating system over to a brand new -- just designed on a placemat -- text encoding format... in less than two days.

Here's the rough time-line:

  • Wednesday (Sep 2) evening: Dinner at a New Jersey Diner.  Ken Sketches out the idea on a placemat.
  • Wednesday night: Coding begins.
  • Thursday: Coding complete.
  • Friday: Entire Plan 9 operating system is now using "UTF-8".
  • Monday (Sep 7): X/Open group votes to use the Ken/Rob encoding design.

On Tuesday, September 8th, 1992 (at 3:22am), mere hours after the official vote to accept their text encoding design, Ken Thompson sends out the following email regarding Plan 9 now using UTF-8:

"The code has been tested to some degree and should be pretty good shape.  We have converted Plan 9 to use this encoding and are about to issue a distribution to an initial set of university users."

That's right.

Ken and Rob got a call asking for feeback on a Wednesday.  By the next Tuesday (at 3am) they were ready to ship a version of their Plan 9 OS with all the changes, and their designs had been voted on by the largest UNIX companies in the world.

Like I said.

A recent picture of the two legends, themselves.

These guys are legends.

What about that placemat?

Considering the vast impact of UTF-8 on the world of computing... whatever happened to that original "design document" (aka "the placemat")?  It would certainly be of historic significance.

"I very clearly remember Ken writing on the placemat and wished we had kept it!"

Let this be a lesson to all of the programmers out there:

Keep all of you doodles, notes, and sketches you make for your projects... you never know when one of those projects will become critical to the entire world... making your quick sketch worthy of being in a museum.

Especially if it's on a placemat.  From a diner.  In New Jersey.


Copyright © 2023 by Bryan Lunduke.  All rights reserved.  The contents of this article are licensed under the terms of The Lunduke Content Usage License.

community logo
Join the Lunduke Community
To read more articles like this, sign up and join my community today
11
What else you may like…
Videos
Podcasts
Posts
Articles
Linux Distro Declared "Non-Woke", Causing New Users & Interest to Skyrocket

While "Woke" Linux Distributions (like openSUSE) are struggling to survive, interest in OpenMandriva - a non-Woke Open Source project - is exploding.

00:29:15
January 19, 2025
Godot Game Engine Went Woke... And Lost $160,000 Per Year.

A few months ago, the Open Source Godot Game Engine enacted a mass "cleansing" of Non-Woke supporters. Then their monthly donations dropped by over 25%.

00:10:46
January 17, 2025
Tech Journalists Rally in Support of Racism and Sexism in Big Tech

"Save me from rich, white men," declares the Tech Journalists in response to some Big Tech firms decreasing racist DEl spending.

00:31:20
November 22, 2023
The futility of Ad-Blockers

Ads are filling the entirety of the Web -- websites, podcasts, YouTube videos, etc. -- at an increasing rate. Prices for those ad placements are plummeting. Consumers are desperate to use ad-blockers to make the web palatable. Google (and others) are desperate to break and block ad-blockers. All of which results in... more ads and lower pay for creators.

It's a fascinatingly annoying cycle. And there's only one viable way out of it.

Looking for the Podcast RSS feed or other links? Check here:
https://lunduke.locals.com/post/4619051/lunduke-journal-link-central-tm

Give the gift of The Lunduke Journal:
https://lunduke.locals.com/post/4898317/give-the-gift-of-the-lunduke-journal

The futility of Ad-Blockers
November 21, 2023
openSUSE says "No Lunduke allowed!"

Those in power with openSUSE make it clear they will not allow me anywhere near anything related to the openSUSE project. Ever. For any reason.

Well, that settles that, then! Guess I won't be contributing to openSUSE! 🤣

Looking for the Podcast RSS feed or other links?
https://lunduke.locals.com/post/4619051/lunduke-journal-link-central-tm

Give the gift of The Lunduke Journal:
https://lunduke.locals.com/post/4898317/give-the-gift-of-the-lunduke-journal

openSUSE says "No Lunduke allowed!"
September 13, 2023
"Andreas Kling creator of Serenity OS & Ladybird Web Browser" - Lunduke’s Big Tech Show - September 13th, 2023 - Ep 044

This episode is free for all to enjoy and share.

Be sure to subscribe here at Lunduke.Locals.com to get all shows & articles (including interviews with other amazing nerds).

"Andreas Kling creator of Serenity OS & Ladybird Web Browser" - Lunduke’s Big Tech Show - September 13th, 2023 - Ep 044
January 13, 2025
Last 24 Hours: 55% off DRM-Free MP4s, & Discounted Lifetime Subscriptions

Holy guacamole, Batman! We are now over 80% of the way to our funding goal… for the end of February. And it’s not even half way through January! I am absolutely gobsmacked.

Want to help push us over the 100% line — and directly fund the truly independent Tech Journalism of The Lunduke Journal?

Lots of options. Big and small (and all amazingly helpful). Scroll down & choose whatever works best for you.

Lunduke Journal Discounts

For the next 24-sh hours (through Tuesday, January 14th), in addition to regular subscriptions, you can grab two limited discounts:

  1. Discounted Lifetime Subscriptions (details below)

  2. DRM-Free, MP4 Downloads for 2024 & 2025 (combined) for 55% off.

There many ways to support The Lunduke Journal. Choose the option that makes you smile. It’s all listed below.

Where to grab a Monthly or Yearly Subscription:

Lifetime Subscription Details:

  • Pay once, full subscription for life (on Locals, Substack, or both).

  • Available only through Tuesday, January 14th. Then the Lifetime Subscription option goes “back in the vault”.

  • Can be purchased via Locals, Substack, or with Bitcoin. Scroll down for steps.

Where to buy a DRM-Free, MP4 video yearly download pass:

Want to be able to download every show The Lunduke Journal releases (and watch them on whatever device you like)? Yeah. You can do that.

The Famous Lifetime Subscription via Locals

The "World Famous Lunduke Journal Lifetime Subscription" is exactly what it sounds like. Pay once and get full access to The Lunduke Journal (with all the perks of subscription on Locals). For life.

New Lifetime Subscriptions are available, for $200, from now through January 14th. Then this option goes "back into the vault".

Here's how to grab one of these coveted bad boys for yourself:

  1. Go to Lunduke.Locals.com/support.

  2. Select "Give Once".

  3. Enter "200" into the amount field.

  4. After checking out, Lunduke will toss you an email once your account is set to full lifetime status. (This usually happens within a few hours.)

The Famous Lifetime Subscription via Substack

You can also snag a Lifetime Subscription via Substack:

  1. Go to Lunduke.Substack.com/subscribe.

  2. Select the “Lifetime Subscription” option.

  3. After checking out, Lunduke will toss you an email once your account is set to full lifetime status. (This usually happens within a few hours.)

If you would like full, Lifetime access to Lunduke.Locals.com (which is included):

  1. Make a free account on Lunduke.Locals.com.

  2. Email “bryan at lunduke.com” with the email address you use on both Substack and Locals (can be different email addresses).

  3. Lunduke will toss you an email once your account is set to full lifetime status on Locals.

The Famous Lifetime Subscription (with Bitcoin discount)

And, finally, you can obtain a Lifetime Subscription via Bitcoin. Save a few bucks with this option, as Bitcoin processing has fewer fees associated with it.

  • Make sure you have a Lunduke.Locals.com account (a free account works just fine).

  • Send $190 worth of Bitcoin (or more) to the following address:

bc1qyjakve8fywm8pz2v99v57yhjj0vzr2vjze6fcq

  • Email "bryan at lunduke.com" with the following information: What time you made the transaction, how much was sent (in Bitcoin), and the email address you use (or plan to use) on Locals.com.

Nice and easy. Lots of options.

Once again, thank you. The Lunduke Journal would not be possible without your support.

You rule.

-Lunduke

Read full Article
January 12, 2025
Discounted Lifetime Subs, 55% Discount on MP4 Downloads for 2 Days

The Lunduke Journal remains 100% free from advertising… and, even more importantly, 100% free from all Big Tech influence.

And, because we never take a penny from any company, this is only possible thanks to you.

All of you amazing nerds showed tremendous support for The Lunduke Journal last month — grabbing Lifetime and standard subscriptions — meeting our necessary funding requirements for January… before January even started.

Mind. Blown.

This allows me to focus on new reporting — without needing to worry so much about funding and “business stuff” during the month. Which means more shows & more investigations. A massive win for all of us.

That also means that, this month, we don’t need to do a big “fundraiser”. Instead, we’re going to just send out this quick reminder of how absolutely critical your support is for the continued success The Lunduke Journal — the last bastion of truly independent Tech Journalism.

Seriously. Couldn’t do this without you.

Scroll down and take a look at the options. See if any jump out at you. Then give yourself a high five for making Tech Journalism that much more awesome.

Lunduke Journal Discounts

For the next 2 days (through Wednesday, January 15th), in addition to regular subscriptions, you can grab two limited discounts:

  1. Discounted Lifetime Subscriptions (details below)

  2. DRM-Free, MP4 Downloads for 2024 & 2025 (combined) for 55% off.

There many ways to support The Lunduke Journal. Choose the option that makes you smile. It’s all listed below.

Where to grab a Monthly or Yearly Subscription:

Lifetime Subscription Details:

  • Pay once, full subscription for life (on Locals, Substack, or both).

  • Available only until Wednesday, January 15th. Then the Lifetime Subscription option goes “back in the vault”.

  • Can be purchased via Locals, Substack, or with Bitcoin. Scroll down for steps.

Where to buy a DRM-Free, MP4 video yearly download pass:

Want to be able to download every show The Lunduke Journal releases (and watch them on whatever device you like)? Yeah. You can do that.

The Famous Lifetime Subscription via Locals

The "World Famous Lunduke Journal Lifetime Subscription" is exactly what it sounds like. Pay once and get full access to The Lunduke Journal (with all the perks of subscription on Locals). For life.

New Lifetime Subscriptions are available, for $200, from now through January 15th. Then this option goes "back into the vault".

Here's how to grab one of these coveted bad boys for yourself:

  1. Go to Lunduke.Locals.com/support.

  2. Select "Give Once".

  3. Enter "200" into the amount field.

  4. After checking out, Lunduke will toss you an email once your account is set to full lifetime status. (This usually happens within a few hours.)

The Famous Lifetime Subscription via Substack

You can also snag a Lifetime Subscription via Substack:

  1. Go to Lunduke.Substack.com/subscribe.

  2. Select the “Lifetime Subscription” option.

  3. After checking out, Lunduke will toss you an email once your account is set to full lifetime status. (This usually happens within a few hours.)

If you would like full, Lifetime access to Lunduke.Locals.com (which is included):

  1. Make a free account on Lunduke.Locals.com.

  2. Email “bryan at lunduke.com” with the email address you use on both Substack and Locals (can be different email addresses).

  3. Lunduke will toss you an email once your account is set to full lifetime status on Locals.

The Famous Lifetime Subscription (with Bitcoin discount)

And, finally, you can obtain a Lifetime Subscription via Bitcoin. Save a few bucks with this option, as Bitcoin processing has fewer fees associated with it.

  • Make sure you have a Lunduke.Locals.com account (a free account works just fine).

  • Send $190 worth of Bitcoin (or more) to the following address:

bc1qyjakve8fywm8pz2v99v57yhjj0vzr2vjze6fcq

  • Email "bryan at lunduke.com" with the following information: What time you made the transaction, how much was sent (in Bitcoin), and the email address you use (or plan to use) on Locals.com.

Nice and easy.

Once again, thank you. The Lunduke Journal would not be possible without your support.

You rule.

-Lunduke

Read full Article
January 09, 2025
post photo preview
The House of Lunduke BBS
Telnet: BBS.LUNDUKE.COM

The House of Lunduke BBS is a 20 node, Telnet BBS (bbs.lunduke.com) focused on the golden age of DOS BBS gaming -- including TradeWars 2002 and Legend of the Red Dragon.

 

 

The BBS is free to use, & creating an account takes only a moment.  All new users are granted 2 hours of access to all games on the system (more than enough to get in your daily turns of every one of these classic games).

Current games on the BBS:

  • TradeWars 2002 (aka TW2002)
  • Legend of the Red Dragon (aka LORD)
  • Legend of the Red Dragon 2
  • The Pit
  • Exitilus
  • DoorMUD
  • Barren Realms Elite

 

How to Connect

 

Accessing The House of Lunduke BBS is incredibly easy.  Simply point any Telnet client at "bbs.lunduke.com" on port 23 (the default Telnet port).

This means, on many systems, a simple:

telnet bbs.lunduke.com

Will, technically, work.

However, if you want to have the absolute best experience -- and see what BBSing was truly like back in the 1980s and 1990s -- you'll want to use a Telnet client capable of displaying ANSI color... specifically one tuned for BBSes.

The following Telnet clients are all recommended and work well with The House of Lunduke:

Once you have chosen your Telnet client, simply add "bbs.lunduke.com" to your address book and connect.  Or, from within some Telnet clients, you can also type "atdtbbs.lunduke.com" (no spaces).

Fun fact: "ATDT" is a command which tells an old-school modem to "tone dial this number".  The "AT" stands for "Attention", "D" stands for "Dial", and ending T stands for "Tone".  In the old days ATDT would be followed by a phone number... but, since we're connecting via Telnet, we now use a domain.

Or, if you want to get really old-school, it is possible to utilize DOSBox and an era appropriate terminal client (such as Telemate).

 

 

F.A.Q.

 

Q: What software does The House of Lunduke BBS run?

A: The core BBS software is known as Virtual Advanced (a successor to VBBS), a DOS-based BBS package that acheived some popularity during the early 1990s.  All of the games on the system (known as "Door games" in BBS terminology) also run on DOS -- and each is the most popular version (or the last version) of each.  A Telnet server handles incoming telnet connections -- which then interfaces with the DOS-based BBS software, using what is known as a FOSSIL driver.

 

Fun Fact: A FOSSIL driver is a serial interface driver -- which makes it easier to use a standard interface to multiple different communication devices (such as a type of dial-up modem or a Telnet connection).  FOSSIL stands for "Fido Opus SEAdog Standard Interface Layer".  Which is a fun name.

 

Q: What does "20 node" mean?

A: In "ye olden times", BBSes used modems and phone lines.  One person, at a time, could call a BBS on each phone line.  Each of these is a "node".  We are now living in the future -- and using Telnet instead of phone lines -- which means that, in theory, we could have a small mountain of people connecting at the same time.  However.  Since we are using 1980s and 1990s DOS software, jury-rigged to work with the modern Internet, some of those old limitations still apply.  20 nodes were chosen as that makes for an incredibly large BBS system... but not too difficult to manage.

 

Q: Why not use a more modern BBS server?

A: There are several "modern" BBS servers -- with many fancy benefits.  MysticBBS, Synchronet, and others.  The House of Lunduke BBS is not about being modern.  Or fancy.  It is about preserving a snapshot of a high-point in BBS gaming history.

Read full Article
See More
Available on mobile and TV devices
google store google store app store app store
google store google store app tv store app tv store amazon store amazon store roku store roku store
Powered by Locals