The rise of the Internet and of Open Standards resulted in the adoption of a number of fundamental mechanisms to enable communication and collaboration between different systems.
One such mechanism is the ability to encode strings or characters to bytes or to decode strings or characters from bytes. Different encoding standards have been developed over the years and Pharo supports many current and legacy encodings.
Another important aspect of collaboration is the ability to describe resources such as files. Both Mime-Type and URLs or URIs are basic building blocks for creating meta descriptions of resources and Pharo also has objects that implement these fundamental aspects.
In this chapter we discuss Character encoding, MIME types and URL/URIs. They are essential for the correct implementation of HTTP, but they are independent from it, as they are used for many other purposes.
We will first show how to get Unicode from characters and strings within Pharo. We will then show how to decode and encode characters and strings from and to bytes.
Proper character encoding and decoding is crucial in today's international world. Internally, Pharo stores characters and strings using Unicode. Unicode is a very large internationally standardized collection of code points (integer numbers) representing all of the world languages' characters.
We can obtain the code point (Unicode value) of a character by sending it the
codePoint message, for example:
Here are some example strings in multiple languages with their Unicode code points:
For a simple language like English, all characters have code points below 128 (which fits in 7 bits, for historical reasons). These characters are part of ASCII. The very first part of the so called Basic Multilingual Plane of Unicode (the first 128 code points of it) are identical to ASCII.
Next come a number of European languages, like French, which have code points below 256 (fitting in 8 bits or one byte). These characters are part of Latin-1 (ISO-8859-1), whose first 256 code points are identical in Unicode.
And finally, there are hundreds of other languages, like Chinese, Japanese, Cyrillic, Arabic or Greek. You can see from the example above: Greece written in Greek, that those code points are higher than 256 (and thus no longer fit in one byte).
Unicode code points are often written using a specific hexadecimal notation. For example, the previous character, the Greek lowercase lambda, is written as
U+03BB. The Pharo inspector also shows this value next to the codepoint.
The good thing is, we can work with text in any language in Pharo. However, to display everything correctly a font must be used that is capable of showing all the characters (or glyphs) needed, for example Arial Unicode MS.
For communication with the world outside Pharo, the operating system, files, the internet, et cetera, we have to represent our strings as a collection of bytes. Yet code points are different to bytes, as will be shown below. Therefore we need a way to transform our internal strings into external collection of bytes and vice versa.
Character encoding is the standard way of converting a native Pharo string, i.e. a collection of Unicode code points, to a series of bytes. Character decoding is the reverse process: interpreting a series of bytes as a collection of Unicode code points, to create a Pharo string.
To implement character encoding or decoding, a concrete subclass of
ZnCharacterEncoder is used, e.g.
ZnUTF8Encoder. Character encoders do the following:
nextPut:toStream:) or string (message
next:putAll:startingAt:toStream:) onto a binary stream
encodeString:) to a byte array
nextFromStream:) or string (
encodedByteCountFor:) or string (
Character encoders do proper error handling, throwing an error of the
ZnCharacterEncodingError when something goes wrong. The strict/lenient setting controls some behavior in this respect, and this will be discussed later in this chapter.
The recommended encoding is the primary internet encoding: UTF-8. It is a variable length encoding that is optimized somewhat for ASCII and to a lesser degree for Latin1 and some other common European encodings.
The first use of encoders is to convert Strings to ByteArrays and vice-versa.
We however deal only indirectly with character encoders. The
String classes have some convenience methods to do encoding and decoding:
Our ASCII string,
'Hello' encodes identically using either UTF-8 or Latin-1.
Our French string,
'Les élèves français', encodes differently though.
The reason is that UTF-8 uses two bytes for the accented letters like é, è and ç.
Note how for Latin-1, and only for Latin-1 and ASCII, the Unicode code points are equal to the encoded byte values.
Our greek string,
'Ελλάδα', gives an error when we try to encode it using Latin-1.
The reason is that the Greek letters are outside of the alphabet of Latin-1. Still, UTF-8 manages to encode them using just two bytes.
The reverse process, decoding, is equally simple:
'Hello', being pure ASCII, can be decoded using either UTF-8 or Latin-1. Our French
'Les élèves français' is another story: using the wrong
encoding gives either gibberish or
ZnInvalidUTF8 error. The same is true for our Greek
You might wonder why in the first case the
latin1 encoder produced gibberish, while in the second case it gave an error.
This is because in the second case, there was a byte with value 149, which is outside its alphabet. So called byte encoders, like Latin-1,
take a subset of Unicode characters and compress them in 256 possible byte values. This can be seen by inspecting the character or byte domains of a
ZnByteEncoder, as follows:
Note that identifiers for encodings are interpreted flexibly (case and punctuation do not matter).
There exists a special
ZnNullEncoder that basically does nothing: it treats bytes are characters and vice versa. This is actually mostly equivalent to Latin-1 or ISO-8859-1. (And yes, that is a bit confusing.)
The second primary use of encoders is when dealing with streams. More specifically, when interpreting a binary read or write stream as a character stream. Note that at their lowest level, all streams to and from the operating system or network are binary and thus need the use of an encoder when treating them as character streams.
To treat a binary write stream as a character write stream, wrap it
ZnCharacterReadStream should be used to treat a binary read stream as a
character stream. Here is an example:
We used the message
on:encoding: here, but there is also a plain message
on: instance creation message that defaults to the UTF-8 encoding.
Internally, the character streams will use an encoder instance to do the actual work.
Up until now we spoke about Strings as being a collection of Characters, each of which is represented as a Unicode code point.
And this is conceptually totally how they should be thought about. However, in reality, the class
String is an abstract class with two concrete subclasses.
This will show up when inspecting
String instances, so it is important
to understand what is going on. Consider the following example strings:
Simple ASCII strings are ByteStrings. Strings using special characters may be WideStrings or may still be ByteStrings.
The explanation of the use of the
ByteString class is very simple when considering the Unicode code points used for each character.
In the first case, for ASCII, the code points are always less than 128. Hence they fit in one byte. The second string is using Latin-1 characters, whose code
points are less than 256. These still fit in a byte. A
ByteString is a
only stores Unicode code points that fit in a byte, in an implementation that is very efficient. Note that
ByteString is a variable byte subclass of
Our last example has code points that no longer fit in a byte. To be able to store these,
WideString allocates 32-bit (4 byte) slots for each character. This
implementation is necessarily less efficient. Note that
WideString is a variable word subclass of
In practice, the difference between
WideString should not matter. Conversions are done automatically when needed.
As the above example shows, in a
'abc' putting the Unicode character
$α, converts it to a
WideString. (This is actually done using a
becomeForward: message.) When benchmarking, this conversion might show up as taking significant time. If you know upfront that you will need WideStrings, it can be better to start with the right type.
There is another implementation detail worth mentioning: for the Pharo virtual machine, more specifically, for a number of primitives,
ByteArray instances are equivalent. Given what we now know, that makes sense. Consider the following code:
In the third expression, we send the message
replaceFrom:to:with: on a
ByteString, but give a
ByteArray as third argument. So we are replacing part of a
ByteString with a
ByteArray. And it works!
The last example goes the other way around: we replace part of a
ByteArray with a
ByteString, which works as well.
What about doing the same mix up with elements ?
This is more what we expect: we're not allowed to do this. We are mixing two types that are not equivalent, like
So although it is true that there is some equivalence between ByteString and ByteArray, you should not mix up the two. It is an implementation detail that you should not rely upon.
Given a string, it is tempting to send it the message
asByteArray to convert it to bytes. Similary, it is tempting to convert a byte array by
sending it the message
asString. These are however bogus conversions that should not be used as for some strings they will work, but for others not. Success depends on the code points of the characters in the string. Basically
the conversion is possible for strings for which the following property holds:
Now, even though the first two can be converted, they will not be using the same encoding. Here is a way to explicitly express this idea:
For pure ASCII strings, with all code points below 128, no encoding (null encoding), ASCII, Latin-1 and UTF-8 are all the same. For other
'Les élèves français', only Latin-1 works. In that case it is also equivalent of doing no encoding.
The lazy conversion for proper Unicode WideStrings will give unexpected results:
This 'conversion' does not correspond to any known encoding. It is the result of writing 4-byte Unicode code points as Integers.
Using this is a bug no matter how you look at it. In this century you will look silly for not implementing proper support for all languages. When converting between strings and bytes, use a proper, explicit encoding.
No encoding (or the null encoder) and Latin-1 encoding are in fact not completely the same. This is because there are 'holes' in the table: some byte values are undefined, which a strict encoder won't allow, and the default encoder is strict.
For example, the Unicode code point 150 is strictly speaking not in Latin-1:
The encoder can however be instructed to
beLenient, which will produce a silent conversion (if that is possible). In this case, Unicode character 150 (
U+0096) is an unprintable control character meaning 'Start of Protected Area' (SPA) and is strictly speaking not part of Latin-1.
You can explicity access both the allowed byte or character values, i.e. the domain of encoder or decoder:
Note that the lower half of a byte encoding, the ASCII part between 0 and 127, is always treated as a one to one mapping.
Pharo comes with support for the most important encodings currently used, as well as with support for some important legacy encodings. Seen as the classes implementing them, the following encoders are available:
ZnByteEncoder groups a large number of encodings. This list is available as
ZnByteEncoder knownEncodingIdentifiers. Here is a list of all recognized, canonical names: arabic, cp1250, cp1251, cp1252, cp1253, cp1254, cp1255, cp1256, cp1257, cp1258, cp850, cp866, cp874, cyrillic, dos874, doslatin1, greek, hebrew, ibm819, ibm850, ibm866, iso885910, iso885911, iso885913, iso885914, iso885915, iso885916, iso88592, iso88593, iso88594, iso88595, iso88596, iso88597, iso88598, iso88599, koi8, koi8r, koi8u, latin2, latin3, latin4, latin5, latin6, mac, maccyrillic, macintosh, macroman, oem850, windows1250, windows1251, windows1252, windows1253, windows1254, windows1255, windows1256, windows1257, windows1258, windows874, xcp1250, xcp1251, xcp1252, xcp1253, xcp1254, xcp1255, xcp1256, xcp1257, xcp1258, xmaccyrillic and xmacroman.
A mime-type is a standard, cross-platform definition of a file or document type or format. The official term is an Internet media type.
Mime-types are modeled using
ZnMimeType objects, which have 3 components:
The mime-type syntax is as follows:
ZnMimeType are created by explicitly specifying its components, through parsing a string or by accessing predefined values. In any case, a new instance is always created.
The class side of
ZnMimeType has some convenience methods (in the protocol
convenience) for accessing well known mime-types, which is the recommended way for obtaining these mime-types:
Here is an example of how to create a mime-type by explicitly specifying its components:
The main parsing interface of
ZnMimeType is the class side
To make it easier to write code that accepts both instances and strings, the
asZnMimeType message can be used:
ZnMimeType also knows how to convert file name extensions to mime-types using the
forFilenameExtension: message. This mapping is based on the
/etc/mime.types file, which is encoded into the
In most applications, the concept of a default mime-type exists. It basically means: we don't know what these bytes represent.
Once you have a ZnMimeType instance, you can access its components using the
An important aspect of mime-types is whether the type is textual or binary, which is testable with the
isBinary message. Typically, text, XML or JSON are considered
textual, while images are binary.
For textual (non-binary) types, the encoding (or charset parameter) defaults to UTF-8, the prevalent internet standard. With the convencience messages
clearCharSet you can manipulate the charset parameter.
Comparing mime-types using the standard
= message takes all components into account, including the parameters. Different parameters lead to different
mime-types. As a result, when charsets are involved it is often better to compare using the
matches: message, as follows:
The charset=UTF-8 that is part of what
ZnMimeType textPlain returns is not taken into account in the second set of comparisons.
The main or sub types can be a wildcard, indicated by a
*. This allows for matching. Obviously, everything matches
ZnMimeType any). Otherwise,
when the sub type is
*, the main types must be equal. Here is an example.
URLs (or URIs) are a way to name or identify an entity. Often, they also contain information of where the entity they name or identify can be accessed.
We will be using the terms URL (Uniform Resource Locator) and URI
(Uniform Resource Identifier) interchangeably, as is most commonly done in practice. A URI is just a
name or identification, while a URL also contains information on how
to find or access a resource. Consider the following example: the URI
identifies and names a document, while the URL
http://john-doe.com/documents/cv.html also specifies that we can use HTTP to access this
resource on a specific server.
By considering most parts of an URL as optional, we can use one abstraction to implement both URI and URL using one class. The class
ZnUrl models URLs (or URIs) and has the following components:
The syntax of the external representation of a ZnUrl informally looks like this:
ZnUrls are most often created by parsing an external representation using either the
fromString: class message or by sending the
convenience message to a string.
The same instance can also be constructed programmatically:
ZnUrl components can be manipulated destructively. Here is an example:
Some characters of parts of a URL are considered as illegal because including them would interfere with the syntax and further processing. They thus have to be encoded. The methods of
ZnUrl in the
accessing protocols do not do any encoding, while those in
printing do. Here is an example:
The ZnUrl parser is somewhat forgiving and accepts some unencoded URLs as well, like most browsers would.
ZnUrl can parse in the context of a default scheme, like a browser would do.
Given a known scheme, ZnUrl knows its default port, and this is accessed by
A path defaults to what is commonly referred to as slash, which is testable with
isSlash. Paths are most often (but don't have to be) interpreted as filesystem paths. To support this, the
isDirectoryPath tests and
directory accessors are provided.
ZnUrl has some support to handle one URL in the context of another one, this is also known as a relative URL in the context of an absolute URL. This is implemented using the
inContextOf: methods. For example:
To add operations to URLs you could add an extension method to the ZnUrl class. In many cases though, it will not work on all kinds of URLs but only on a subset. In other words, you need to dispatch, not just on the scheme but maybe even on other URL elements. That is where
ZnUrlOperation comes in.
The first step for its use is defining a name for the operation. For example, the symbol
#retrieveContents. Second, one or more subclasses of
ZnUrlOperation need to be defined, each defining the class side message
operation to return the name,
#retrieveContents in the example. Then all subclasses with the same operation form the group of applicable implementations. Third, these handler subclasses overwrite
performOperation to do the actual work.
Given a ZnUrl instance, sending the message
performOperation:with: will send the message
ZnUrlOperation. In turn, it
will look for an applicable handler subclass, instanciate and invoke it.
Each subclass will be sent
handlesOperation:with:on: to test if it can handle the named operation with an optional argument on a specific URL. The default implementation already covers the most common case: the operation name has to match and the scheme of the URL has to be part of the collection returned by
For our example, the message
retrieveContents on ZnUrl is implemented as an operation named
#retrieveContents. The handler class is either the class
for the schemes
https or the class
ZnFileRetrieveContents for the scheme
This dispatching mechanism is more powerful than scheme specific
ZnUrl subclasses because other elements can be taken into account. It also addresses another issue with scheme specific
ZnUrl subclasses, which is that there are an infinite number of schemes which no hierarchy could cover.
Sometimes, the combination of a host and port are referred to as authority, and this is accessable with the
There are convenience methods to download the resource a ZnUrl points
saveContentsToFile. The first retrieves the contents and returns it directly, while the expression saves the contents directly to a file.
ZnUrl can be used to handle file URLs. Use
isFile to test for this scheme.
Given a file URL, it can be converted to a regular
FileReference using the
asFileReference message. In the other direction, you can get a file URL from a
FileReference using the
asZnUrl messages. Do keep in mind that there is no such thing as a relative file URL, only absolute file URLs exist.