UTF-8LE - a more efficient, saner version of UTF-8

I just realised that UTF-8 is stupidly defined as big-endian!

U+20AC = 0010 000010 101100
In UTF-8 -> 11100010 10000010 10101100

...Meaning that to convert a codepoint into a series of bytes you have to shift the value before anding/adding the offset, viz.

b0 = (n >> 12) + 0xe0;
b1 = ((n >> 6) & 63) + 0x80;
b2 = (n & 63) + 0x80;

Just looking at the expression it doesn't seem so bad, but shifting right before means throwing away perfectly good bits in a register! The worst thing is, the bits thrown away are exactly the ones needed in the upcoming computations, so you have to needlessly waste storage to preserve the entire value of the codepoint throughout the computation. Observe:

push eax
shr eax, 12
add al, 224
stosb
pop eax
push eax
shr eax, 6
and al, 63
add al, 128
stosb
pop eax
and al, 63
add al, 128
stosb

14 instructions, 23 bytes. Not so bad, but what if we stored the pieces the other way around, i.e. "UTF-8LE"?

U+20AC = 001000 001010 1100
In UTF-8LE -> 11101100 10001010 10001000

b0 = (n&15) + 224;
b1 = ((n>>6)&63) + 128;
b2 = (n>>12) + 128;

Observe that each time bits are picked off n, the next step's shift removes them, so there is no need to keep around another copy of n (including bits that wouldn't be used anymore).

shl eax, 4
shr al, 4
add al, 224
stosb
mov al, ah
and al, 63
add al, 128
stosb
shr eax, 14
add al, 128
stosb

11 instructions, 22 bytes. Clearly superior.

The UTF-8 BOM is EF BB BF; the UTF-8LE BOM similarly will be EF AF BF.

And ditto for UTF-16's placement of bits within surrogates... but since UTF-16 already has UTF-16BE/UTF-16LE, I propose the more logical alternative be named UTF-16LEL.

UTF-16BEL could also be defined, but it makes about as much sense as UTF-16BE, i.e. none.

Stop using shitty little endian systems

What a coinkydink, my anus is big-endian.

>>3
Stop using shitty big-endian systems.

It doesn't take a genius to realise that the byte at offset n having value 256^L-n-1 is absolutely bloody-panties-on-head-retarded compared to it having value 256ⁿ.

Why is this level of optimization this so important for Unicode? It's no longer 1980, we can actually afford multi-Ghz CPU and many gigabytes of RAM.

>>6
Unicode is supposed to be used everywhere, not just on the latest and greatest hardware.

>>6
Even in 1980 this would not be a problem

no! the ship has sailed! i do not want to deal with colleagues struggling to understand that there are multiple versions of Unicode.

they understand now is that UTF8 = Unicode and everything that is not UTF8 should be made UTF8 and to assign the issue to me if that fails. i don't want to be dealing with this shit.

even right at this very moment, they're having problems. one of the "apps" (FUCK. THAT. WORD.) isn't outputting "special characters".

one world
one people
one encoding

>>9
U MENA ASCII

>>1
Shalooooooom!

>>6
My toaster shouldn't have to have that sort of hardware just to display it's settings.

>>12
An 8086 and PCB should be enough and cost, like, $5 (probably). Do you really want to pay two extra dollars for something more powerful?

>>13
x86 for embedded systems? Terrible!

>>14
8086 is commonly used for this job

>>12
its

>>16
it's
my toaster is settings, learn to think

Fucking English and it's punctuation overloading.

>>15
Just because lots of people use it doesn't make it good.

>>1
UTF8-LE is an abomination because it requires a BOM, and including a BOM renders files 7 bit unclean even if they are using only the ASCII subset of UTF-8. If the performance of your program is seriously impacted by the text serialization format, something is wrong.

>>1

There other metrics than time, which you ought to concern yourself with when designing a character encoding format: Simple and portable implementation, compatibility with commonly used formats, size of the data using the encoding, etc.

At this point in time, your suggestion breaks compatibility with commonly used formats, which is far more important than three fewer instructions on little-endian CPUs that happen to run time-optimised implementations of UTF-8. If you want to talk about sanity, introducing an incompatible character encoding format just so you can squeeze three instructions out of your implementation is fucking crazy.

>>21
Hey, this is Cudder we're talking about. No optimization is too premature.

>>20
UTF-8 and UTF-8LE are identical for the ASCII subset, so a BOM is not required at all for that.

>>21

At this point in time, your suggestion breaks compatibility with commonly used formats

You can default to standard UTF-8 if there is no UTF-8LE BOM. The only thing that breaks is autodetection for UTF-8 files starting with "ARABIC LETTER FARSI YEH MEDIAL FORM", and that's something whose occurrence has around the same likelihood as '1252 text files starting with the same series of characters as the UTF-8 BOM: "".

I've got the UTF-8 one down to 8 instructions and 22 bytes:

shl eax, 4
shr ax, 2
shr al, 2
add eax, 0e08080h
ror eax, 16
stosb
bswap eax
stosw

But the UTF-8LE one can be taken down to 7 instructions and 20 bytes:

shl eax, 6
shr ax, 2
shr al, 4
add eax, 8080e0h
stosb
shr eax, 8
stosw

Yes, I know about http://xkcd.com/927/ . But what I don't know is what the bloody hell the guys who designed UTF-8 were thinking. Put the most significant bits of the codepoint in the byte at the lowest offset!? That's completely backwards.

There is a symmetric issue when decoding UTF-8 into codepoints too: as you read each byte you have to get rid of the upper "indicator" bits which means having to do additional masking operations, whereas with UTF-8LE successive bytes contain increasingly higher-order bits, so can be read in to simply overwrite the indicators as they come (except for the last one). Compare:

UTF-8:

1110aaaa 10bbbbbb 10cccccc

Read first byte:
00000000 00000000 00000000 1110aaaa
Kill indicator bits:
00000000 00000000 00000000 0000aaaa
Shift:
00000000 00000000 0000aaaa 00000000
Read second byte:
00000000 00000000 0000aaaa 10bbbbbb
Shift to kill indicators:
00000000 00000000 0000aaaa bbbbbb00
Shift again:
00000000 000000aa aabbbbbb 00000000
Read third byte:
00000000 000000aa aabbbbbb 10cccccc
Shift to kill indicators:
00000000 000000aa aabbbbbb cccccc00
Shift to finish:
00000000 00000000 aaaabbbb bbcccccc

UTF-8LE:

1110aaaa 10bbbbbb 10cccccc

Read first byte:
00000000 00000000 00000000 1110aaaa
Rotate
aaaa0000 00000000 00000000 00001110
Read second byte:
aaaa0000 00000000 00000000 10bbbbbb
Rotate:
bbbbbbaa aa000000 00000000 00000010
Read third byte:
bbbbbbaa aa000000 00000000 10cccccc
Kill indicators (only once):
bbbbbbaa aa000000 00000000 00cccccc
Rotate to finish:
00000000 00000000 ccccccbb bbbbaaaa

7 vs 9 operations, and decoding is independent of CPU endianness since you have to read individual bytes anyway. Clear victory for UTF-8LE.

The XBTS/IBTS instructions would've been a great way to do this on x86, but unfortunately were only present on the first stepping of the 386 for some reason. Haswell now has similar instructions... although XBTS/IBTS didn't make it back into the opcode map where they should've been. :facepalm:

>>23
Shalom!

Cudder, you're like that programmer, who optimized the wait loop
http://c2.com/cgi/wiki?OptimizingTheIdleLoop

UTF8 is fast enough

What does Lord Master Gordon Moore-sama-kami-san say about character encodings, Cudder?

>>26
Use UCS-2-LE?

>>25
Idle loops need optimisation too! Putting hlt in a loop is acceptable but for lowest idle power consumption you'd ideally want to lower the clock frequencies and voltages too (and put them back up at the right time.) Especially for mobile devices this is extremely important, but even on a desktop where an idle core is a few watts but one spinning in a tight loop takes more than 10x power, the savings are significant. I believe this is automatic (or at least accomplished via SMI) on x86, but on ARM SoCs it must be done manually:

http://processors.wiki.ti.com/index.php/AM335x_Linux_Power_Management_User_Guide

https://blogs.oracle.com/bholler/entry/the_most_executed_code_in

And saying that UTF8 is "fast enough" misses the whole point: every little bit of time saved running that code is potentially more time the CPU can go into low-power mode. You're like those idiots who thought petrol was "cheap enough" - it is until it isn't, and then you have a huge problem because by then the old inefficient standard has already become so widely adopted.

>>28
Shalom!

>>28

And saying that UTF8 is "fast enough" misses the whole point: every little bit of time saved running that code is potentially more time the CPU can go into low-power mode. You're like those idiots who thought petrol was "cheap enough" - it is until it isn't, and then you have a huge problem because by then the old inefficient standard has already become so widely adopted.

Have you done any profiling?

naidne-elttiL si TIHS

>>25-san, you're like that guy, who posts unfunny TV Tropes ``articles'' whenever he has the chance.

Stop posting links to C2, aka Know Your Meme Le Brogrammer Edition.

>>32

you're like that guy, who is against enterprise quality software, design patterns and extreme programming.

Cum-in-ham 2 is the God's wiki. Prove me wrong.

>>32
I like C2

>>33,34
/r/programming has lots of epic programming memes too :)

>>35
I think you should go back there

>>34
So do I, it's like a computer-science/software-developer version of the Talmud.

A startling level of anal prowess on the side of Cudder.

>>2
I just realised that UTF-16LEL doesn't require much change to process compared to regular UTF-16LE, since it only means the surrogate pairs are in the other (i.e. saner) order, with the low surrogate at the lower address and the high surrogate at the higher address, as they should've been.

That's right, some idiots at the Unicode Consortium thought the low surrogate should be at the higher address, and vice-versa. Maybe the same ones who came up with UTF-8. :facepalm:

>>39
vectors ain't voxels, dolt

1. Lexicographic sorting
2. memcmp()

>>41
1. Sorting by Unicodepoint really makes sense just for the ASCII subset, and only with a binary collation. For everything else there's http://www.unicode.org/reports/tr10/

2. UTF-8LE has all the restrictions of regular UTF-8 like no overlong forms etc. and can be treated like UTF-8 strings for prposes of hash tables and other data structures.

stop being a namefag

Check dubs

>>43
stop being an imageredditor

>>45

defending namefags

fuck off

>>46

defending imageredditors

>>46
ebin

le implyin/g/

meme /g/ro!!!!!

Little endian is a kike right-to-left scam.

More significant (higher value) bits are on the left, because the goyim who designed bit shifting wrote in big endian.

Cudder doesn't even understand that even on his kiketel chips, you ``shift left'' to make bits more significant (increase their value). Left and right are goy, even though his scheme is kike.

Shift to finish:
00000000 00000000 aaaabbbb bbcccccc

Rotate to finish:
00000000 00000000 ccccccbb bbbbaaaa

The c, b, and a are backwards in his version, but forwards in our version, proof that it's a kike scam. The alphabet goes ABC, not CBA.

In decoding, he is using the upper part of the register as a temporary, disguised by partial register writes to the lower byte.

He also wrote his bytes in big endian to disguise the abomination of little endian. If he wasn't scamming about little-endian, it would look like this:

Rotate to finish:
bbbbaaaa ccccccbb 00000000 00000000

That is how they would be stored in memory on a little endian CPU.

>>49
easy to see

'goy': first = god/nature = greater
'kike': last = man/science = greater

(always been a big endian man myself :)

>>49
Little-endian makes more sense, because if you have a value small enough to fit in an unsigned byte, the address of that variable is the same regardless of whether its storage is specified as an unsigned byte or an unsigned dword.

>>49
"left" and "right" are arbitrary physical directions, numbers are not. Roughly 90% of the "explanations" of endianness out there are misleading or incorrect. Any explanation that uses "left", "right", "first", "last", "up", "down", or any other physical direction fits into that category. Little endian bit ordering means bit address n has value 2ⁿ. Little endian byte ordering means byte address n has value 256ⁿ. Endianness is a property of address-to-value mappings, not physical directions. It doesn't matter one bloody bit (pun intended) what it looks like to you since that's just a physical manifestation of the data.

http://www.verilab.com/files/VL_Whitepaper_Endian_July2008.pdf

IHBT.

big endian is pretty convenient because reading numbers in left-to-right languages doesn't confuse you from what the data layout actually is.

people always say oh well you can just have a disassembler (or whatever) show you the bytes in readable order, but that's just stupid and bound to cause confusion

>>54
That rarely comes up these days, because flat binary files are so rarely used, so trying to read a binary without a disassembler is just making a lot of extra work for yourself, because you have to make sense of things like distinguishing the file header from the actual instructions. It makes more sense to focus on what makes things easier when working with asm/C and upwards.

>>55

because flat binary files are so rarely used

Everyone uses XML and JSON in your world, right?

>>56
I'm talking about things like the .EXE format.

>>54,55
LE doesn't confuse you either if you have a working brain. Little endian is the Logical choice. Big endian is Backwards.

>>1
This is only because x86 is shiiiiit

>>53
Big endian: there are 8000000000 people
little endian: there are 0000000008 people

>>58
You are a faggot, and a fucking retard.

I don't like that, too much space and not enough information.

1 	U+0000 	U+007F 	0xxx xxxx 	
2 	U+0080 	U+07FF 	110x xxxx 	10xx xxxx 	
3 	U+0800 	U+FFFF 	1110 xxxx 	10xx xxxx 	10xx xxxx 	
4 	U+10000        	1111 0xxx 	10xx xxxx 	10xx xxxx 	10xx xxxx

It does look a bit clunky

The only good thing about padding every byte like that is you can check if any byte is a first byte?

So the first nibble is mostly redundant if you are checking 10xx for the byte continue

eg 110x xxxx 	10xx xxxx   10xx xxxx 	10xx xxxx

[m]0xxx xxxx 10xx xxxx 10xx xxxx 10xx xxxx[m] is the largest

11zz could be a two bit space

00xx and 01xx can use 10xx extensions

1 	U+0000 	U+003F 	00zz xxxx
2 	U+0040 	U+007F 	01zz xxxx
3 	U+0080 	U+00BF 	10zz xxxx 	
4 	U+0080 	U+07FF 	00zz xxxx 	11xx xxxx
5 	U+0080 	U+07FF 	01zz xxxx 	11xx xxxx
6 	U+0080 	U+07FF 	10zz xxxx 	11xx xxxx

costs half a bit?

plus the two per byte of extention

1-1 bit / 0.5 bit
2-5 bit / 2.5
3-8 bit / 4.5
4-11 bit / 6.5

50% less control bits

(◞‸ლ)
You're still presuming people are still using x86 sweetie.
assemble in any other *ISC and your LE observation goes out the window.

when's your CISC?

You can have

0nnnnnnn -> ascii
0nnnnnnn 1nnnnnnn -> unicode between 16#80 to ​16#407f
0nnnnnnn 1nnnnnnn 1nnnnnnn -> unicode between 16#4080 to 16#20407f

and the alternative is

0nnnnnnn -> ascii
10nnnnnn nnnnnnnn -> unicode between 16#80 to ​16#407f
110nnnnn nnnnnnnn nnnnnnnn -> unicode between 16#4080 to 16#20407f

It is impressive how much they messed up something as simple as character encoding.
That way utf8 would need up to 3 bytes rather than 4.

>>69
RISC-V, ARM, etc are all LE.

I meant this for the first one:

0nnnnnnn -> ascii
1nnnnnnn 0nnnnnnn -> unicode between 16#80 to ​16#407f
1nnnnnnn 1nnnnnnn 0nnnnnnn -> unicode between 16#4080 to 16#20407f

>>71
branch me seL4 with your changes and you got yourself a deal.

>>72
You do not need kernel support.

>>73
I need an entire distro's support.
If it can operate nicely on a daily driver I can give to my niece when she plays パネルでポン and I submit my taxes, then it's good.

Another UTF-8 stupidity: overlong encodings. Instead of actually starting the 2-byte encodings at U+0080, they started it at U+0000 but just declared the first 128 codepoints encoded in 2 bytes to be invalid, introducing overlong forms and causing tons of extra edge-case complexity in every decoder.

Others have noticed this too:

https://stackoverflow.com/questions/69318921/why-does-the-utf-8-encoding-have-the-concept-of-overlongs

https://stackoverflow.com/questions/57431095/whats-the-rationale-for-utf-8-to-store-the-code-point-directly

>>6

Why is this level of optimization this so important for Unicode? It's no longer 1980, we can actually afford multi-Ghz CPU and many gigabytes of RAM.

>It's no longer 1980,
Soon it will be.
We're in WW3

ur a dumb cunt cudder

you shouldnt use that kind of language with a lady

This thing about unichode being an inefficient mess:
it doesn't matter if you have the hardware, it still
a completely disgusting design that requires tons of
edge case handling and clever workarounds.
The obsession with adding useless emojis and sponsored symbols, the whole idea of malleable, updatable translation
layer is something you'll think if the text was controlled
by a corporation and treated as product instead of
concrete standard. Normies will not understand until
its 90% emoji and trademark symbols, but by then it would be something cringe like PDF format(with typesetting incorporated as modifier symbols) that requires a specialized library to see.

💩

>>79
Normies don't like cute little virgin girls.
They are castrated faggots and women.
They like to send men to die in wars.

(defun EMOJI-QUALITY-PARENTHESES-READER (stream char)
(declare (ignore char))
(read-delimited-list #\🫷 stream))
(set-macro-character #\🫸 'EMOJI-QUALITY-PARENTHESES-READER)
(set-syntax-from-char #\🫷 #\))
🫸setf 🫸fdefinition '🗣🫷 #'print🫷
(defparameter 🔥 "i'm the renegade dog with the twelve inch dick
prostitutes pay me for a little sniff
never paid a dollar nor a single dime
for a fuckin' or a suckin' having a good time")

🫸🗣 🔥🫷
=> "i'm the renegade dog with the twelve inch dick
prostitutes pay me for a little sniff
never paid a dollar nor a single dime
for a fuckin' or a suckin' having a good time"

>Can you stop and think out of the box for two seconds about your behavior,
What "behaviour". This is the internet, nothing takes place here.
The only "behavior" that is occuring is the pressing of square buttons.

>or are you a psychopath incapable of reasoning?
"If you don't agree with me you are a psychopath"
A very reasonable argument.

>Why are you so obsessed with torture?
You said people in europe came after you, including police.
You indicated you feared prison and consequences.
The solution is the make the consequences for your adversary far worse than what they could ever inflict upon you:
because they will.

Don't ask a question you do not want answered.
Not even by omission.

>Do you watch gore and snuff in your free time?
Nope. You sound like a woman. Are you a woman?
Your way of arguing is not like a mans.
It leans heavily on emotional manipulation and shaming,
and shys away from contemplating direct physical conflict,
or any direct action at all.
It imagines consequences from third parties, instead of the first party or the principael itself.

>From the moment we start talking,
I've been silent this whole time, are you hearing things?
>these words are in your mouth,
I haven't spoken a word to you.

>it's repulsive.
I'm not trying to fuck you, bitch.

>It's no wonder that no one wants to associate with you,
Again: we've never met, and I have never met any of these "others".
Do you have any map formats in hand that you will graciously donate to this GPL'd project?
If not; what value is it to me to "associate" with you (whatever that means)?
Why would I seek out this "association"?
Additionally: I did not email you: you emailed me: remeber?

Again another womanly characteristic: blaming me for what you "initiated"

>your behavior is fucking creepy.
No behavior has taken place.
Save for the rhytmic up and down motions of a number of depressed keys, not exceeding 128.

>Do you worship the devil?
YHWH.
The Dragon.

>Do you want to drink the blood of your victims and harvest adrenochrome too?
I want to marry cute little girl virgins.
Same as any man.

>How many corpses do you hide in your basement?
Zero. The basement is a night club.
You should know this.

>Can you stop and think out of the box for two seconds about your behavior, or are you a psychopath incapable of reasoning? Why are you so obsessed with torture? Do you watch gore and snuff in your free time? From the moment we start talking, these words are in your mouth, it's repulsive. It's no wonder that no one wants to associate with you, your behavior is fucking creepy. Do you worship the devil? Do you want to drink the blood of your victims and harvest adrenochrome too? How many corpses do you hide in your basement?

💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩

but network order is big endian as decreed by the bearded old men at IEEE. Appeal to tradition