Replace stb_c_lexer.h with a custom Lexer written from scratch in Crust

[rexim]

@yui-915 @deniska @mikmart @nullnominal let me know if you see any problems

Closes #64

[deniska]

Tried it, seems to work fine.

How "historically accurate" are we aiming for a language to be? Hex literals are nice to have for example, even if the original didn't have them.

[rexim]

How "historically accurate" are we aiming for a language to be? Hex literals are nice to have for example, even if the original didn't have them.

I actually don't know yet... My plan was to implement full set of described B features, but with the C lexer (since stb_c_lexer.h was a C lexer) and then go over the entire codebase and make it historically accurate as much as possible given the targets and the code we already have. Something like that.

[nullnominal]

build/b -run -o build/tests/inc_dec tests/inc_dec.b ./std/test.b
Generated build/tests/inc_dec.asm
flat assembler version 1.73.32 (16384 kilobytes memory, x64)
3 passes, 1624 bytes.
x: 3
++x: 4
x++: 4
x: 5
x--: 5
--x: 3

seems off

[nullnominal]

that might be in the previous thing but we're paying more attention to the tests for this

[rexim]

build/b -run -o build/tests/inc_dec tests/inc_dec.b ./std/test.b Generated build/tests/inc_dec.asm flat assembler version 1.73.32 (16384 kilobytes memory, x64) 3 passes, 1624 bytes. x: 3 ++x: 4 x++: 4 x: 5 x--: 5 --x: 3

seems off

Hm... Don't see any problems. That's literally how Post/Pre-increments work.

[nullnominal]

Hm... Don't see any problems. That's literally how Post/Pre-increments work.

looked off to us because x is set to 3 then incremented twice

[nullnominal]

im prolly just being dumb tho

[nullnominal]

(blocking #69)

[rexim]

Alright, I see no objections. Merging.

[mikmart]

@rexim looks great! Storing the input path and the Token enum make it a lot more ergonomic. I have some tweaks I noticed which I'll submit a PR for.