mastodon - Link to source

daniel:// stenberg://

3 months ago

daniel:// stenberg://
3 months ago

We have a CI job to spot unwanted utf8 letters in #curl PRs as we have noticed that GitHub will gladly show the for example (identical) Cyrillic version of a letter next to the Latin version in a diff and it is yes, entirely impossible for a human to spot the diff. I mean the diff is shown, but the significance of it is not.

Changing just a single letter like that in a URL hostname opens up for a world of grief.

GitHub diff output showing what looks like identical strings as being different. The right one has the Latin 'g' replaced with an Armenian 'g'.

#curl

This entry was edited (3 months ago)

in reply to daniel:// stenberg://

mastodon - Link to source

FOSS Unleashed

in reply to daniel:// stenberg:// 3 months ago

I feel like there needs to be tools that make safer handling of Unicode easier. Anyone know of the full list of Unicode ranges? I know there are some sites that give partial ones. But I'd like the information needed to detect "this sentence contains Unicode characters consistent with language X" vs "this sentence contains Unicode characters for 45 different languages"

in reply to FOSS Unleashed

mastodon - Link to source

daniel:// stenberg://

in reply to FOSS Unleashed 3 months ago

@fossunleashed agreed!

@FOSS Unleashed

in reply to daniel:// stenberg://

mastodon - Link to source

Wolf480pl

in reply to daniel:// stenberg:// 3 months ago

was this malicious?

in reply to Wolf480pl

mastodon - Link to source

daniel:// stenberg://

in reply to Wolf480pl 3 months ago

@wolf480pl no, this commit is done by me on purpose in order to test the CI and allow me to do this screenshot!

@Wolf480pl

in reply to daniel:// stenberg://

mastodon - Link to source

Will Orr

in reply to daniel:// stenberg:// 3 months ago

Out of curiosity, why is it a cronjob instead of just on the pull_request event?

in reply to Will Orr

mastodon - Link to source

daniel:// stenberg://

in reply to Will Orr 3 months ago

@worr sorry, just me not bringing my brain. It is a CI job. Edited now.

@Will Orr

in reply to daniel:// stenberg://

mastodon - Link to source

Marcel Menzel

in reply to daniel:// stenberg:// 3 months ago

That's very interesting, as Forgejo displays a big warning on top if something like this is being detected, a button to escape them and is marking the line with a warning:

"This file contains Unicode characters that can be confused with other characters."

Really confused that this is missing in GitHub.

in reply to Marcel Menzel

mastodon - Link to source

daniel:// stenberg://

in reply to Marcel Menzel 3 months ago

@wrmsr indeed!

@Marcel Menzel

in reply to daniel:// stenberg://

mastodon - Link to source

VencDvorak

in reply to daniel:// stenberg:// 3 months ago

🤣🤣🤣

in reply to daniel:// stenberg://

mastodon - Link to source

nemo

in reply to daniel:// stenberg:// 3 months ago

GitHub recently added warning for Hidden Unicode characters.

Maybe they will get to homograph attacks next.

github.blog/changelog/2025-05-…

GitHub now provides a warning about hidden Unicode text - The GitHub Blog

A warning is now displayed when a file’s contents include hidden Unicode text on github.com. Such text can be interpreted differently than it appears in a user interface. For example,…

^{The GitHub Blog}

This entry was edited (3 months ago)

in reply to daniel:// stenberg://

mastodon - Link to source

Efi (nap pet) 🦊💤

in reply to daniel:// stenberg:// 3 months ago

isn't this literally an attack vector if it was a malicious pr?

in reply to Efi (nap pet) 🦊💤

mastodon - Link to source

daniel:// stenberg://

in reply to Efi (nap pet) 🦊💤 3 months ago

@efi yes indeed

@Efi (nap pet) 🦊💤

in reply to daniel:// stenberg://

mastodon - Link to source

Efi (nap pet) 🦊💤

in reply to daniel:// stenberg:// 3 months ago

wait, did I misunderstand and it is your job to find these issues?

in reply to Efi (nap pet) 🦊💤

mastodon - Link to source

daniel:// stenberg://

in reply to Efi (nap pet) 🦊💤 3 months ago

@efi it is my job to not let malicious content into repositories I maintain, yes

@Efi (nap pet) 🦊💤

in reply to daniel:// stenberg://

mastodon - Link to source

Efi (nap pet) 🦊💤

in reply to daniel:// stenberg:// 3 months ago

best of luck getting this solved 🐾🍀

in reply to daniel:// stenberg://

mastodon - Link to source

Steve Holden

in reply to daniel:// stenberg:// 3 months ago

Python went through a few traumas with Unicode normalisation - peps.python.org/pep-0672/.

PEP 672 – Unicode-related Security Considerations for Python | peps.python.org

This document explains possible ways to misuse Unicode to write Python programs that appear to do something else than they actually do.

^{Python Enhancement Proposals (PEPs)}

in reply to daniel:// stenberg://

mastodon - Link to source

dusoft

in reply to daniel:// stenberg:// 3 months ago

They could still make it better - showing non-ASCII (UTF8) characters in URLs with different background for easy identification.

in reply to daniel:// stenberg://

mastodon - Link to source

Brokar

in reply to daniel:// stenberg:// 3 months ago

That means that somebody actually sat down and browsed all the fonts to find the 2 characters which look exactly alike?

Just imagine these people would spend their time doing something productive for a change...

in reply to Brokar

mastodon - Link to source

daniel:// stenberg://

in reply to Brokar 3 months ago

@Brokar there are actually lots of tools that do exactly that. Here's one: util.unicode.org/UnicodeJsps/c…

Unicode Utilities: Confusables

^{util.unicode.org}

@Brokar

in reply to daniel:// stenberg://

mastodon - Link to source

Mustaque Ahmed

in reply to daniel:// stenberg:// 3 months ago

would you mind sharing cron code? GitHub link or something!

in reply to Mustaque Ahmed

mastodon - Link to source

daniel:// stenberg://

in reply to Mustaque Ahmed 3 months ago

@amustaque97 the check was merged into into another script for generic checks, here: github.com/curl/curl/blob/mast…

curl/.github/scripts/spacecheck.pl at master · curl/curl

A command line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP...

^GitHub

@Mustaque Ahmed

in reply to daniel:// stenberg://

mastodon - Link to source

Mr. Lance E Sloan (IRL) 👤

in reply to daniel:// stenberg:// 3 months ago

You're right, I can't spot the character by eye. Which character is it?

in reply to Mr. Lance E Sloan (IRL) 👤

mastodon - Link to source

daniel:// stenberg://

in reply to Mr. Lance E Sloan (IRL) 👤 3 months ago

@sloanlance the alt text says it: it is the 'g' in github that is changed into an Armenian g

@Mr. Lance E Sloan (IRL) 👤

in reply to daniel:// stenberg://

mastodon - Link to source

Elric

in reply to daniel:// stenberg:// 3 months ago

Is that CI job public? Sounds like a lot of projects could benefit from that.

in reply to daniel:// stenberg://

mastodon - Link to source

Gerard Braad

in reply to daniel:// stenberg:// 3 months ago

Tried something like: gist.github.com/gbraad/551eabc… for golang projects.

Seems our vendor folder with lots of k8s libs uses non-ASCII characters.

README.md

GitHub Gist: instantly share code, notes, and snippets.

^Gist

in reply to daniel:// stenberg://

mastodon - Link to source

kaiserkiwi

in reply to daniel:// stenberg:// 3 months ago

Can you share this job? I'm really interested in how it's built.

in reply to kaiserkiwi

mastodon - Link to source

daniel:// stenberg://

in reply to kaiserkiwi 3 months ago

@kaiserkiwi the job is not cleanly only doing this but is done as part of a bunch of other scanning duties by this script: github.com/curl/curl/blob/mast…

curl/.github/scripts/spacecheck.pl at master · curl/curl

A command line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP...

^GitHub

@kaiserkiwi

in reply to daniel:// stenberg://

mastodon - Link to source

vsz

in reply to daniel:// stenberg:// 3 months ago

forgejo / gitea showing it like this:

A screenshot of gitea showing the diff between two URLs with the second one replacing the letter 'g' with the Armenian Small Letter Co, looking like latin 'g'. gitea highlights the change with a yellow warning triangle and a tooltop saying "This line has ambiguous unicode characters".

in reply to vsz

mastodon - Link to source

daniel:// stenberg://

in reply to vsz 3 months ago

@vsz neat!

@vsz

in reply to daniel:// stenberg://

mastodon - Link to source

Grimmauld

in reply to daniel:// stenberg:// 3 months ago

Hmm, now i am curious... might it make sense to pair this with a treesitter to specifically not scan comments?

in reply to Grimmauld

mastodon - Link to source

daniel:// stenberg://

in reply to Grimmauld 3 months ago

@grimmauld possibly, yes.

@Grimmauld

in reply to daniel:// stenberg://

mastodon - Link to source

Mx Autumn

in reply to daniel:// stenberg:// 3 months ago

a few years ago when Confusable Homoglyphs where last a popular talking point I ported a Python package to PHP so I could do similar filtering github.com/photogabble/php-con…

GitHub - photogabble/php-confusable-homoglyphs: A PHP port of vhf/confusable_homoglyphs

A PHP port of vhf/confusable_homoglyphs. Contribute to photogabble/php-confusable-homoglyphs development by creating an account on GitHub.

^GitHub

in reply to daniel:// stenberg://

mastodon - Link to source

LinuxUserGD

in reply to daniel:// stenberg:// 3 months ago

Gitea/Forgejo displays invalid Unicode characters by default
github.com/jirutka/setup-alpin…

Update README.adoc by LinuxUserGD · Pull Request #13 · jirutka/setup-alpine

Fixes invalid unicode characters reported by Gitea/Forgejo (see go-gitea/gitea#23682)

^GitHub

in reply to daniel:// stenberg://

mastodon - Link to source

WildRikku @ Gamescom

in reply to daniel:// stenberg:// 3 months ago

Had your post bookmarked and now pulled it out again since your blog post was featured in a popular German tech magazine. Thanks for the good work.

⇧