I just noted that a user changed some normal tracklists and album titles back to mojibakes! I cannot believe that such a veteran user (actually a trusted user) could do this thing. These tracklists have already been modified to correct letters by users who know the languages and it seems that this user simply copied deformed info from v1 MA to here. Apparently this user does not know the specific language, or even the alphabet, so he changed some correct polish tracklists to deformed one, where a lot of non-Polish letters appear. He also changed a Chinese album to complete shit. (I changed it back already.)

I have sent a warning to this user. But I do not know how wide this phenomenon could have gone. If one veteran user could do this, then there is no guarantee that others won't.

=============================================================
EDIT: After a close examine of that user's modification history, I think it's not the user's mistake that the mojibakes came into being. It seems that the cause is some encoding incompatibility issue. This is really weird, since this site already uses UTF-8 encoding as default. It would be great if a computer expert could look into this question.
=============================================================

So I propose the following policy: Never copy info from v1 without consideration or distinction. If you do not have decoding experience, do not rely on v1 for non-English info. For veteran users who can modify the tracklist/lyrics/album title, make sure you are familiar to the language. --You don't need to know how to write/speak that language, but you need to know at least how it looks like . For languages that user letters, you need to know the alphabets. It's very easy since we have wikipedia and google. For Far Eastern languages (Chinese, Japanese, Korean) that use logograph, or languages that you have completely no idea about, you should be extremely careful. Better use "report" and let a knowledgeable user do this than doing it yourself.

I propose that past ignorance shall be forgiven. But in the future, such stupid actions should not be tolerated.

I don't know what happened with this user exactly, but it seems that the data got garbled when he was entering song lengths. I don't think he intentionally modified the album/song titles at all. The site's encoding is indeed UTF-8 by default, but if the user manually changes his browser's encoding, it will garble everything. Other than manually changing his browser settings (and that would be -after- the page was loaded, too), I don't know how else it could have happened, especially considering one of the albums he edited has no v1 version (added in late 2011).

tl;dr, I fully endorse the above post in every way.

Since we're on this topic, can someone fix this report? A trusted user (most likely the same one) changed the mojibake'd title to a transliteration instead of the native album name, while leaving the title tracks broken.

Maybe he is using some Soviet era web browser that is set to override encoding, possibly only for submitted form data. I'm not a "computer expert" either but this is just an idea I found in my butt. You should ask him about his browser.

If he went and manually overrode the page's encoding (probably because he had some sick idea that he could understand Chinese by transliterating it to another alphabet), then yeah, it is possible.

Not stritly about mojibakes, but more about html entities. Sorry if I derail this thread a litle.

I started some months ago to correct html entities in song titles, using the following search: *#*;*
I don't know what happened, but it's not working anymore: search never stops and never gives a result. Too many items found, maybe ? But it used to work, giving something like 20,000 results, in quite a short time (10s or so).

It's still working for band names though, but not for album or song titles.

May someone of higher knowledge enlighten me on this subject, please?

Well I just checked that out, and apparently, it was one specific result in the result set that bugged the datatable from displaying correctly (caused an unknown json error). You didn't see it before, probably because that result wasn't on page 1 then. I fixed the invalid entries (including one Animetal release with 42 tracks, ugh) and now the results show up correctly, but it might very well happen again.

I don't know why it causes json errors as I escape all the data before displaying it, and even one of the two json validators that I use say the json is valid and the other validator says there's an error but shows the wrong line...

Anyway, if it happens again, let me know and I'll manually fix it I guess. -_-

And thank you very much for taking care of those html entities. You've certainly got your work cut out for you. I might add links to these search results as new "todo" lists, come to think of it. You certainly could use the help.

Edit: added the three links using these search terms to the TODO lists: http://www.metal-archives.com/todo

Thank you.

Just a head's up for anybody reading: PLEASE stop doing this:

• Using '' (two single-apostrophes) instead of "
• Using ´ or ` instead of '
• Using "decorative" punctuation like “” or ‘’ instead of "" and '', respectively.

It's a little unhygienic.

Cleared the HTML entity list for band names!

I noticed there were a lot of entries that had location fields with HTML entities, too. Probably might wanna make a "to-do" list for those too.

EDIT: Okay, I wrote a script that expands all HTML entities to their respective values. Should save a lot of copy+pasting.

http://artofalhadis.com/labs/bookmarklet.php?i=5

As usual, just click the button in the edit screen and the HTML entities should be swept away.

Should also point out that this script can be used anywhere, not just on Metal Archives. =)

Bands with htmlentities in their location fields:
http://www.metal-archives.com/todo/html-entities
Copy the data from the first column into the location field to overwrite.

Bands with htmlentities in their additional notes:
http://dev.metal-archives.com/todo/html-entities
Those are easy to fix, since the textarea usually renders the entities and saves the text correctly, rarely any copy/paste required, just save to overwrite.

Albums title?

It was already posted earlier, I said I added the three links (bands/albums/songs) here.

But to summarize, html entities in:

Band locations
Album titles
Song titles
Band locations (very temporary and not necessarily accurate, hence the hosting on dev. subdomain, for example I found one that simply had a catalog # followed by a ;... still, helpful somewhat)

I received an email from the user. He works on two computers, and on of them the default encoding was "Baltic". Now I believe that it was the encoding thing that caused the glitch I mentioned.

There is also another glitch where occurs on IE9 and possibly other browsers. When you come to
http://www.metal-archives.com/bands/%E5 ... 3540256639
it first look like this. The band name is deformed, while there is no problem on the page at all.


After clicking the band name once or twice, it becomes normal:


This glitch is not found on firefox or Chrome.

Hmm looks fine from the start on my IE9...

Yeah, same. I'm not sure what the problem is.


Did he say what platform/browser he was running?

I've added a link to Alhadis's tool on the todo page. Keep in mind that you need the bookmark toolbar enabled for it to work (you can't drag the tool otherwise). I don't generally use the bookmark toolbar but I've enabled it just for that while I work on those lists. Neat tool, especially for cleaning song titles in one swoop. Thanks Alhadis.

You're welcome. Keep in mind that even if you're not using the bookmarks toolbar, you can still use it by selecting it from the bookmarks menu.

Just to remind you that html entities does not necessarily stand for correct letters. While most of them are correct, I have seen examples in which html entities actually renders mojibakes. In this case, you still have to figure out what the correct thing should be.

I compiled some basic information for the "main" languages that may be used in MA. The list should be complete enough for the purpose of removing mojibakes. There are lots of writing systems, among them Cyrillic, Latin, Greek, Hangul, abjad (Arabic, Hebrew, Thai, etc.) Hangul, Chinese characters are widely used. I did not include Armenian and Georgian scripts, which are quite different from above, because in MA there are only 20 bands from the two countries, and it seems none of them use their own script.

Abjad: I don't have any knowledge in abjad, so I can say nothing about Arabic, Hebrew, Thai, and others. I know Thai is used a lot in MA. As to the use of Arabic, Hebrew in MA, I have no idea.

Hangul (Korean), Chinese characters (Chinese, Catanese, Taiwanese, Japanese). They use characters instead of letters. In Chinese, there are 1500-2000 most commonly used characters, and one needs 3500-4000 characters to read newspapers, books without difficulty. So it's impossible to list the characters.

Basic Cyrillic letters: (based on Russian alphabet)
А а Б б В в Г г Д д Е е Ё ё Ж ж З з И и Й й
К к Л л М м Н н О о П п Р р С с Т т У у Ф ф
Х х Ц ц Ч ч Ш ш Щ щ Ъ ъ Ы ы Ь ь Э э Ю ю Я я

The following lists ''exceptional'' letters (not found above) only.
Belarusian: Ў ў
Bulgarian: normal letters
Macedonian: Ѓ ѓ, Љ љ, Њ њ, Ќ ќ, Ѕ ѕ, Ј ј, Џ џ
Russian: normal letters
Serbian (Cyrillic alphabet): Ђ ђ, Ј ј, Љ љ, Њ њ, Ћ ћ, Џ џ
Ukrainian: Є є, І і, Ї ї, Ґ ґ
(Note: The letter І і in Ukrainian is NOT the Latin letter I i. They have different positions in unicode. Similarly for Ј ј, Ѕ ѕ above.)

Greek alphabet:
Αα Ββ Γγ Δδ Εε Ζζ Ηη Θθ Ιι Κκ Λλ Μμ
Νν Ξξ Οο Ππ Ρρ Σσ Ττ Υυ Φφ Χχ Ψψ Ωω
Diacritics:
Άά Έέ Ήή Ίί Όό Ύύ Ώώ ΐ ΰ Ϊϊ Ϋϋ


Latin-derived alphabets:
The following only lists ''exceptional'' letters in each language. Diagraphs (combination of normal Latin letters) are not listed separately even if they are considered letters in some language in their own right, e.g. the letter lj in Croatian. Not all Latin letters are used in certain languages. For example, there is no letter Q in Icelnadic. This will not be indicated in the following.

Albanian: ç, ë
Catalan: à, è, ò, é, í, ó, ú, ï, ü, ç
Bosnian: č, ć, Đ đ, š, ž
Croatian: Same as Bosnian
Czech: á, č, Ď ď, é, ě, ň, ó, ř, š, Ť ť, ú, ů, ý, ž
Danish: æ, ø, å
Dutch: normal Latin letters
English: normal Latin letters
Estonian: š, ž, õ, ä, ö, ü.
Faroese: á, ó, Ð ð, í, ú, ý, æ, ø
Finnish: å, ä, ö.
French: é, à, è, ù, â, ê, î, ô, û, (î, û sometimes drop the circumflex) ë, ï, ü, ÿ (ü, ÿ only used in very rare special cases), ç, Œ œ, Æ æ
German: ä, ö, ü, ß
Hungarian: á, ë, é, í, ó, ö, ő, ú, ü, ű
Icelandic: á, Ð ð, é, í, ó, ú, ý, Þ þ, æ, ö
Indonesian: normal letters only
Irish: á, é, í, ú
Italian: é, ó, à, è, ò
Latvian: ā, č, ē, Ģ ģ, Ī ī, Ķ ķ, Ļ ļ, Ņ ņ, š, ū, ž
Lithuanian: plus Ą ą, č, ę, ė, Į į, š, Ų ų, Ū, ū, ž.
Maltese: ċ, ġ, Ħ ħ, ż
Malay: normal letters only
Norwegian: æ, ø, å.
Polish alphabet: ą, ć, Ę ę, Ł ł, ń, ó, ś, ź, ż
Portuguese: á, â, ã, à, ç, é, ê, í, ó, ô, õ, ú, ü (letter ü ceased to be used from 2009.)
Romanian: ă, â, î, Ș ș, Ț ț. (special note ii)
Serbian (Latin alphabet): ć, č, Đ đ, ž, š
Slovakian: á, č, é, í, Ĺ ĺ, Ľ ľ, ň, ó, ô, ŕ, š, Ť, ť, ú, ý, ž
Slovene (i.e. Slovenian): č, š, ž
Spanish: ñ, ü, á, é, í, ó, ú, Œ œ, Æ æ
Swedish: å, ä, ö
Turkish: ç, ğ, İ i, I, ı, ö, ş, ü (special note i)
Vietname: ă, â, Đ đ, ê, ô, ơ, ư (special note iii)
Welsh: normal letters.

Special notes:
(i) Turkish contains two versions of letter i, one dotted, one dotless. The capital letter for dotted i is İ, eg. İstanbul.
(ii) The letters ș, ț (s-comma, t-comma) in Romanian are NOT ş ţ, (s-cedilla, t-cedilla). The later letters are still widely (incorrectly) used because s-comma, t-comma were not part of early unicode, and lack an appropriate font support.
(iii) Vietnam is a tonal language. There are 6 tones. The first is unmarked, and the rest are marked with different diacritics: Second tone by grave accent, e.g. Huyền, third tone by hook, e.g. Hỏi, fourth tone by tilde, e.g. Ngã, fifth tone by acute accent, e.g. Sắc, sixth tone by dot below, e.g. Nặng.
(iv) The upper case of Đ, đ (d with stroke) in Bosnian, Vietnamese is identical with the upper case of Ð, ð (eth) in Icelandic, Faroese. They are not the same letter and take different positions in Unicode.
(v) Serbian is the only European language with active digraphia, using both Cyrillic and Latin alphabets.

Wow that's quite the thorough list. Good work. Out of curiosity... are you a professional linguist? You seem to really know your shit.

I'll note that in French, the characters Œ œ, Æ æ are not necessarily used per se; it's perfectly acceptable to write "oe" and "ae" instead. Luckily, the search engine understands both, so they can be used interchangeably.

No. I am a grad student studying math in... well you know where from my IP address.

You should also note that Serbian (and maybe other nearby languages?) can be spelled in latin script too.

Also, in your (iv) note, I think you meant to say that D with stroke (which is also used in latin-script Serbian) and Eth are NOT identical.

He meant that the uppercase version of that letter is identical-looking in both scripts even if they are not the same character in unicode.

More html entities
Some html entities slipped through the queries posted on the "to do" page: for example, when é is used instead of é. Alhadis script is also not working on them
Those can be found using *&*;* in the search field.

I correct all those in song titles, but a few are also to be found in band name (1) and album titles (21).

Oh wow. Okay, added that list. But wow, the cleanup taskforce has been pretty efficient. We're almost rid of all the damn htmlentities!

Thanks for pointing it out. I have modified my post. Actually, Serbian is the only European language with active digraphia, using both Cyrillic and Latin alphabets, according to wikipedia.

I noticed that there are still many html entities in the alternative spelling (for band names) and possibly in Additional Notes. I don't know how to get a complete list for these.

I've adjusted the lists once more:
Alt spelling with html entities: http://www.metal-archives.com/todo/html-entities
Additional notes with html entities: http://dev.metal-archives.com/todo/html-entities (note, it's a search for '*&*;*', so sometimes they're not actual html entities but simply a & followed by a ; later in the text, such as in the Dio notes, you can leave those alone)

WAIT FOR A MOMENT. Please do not use Alhadis' tool to clean up html entities in BAND'S ALTERNATIVE NAME. This list is valuable since it tells us lots of bands whose name MIGHT be changed to original languages. Of course NOT all such names should be changed. Furthermore, from my first look, some html entities in band's alternative name are completely wrong! If we clean them up already, it would be difficult to find them, especially the wrong html entities.

Sofeshue, fixing the HTML entities isn't going to introduce any problems with fixing mojibakes. Any HTML entites that existed in band ANS fields in V1 were probably already wrongly encoded when they were submitted as HTML entities.

I know. What I mean is, since there are htmls in bands ANS field, these bands may need a close look, rather than simply converting htmls into letters. Once the list is gone, how do we find these bands easily?

http://www.metal-archives.com/todo/alt-spelling

And sort by the "Alternate spellings" column. A lot ANS fields will be bunched together alphabetically.

Either that, or you can always fix the ANS on the spot.

It would appear that all HTML entities have been removed.

So only false positives remain in this list, then? I'll remove it.

Oh, no. I did not even see that list. We should clear that one too. I meant the one with the AS field.

Everything left in that list seems to be legitimate html entities (ampersands, quotes, angle brackets for html tags).

Ok, thanks.