Syntax highlighting added to Wiki

This morning I added syntax highlighting support to the Wiki. The syntax highlighter is called GeSHi. es I am aware the documentation link on their page returns 404; I can't do anything about that, so use Google to look for documentation hosted on other sites (or download it yourself and look in the docs/ directory).

The MediaWiki bits are documented here.

You use the Wiki tag <source lang="..."> to enable syntax highlighting, then use </source> when finished. If you forget the lang parameter or make a mess, you'll visually (at Preview/Save time) get an error. There are other parameters that you can specify too; please see the above MediaWiki link. You can also use <syntaxhighlight> as an alternate tag if you want, but I find <source> to be more convenient.

The highlighter supports many languages (see attachment for the official list that's on the server), but most relevant is the fact that it supports 3 different types of 6502 models:

• 6502acme -- MOS 6502 (6510) ACME Cross Assembler format
• 6502kickass -- MOS 6502 (6510) Kick Assembler format
• 6502tasm -- MOS 6502 (6510) TASM/64TASS 1.46 Assembler format

6502acme and 6502tasm tend to look fairly similar, while 6502kickass has nice colouring for labels but was doing things like breaking badly (colouring-wise) on apostrophes.

You do not need to <pre> or space-indent your code to use this (I imagine many will find that useful); you can still do all that if you want of course.

I went through most of the Wiki pages (searching for opcodes like bpl, bmi, and rts) and edited them to use <source lang="6502tasm">. I'm sure I missed some, obviously, but my wrists/fingers were getting sore. Here's an example:

http://wiki.nesdev.com/w/index.php/VRC7 ... rite_Delay

I'll state up front that the colour scheme for 6502tasm is not exactly what I'd like (ex. the extreme light-grey for comments is hard for me to read,). There are ways to modify the colours used in the CSS for the highlighter for those languages (within MediaWiki itself), so if we all come up with recommended improvements (please learn how to use things like Tools/Web Developer/Style Editor in Firefox, and/or Firebug for Firefox!) and want to make our own "theme" for the source code, that's great. I'm always open to suggestions. Upload screenshots of what looks good to you, with the ideal CSS changes, and any of us who are Wiki admins can put them into place.

Changing the syntax highlighter language itself (ex. "make address labels named PPUADDR bold and black") is something I would rather stay away from -- the reason is that if we upgrade MediaWiki (and especially the extension itself), it's extremely likely we'd lost those changes (meaning we have to start keeping a patch repository and so on -- we used to do this, and it's difficult to manage). However, if someone did make a new language called (for example) nesdev that was specific to all our needs (and was NES-specific), I'd be willing to make an exception and deal with keeping track of that one-off.

Hope this helps or at least gets people editing. :-)

Footnote: warning to those poking about: avoid looking at "Recent changes". I spent almost 3 hours (I think?) messing about with unrelated things (mainly MediaWiki templates) which failed miserably, so you'll see all sorts of template-related things in there, followed by a bunch of template deletes. I did get a couple templates working like {{sub}} and {{sup}} though (for sub- and superscript -- see MediaWiki's Template page on those for how to use them), so my efforts there weren't a total failure. Templates in MediaWiki are such a pain in the ass. I think I spent the past 4 hours just on the wiki. Gruhhh...

Font should be bigger.

Font is the same size as <pre>; nothing there has changed.

Here's a test sandbox.

Using the Inspector in Firefox to look at the relevant "computed" CSS attributes, the only thing I see different that pertains to the "size" of the font (presumably height) is the line-height (16.5167px for pre, 15.2333px for 6502tasm et al).

Please provide screenshots of what you're seeing otherwise.

Google Chrome Version 29.0.1547.57 m (on Windows 7):

More info:

Chrome's Developer Tools shows the "Computed style" font-size as 13px for the normal pre and 10px for the one with the syntax highlighting. Not really sure where the smaller font size comes from, though.

Cool!

Can you add the format used by NESASM/Unofficial MagicKit? (Unofficial MagicKit is a more powerful macro assembler so syntax highlighting might be more difficult.)

Please see (fully read) my initial post.

In cases like this it's almost always something to do with CSS inheritance, or it has to do with the differing line-height (though AFAIK that's not supposed to affect font size, just the "height of the line"; I believe overlap occurs if they're too small). They're the same size (as <pre>) here, using Firefox 23.

I'm more familiar with Firebug than Firefox's native Developer Tools, so I'll spend some time poking around in that today to see if I can figure out the inheritance. If push comes to shove, I can always exclusively set the font sizes in the CSS natively (hence how we'd get a "different colour theme" as mentioned initially).

Okay, I can reproduce the problem in Chrome (but not Firefox or IE8). I'll poke about to see if I can figure out what the deal is. Fucking "browser differences" in 2013, always makes me blow a gasket.

Those light gray comments are hard to read due to being so light. The green text is also a hair difficult to read, also because of reduced contrast (oddly an online contrast checker flagged the red but not the green).

Okay, here we go:


Using Chrome, I can see that the initial container <div> (uses classes mw-geshi, mw-code, and mw-content-ltr) still results in a font-size of 13px. However, the subsequent <div> (uses classes _6502acme and source-6502acme) results in a font-size of 10px -- however I don't think this is really the case (meaning I think the developer tool may have a bug or is behaving very oddly, I'll explain in a moment). Adding a font-size: 13px; to either of those 6502acme-related classes fixes the problem, but I'd much rather know how/why the font is getting changed. Those two classes don't do anything with the font size:


So I moved on to the <pre> with the de1 class:


I toggled the font attribute in Chrome and voilà -- looks correct now. The font-size there is being set explicitly to 1em (the 1.2em part is line-height). So where does this CSS come from? It's hard-coded into the main geshi.php code of all places:


Not per-language either, just flat out "let's use this for everything". It's fairly obvious to me whoever decided this has/had no idea why explicitly setting the font size there is a Bad Idea(tm) because of how it conflicts with the already-chosen defaults per stock MediaWiki CSS templates. They should have just said font-family: monospace and that's all.

Seems some other people have complained about this problem too -- and the 2nd reply in that thread is wrong in so many ways too (especially the "doesn't enforce any CSS rules on its own" -- uh huh, and what do you think the above does? Sigh, Internet people...). My point is that most/all of the font-related style crap in $code_style shouldn't be there at all, because all it does is stomp over the already-established MediaWiki styles.

I also can't tell who/what introduced this idiocy because it looks like they changed from SVN to git and probably repos, so it's been there since at least April 2012 from what I can tell.

Blabbering aside, is it fixable? I think so, but I'm not entirely sure. I will be tinkering momentarily.

Yeah, mentioned that initially, but I need mock-ups (pref. screenshots) of what people would like. All the syntax highlighting colours/styles are adjustable with ease, but the community needs to hash out what they'd prefer and I'm happy to put it all into place. If there are existing syntax highlighters folks out there use, provide a PNG or GIF (not JPEG!) screenshot of some code and I can pull out the RGB values + make appropriate changes to the CSS.

Okay, for whatever reason, this does not appear to be fixable through proper CSS means. The problem has to do with the inheritance order used by GeSHi. In fact, even setting font-family: monospace in Chrome at all causes it to start behaving oddly with font sizes and line heights -- I am not going to get into a battle with that. I started finding other places where this was being done (per-element for example), and I was just like "you gotta be kidding me".

Simple version: GeSHi -- what a pile.

Because I've already changed a bunch of our Wiki pages (hours worth) to use a syntax highlighter, I will begin looking at others. There are many, including some which are JavaScript-based (which would take load off the server). I imagine I can come up with the necessary 6502 highlighting code bits once I find one that behaves decently.

I will likely have to go through all the pages again and fix their syntaxes and so on. I shouldn't have gone through and edited all our Wiki pages to use this thing until I had tested it fully in a sandbox first and asked for community feedback first. But I can tell say up front I really did not expect Chrome to bring out oddities like this (I am not blaming Chrome; this is just pure developer idiocy on the part of GeSHi, or at least the MediaWiki version). So in turn this is my own doing/my own fault and thus I should be the one to correct it all.

@thefox --

I remembered/found this post of yours, mentioning Alex Gorbatchev's SyntaxHighlighter.

There's a MediaWiki SyntaxHighlighter extension which is exactly that. However, that dudes' "implementation" is kind of half-ass, meaning he doesn't implement a bunch of the configuration tie-ins for certain things (hard to explain tersely), so I may have to modify the underlying code.

Anyway -- possibly you could provide the "brush" (what he calls it) for your 6502 syntax highlighting and I can work it into the MediaWiki extension for the nesdev wiki? It'd be a "one-off" but in the case of one extension I can manage that. :-)

Brush is in the CRX file (which is in fact just a renamed ZIP file) linked in the thread you pasted, the filename is shBrush6502.js.

Ah, didn't know CRX files were just renamed ZIPs. Thanks! I'll poke about with this.

Got it. :-)

http://wiki.nesdev.com/w/index.php/User ... source_asm

One modification I had to make was renaming shBrush6502.js to shBrushAssembly6502.js so that the filename matched the brushes object/array declaration (SyntaxHighlighter.brushes.Assembly6502 = Brush;) in the .js file itself. I wasn't sure if Javascript language-wise would permit an object/attribute name that starts with numbers (i.e. SyntaxHighlighter.brushes.6502).

You might be saying "Ah yeah, but then I'd have to refer to it as language='Assembly6502' or the like" -- the difference in the MediaWiki extension is that there's an array of "aliases" so you can say things like "asm" => "Assembly6502" and so on.

So as of right now to use thefox's syntax stuff, use <source lang="6502"> (or lang="asm"). I can add other aliases if people want.

The one downside to the MW SyntaxHighlighter is that the theme it uses is non-existent (it doesn't define one). All you get is the stock defaults in shCoreMinit.css. I have to make a one-line change (in the PHP code itself) to make use of a theme; there's no, say, theme attribute or anything within <source> I'm sorry to say.

So we're back to the community deciding upon a good-looking colour scheme that works with the wiki theme. Here are the stock themes that are available -- we can change any of the colours/details in any of these (probably make our own):

http://alexgorbatchev.com/SyntaxHighlig ... al/themes/

One bug I've found so far has to do with use of spaces vs. literal hard tabs within a <source> block. I remember seeing this elsewhere too (meaning on things like Wordpress.com), so it's some kind of quirk/bug with SyntaxHighlighter.

A good example of what I'm talking about is here (see "Russian Peasant Algorithm"):

http://wiki.nesdev.com/w/index.php/8-bit_Multiply

Yup it also has a problem with < and > characters (and presumably other reserved characters which get converted to &lt; &gt; &amp; etc), unfortunately I forgot that this problem existed.

As far as I understand it, the problem is caused by the fact that ";" is used as a comment character and SyntaxHighlighter seems to be (?) applying the syntax highlighting to a string contains the HTML entities (&lt; ...). If that's the case it's obviously a bug (it should convert entities to characters, then apply the syntax hilighting, THEN apply the necessary entities back where needed).

You can see the problem here: http://wiki.nesdev.com/w/index.php/User:Thefox

And here, where I came up with a plausible reason why one might accidentally use &amp; in 6502 code.

Yup, I've seen these problems on Wordpress (where my blog is) as well, and they've existed for years. Sadly what needs to happen is that the HTML entity version (i.e. a literal string of lda #&lt;bar) needs to have its HTML entities decoded, then the syntax highlighting applied, then have its HTML entities re-encoded -- all at the Javascript level.

But in general I think what we have now is better than what we had before, we just have to be aware of some of the brokenness.

Could we go with a different syntax highlighter? Sure; GeSHi as I said is a piece of shit given its moronic CSS decisions (I'm still amazed at all of that), which pretty much leaves GoogleCodePrettify -- all the other MW syntax highlighters I've seen are out of date or marked insecure by MW.

I've installed (and deinstalled) the Google one on our wiki a few times already. The biggest problem I've seen with it is that the MW version is outdated compared to the official version, and that someone would have to write the Javascript lexer for the 6502 part. Sure, it's just a bunch of regex, but it's something I personally don't want to put the time into.

Sounding like a broken record, but I'm happy to go with whatever people want/etc..

I thought you didn't have to decode and reencode if you were using textContent and using document.createElement("span") for each colored token rather than using innerHTML. So the first part of making a lexer is coming up with a syntax spec for ca65, asm6, and NESASM.

@tepples -- I don't know, because as I said, I don't fully speak Javascript. Furthermore, I'm having a serious problem understanding this (honest: I have read it 4 times over now, and it becomes more confusing every time):

http://benv.ca/2012/10/4/you-are-probab ... t-methods/

Here's the impression I get from the linked article:

• If you read from textContent and write to innerHTML without manual escaping, you are vulnerable. To remove tags from a String, you need to do four steps: write innerHTML, read textContent, write textContent, read innerHTML. Or write innerHTML, read textContent, createTextNode, append to element.
• If you read innerHTML and then use this as the value of an element's attribute by concatenating HTML, you are vulnerable. Instead, read innerHTML and then replace /"/ with "&#34;" and /'/ with "&#39;", or read textContent and escape /&/, /</, /"/, and /'/, or read textContent and call setAttribute.

If you're familiar with preventing Bobby Tables attacks, walk away with this: Using createTextNode or setAttribute is like using parameterized queries in SQL.

I know some JavaScript and could probably write some sort of lexer myself given the EBNF grammar for each assembler. I ran into the same lack of grammar when I tried to write a call graph analysis program in Python.

I ran into two small bugs with the syntax highlighting when converting Disch's "The Frame and NMIs" to the wiki:

1. Any label named ":" or any instructions referencing such a label will be commented out. (e.g. beq :+ becomes beq ;:+)
   
2. Any use of high/low byte commands (<, >) also causes the rest of the line to be commented out. (e.g. lda #>oam becomes lda #>;oam)

The syntax highlighter shouldn't be modifying the text, only changing the style of the text. I've found it best to just disable JavaScript for the Wiki. Text comes out black and no ;; comments etc.

I can't fix the problem in question. If someone who is good at JavaScript wants to take a stab at it, or who is familiar with the MediaWiki PHP framework (it may be there where the issue is, I do not know), be my guest.

Switching to an alternative syntax highlighter, as I said earlier in this thread, is perfectly fine -- but there are some that are just utter garbage (ex. GeSHi) that should be avoided.

None of these syntax highlighters appear to be being actively maintained.

2012-09-01: https://www.mediawiki.org/wiki/Extensio ... ighlighter (what we currently use)
2010-07-02: https://www.mediawiki.org/wiki/Extensio ... eColorizer
2008-04-22: https://www.mediawiki.org/wiki/Extension:ASHighlight

I did notice that the author to the one we use updated something a month ago in his git repo version, but it doesn't appear to be related to the issue described previously:

https://github.com/seongjaelee/SyntaxHighlighter
https://github.com/seongjaelee/SyntaxHi ... ighter.php

However he still appears to be using SyntaxHighlighter (the Javascript stuff) version 3.0.83, which is the last official version, but their git repo has lots of activity: https://github.com/alexgorbatchev/SyntaxHighlighter

TL;DR -- Our choices are limited:

1. Find something that works, however this is a lot more tedious than you think, and as I said GeSHi has to be avoided due to how awful it is,

2. Fix what's broken in SyntaxHighlighter, which as stated requires familiarity with either JavaScript or the MediaWiki PHP framework for extensions,

3. Get rid of the syntax highlighter entirely. If people want me revert all the syntax highlighting enhancements I did across a series of wiki pages, then I will do that without complaint.

The same issue (I believe) has been raised at GitHub a couple of times already (e.g. https://github.com/alexgorbatchev/synta ... issues/144 and https://github.com/alexgorbatchev/synta ... issues/252) but nothing has been done about it.

As far as I've understood the issue, SyntaxHighlighter seems to be applying the syntax highlighting to text that still has HTML entities in it (e.g. "&lt;" instead of "<"), which messes things up because ";" is used as the comment character. So I believe a fix would involve decoding the HTML entities before syntax highlighting, then applying the highlighting, and then re-encoding. So the modification could be pretty simple if we find the place where the script actually grabs the text from the page.

But for now, I think it'd be best to disable the syntax highlighter entirely until it's fixed. It's just not cool to have it breaking peoples' code.

Bump.

I will try to make the time to undo this come the weekend. Remember: I have many, many wiki documents to manually edit by hand any time this thing is enabled or disabled.

Well, I might as well try to fix the problem before that. It needs a modification in the SyntaxHighlighter scripts.

The attached script should fix the issue. Replace the <script> include of shCore.js with an include of this shCore-nesdev.js. I also updated the brush slightly.

Test case: http://wiki.nesdev.com/w/index.php/Scan ... rge_Tables

Thank you! I'll take care of this later tonight after work.

Okay, this is all extremely complex, with many things that could go wrong any moments, so I have opted not to do this yet. Let me discuss the situation.

1. ALL CHANGES THAT DIFFERENTIATE FROM STOCK SOFTWARE MUST BE DOCUMENTED/PATCHES/DIFFS MADE. I do this any time I make a change; this responsibility falls on me, but minimising the number of diffs from stock software is important. Failure to do this results in basically 100% inability for ANY kind of software upgrades to reliably be done. Rephrased: the more one-offs we have to deal with, the more painful upgrades (of MediaWiki, MW extensions, forum software, whatever) become. I think Tepples knows bits/pieces of what the situation is, and I know other bits/pieces.

At this time we already have one diff for SyntaxHighlighter, and I will explain what it contains (two things):

i) Modifies SyntaxHighlighter.php to include "6502" and "asm" SyntaxHighlighter engine references, which refer directly to the engine called "Assembly6502".

ii) Installs a new file, scripts/shBrushAssembly6502.js, which is quite small (1.9KBytes). Your .zip file contains a replacement that appears to be almost identical to this (but not entirely), but the filename is called shBrush6502.js so I would need to rename it (not a problem).

2. The default shCore.js file is 16175 bytes. The shCore-nesdev.js file in the .zip you uploaded is 72517 bytes. This is enormous for Javascript, by the way. After a review of it, it looks to me like it needs to be packed or what's otherwise known as "minified". I have no idea how to pack anything related to Javascript (edit: maybe this is helpful), but there's no way we're putting a 72KByte JS file into place. Minified should be significantly smaller.

3. The author of SyntaxHighlighter, on his site last month, stated that he's working on maintaining the thing once more. However no where in github does he provide the built versions (and I don't know if they're packed/minified).

I won't be changing anything to "include shCore-nesdev.js rather than shCore.js" because that just creates more one-offs. Instead I'll just have to name the thing shCore.js, once a packed version can be provided.

Don't worry about the comments at that Javascript packing site about gzip or webserver compression; that's stuff that has to happen within Apache/on the server side and would require WhoaMan to make changes (it helps substantially). What I used on Parodius was below (and greatly relied upon mod_mime_magic to help properly detect things too):

Meh, I agree that 70 KB is on the large side, but the JavaScript files typically are static, so the browser will download them once and cache them for a good time. More often than not, the browser will also accept them gzipped.

The reason I didn't compress the JavaScript was that the first packer that popped up on Google (http://dean.edwards.name/packer/) broke the script.


Feel free to do whatever you want with the files.

Anyway, I guess I should tell you how shCore-nesdev.js was (manually) built:
1) Started with src/shCore.js from the distribution ZIP.
2) Took the file scripts/XRegExp.js from the repo (https://github.com/alexgorbatchev/synta ... 83/scripts) and pasted it on top of the shCore.js file. This dependency was missing from the distribution ZIP, but was probably added to the packed shCore.js file in the build process.
3) Applied my modifications to the shCore source. You can find them from shCore-nesdev.js by searching for "thefox". It's only 3 lines.

There are some build scripts in the SyntaxHighlighter repo to automate the packing/etc, but I couldn't bother to install PHP to run them.

I've tried to help with this by dividing LocalSettings.php on the wiki into two parts: the standard stuff that comes with vanilla MediaWiki above the line, and site customizations (extensions, etc.) below. And I've tried to put a date on each substantial change to LocalSettings.php. Is there something else that I should have been doing?

So here's the status:

1. I've put things into place verbatim, and they do seem to work. That said:

2. I had to Shift-Reload to pick up the .js changes. I was a little surprised by this, given that the file sizes differ substantially (old vs. new) as well as file timestamps, so I'm not sure what's going on there. I think MediaWiki does offer some way to force a cache update for clients, but I don't think that'd affect this because its not the MediaWiki page content itself that changed, only the stuff referenced by <script> (the filenames haven't changed, etc.). That should all be server-side/Apache-side as far as HTTP server vs. client headers go.

3. All the packers I tried did in fact break the Javascript as you said. Some got them down to reasonable sizes (17KBytes), but they'd break with syntax errors / missing semicolon and the like. As such, I'm going to put in a request to WhoaMan to enable mod_deflate (either for all of Apache or just within the nesdev VirtualHost contexts) to rectify the situation. It's the best I can do.

The reason the file size matters so much is both an issue of wasted network bandwidth and overall speed (it now takes 6x longer to download that single .js file, plus the time it takes to parse it/run it on the client). 76KBytes of JavaScript probably isn't going to make things like tablets (ex. Kindle devices) and other embedded devices very happy. I'm crossing my fingers though. If we get complaints, I'm pulling the entire syntax highlighting crap out entirely and will manually go through all the pages and remove the tags (this takes me an hour plus to do).

I think it's utterly ridiculous how crappy all these syntax highlighters are. I keep seeing new ones pop up on the web all over the place on blogs and so on, but god only knows what's going on under the hood there.

No I think that's correct (I follow your same methodology when it comes to LocalSettings.php), but when it comes to extensions I just wanna make sure any hand-modifications to things get backed up as diffs and so on.

I put everything in /www/vhosts/nesdev.com/wiki/mediawiki_extensions/ as you've probably noticed, and fix perms (664 not 644, since group-write (apache) is the only way we can modify each other's stuff. For example your Cite.tar is 644 right now).

For the file permissions stuff, just stick umask 0002 (default is 0022) in your .bashrc on the server and that should take care of new file creation permissions being correct going forward.

If you want to try, there's a packer written in PHP in the build directory of Syntax Highlighter repo at https://github.com/alexgorbatchev/synta ... ee/3.0.83/. This was probably used to pack it up for the official release, so it may not break it.


The parsing/running could very well be even slower if the file is packed, depending on what kind of packing is done. Minifying (white space stripping) is probably the best option.

Now I got a little bit curious why the minifying/packing was breaking the script, so I may check out what's wrong at some point.

OK, I think I found the problem. A semicolon was missing on one of the lines (gotta love JavaScript's automatic semicolon insertion). The attached file should fix the problem. It's not compressed or minified, feel free to do that with whatever you want.

Bump again: it just came up in #nesdev.


I'm not familiar with the syntax highlighting codebase; koitsu, are you still around?

You can search for "thefox" in shCore.js, there are a bunch of replace() lines. If you add "&#160;" -> " " in there, that should take care of the issue. " " should preferably be a non-breaking space, as that's what the HTML entity 160 stands for.

Unfortunately this would be, like the other existing replace() lines, just another hack, because all HTML entities are not covered.

I'm still around. I said I'd get to this when I can. And I will. I just have to go through all the pages I added to make use of the syntax highlighter and remove the syntax highlighting bits by hand, followed by removal of the syntax highlighter altogether.

Eh, that was not the point. The bug can be fixed by applying the small modification I mentioned in the previous post.

If you still want to remove the syntax highlighting, though, that's fine by me.

It will be removed. I am certain whatever this matter is will become a problem again in the future, and it is difficult to maintain from an administrator's point of view given the manual modifications/file placements/etc. that have to be done.

I have edited (by hand) all the pages I previous improved to use the syntax highlighter, but because MediaWiki does not offer what I need, I will need to go through MySQL directly to find any remaining active pages that use the old source tag and fix them. There have been some added/changed since, so I need to find them to do a thorough/proper cleanup job before I can call it a "done deal".

All pages containing use of <source> and </source> tags, excluding User pages, have been modified to use <pre> and </pre> tags instead.

I did most of this by hand, and only later did I find the Replace_Text extension for MediaWiki, which helped me find 2 others to fix up.

Anyway, because of the nature of all of this, I'm locking this thread. It would be nice if the syntax highlighter maintainers would actually maintain their software through official MediaWiki channels and so on, but it really doesn't look like that's going to happen. It was a good run, folks.