Find non-commented Python lines in Komodo
I’ve been doing a lot of large scale refactoring recently.
This entails a lot of “find all instances of this and replace it with that” – in non-trivial ways (of course – any monkey can do a search & replace).
Obviously I also want to only bother with non-commented lines of code.
I use Komodo for my Python coding, and while it’s a great IDE in a lot of ways, it would appear I’m the first coder that’s ever wanted to search only active lines of code (/sarcasm). Komodo does have a great regex search feature though, so I put that to use.
After much head scratching (every regex engine has its own delightful little quirks) I found this incantation:
^\s*[^#\n]*find.*$
Which will find all single-line non-commented instances of ‘find’.
Now, bugger typing that mess in every time I want to find something, so here’s a better way.
Go View | Toolbox (so the toolbox appears on the right hand side). Then right-click & “Add New Macro”. Give it a sensible name and enter this into the main text area:
if (komodo.view) {komodo.view.setFocus();}
var search = ko.interpolate.interpolateStrings(‘%ask:SearchFor:’);
Find_FindAllInMacro(window, 0, ‘^\\s*[^#\\n]*’ + search + ‘.*$’, 2, 0, false, false);
It has to be Javascript – Komodo doesn’t offer the %ask functionality in Python macro scripting (nice one, guys).
Next give it a decent key-binding on the second tab. Click in the “New Key Sequence” box and hit a vulcan key combo that works for you – I’ve used Ctrl-Alt-F – followed by clicking Add.
Hit OK & you’re ready to roll. Anytime you want to find non-commented lines of code, hit your key combo, type your search string and voila!
Insert/Edit link button not working on WordPress (Firefox, Adblock Plus)
Just figured out a bug/conflict that’d been niggling at me for a while.
A few months back, the Insert/Edit link button (on the visual editor page of WordPress) stopped working under Firefox.
I’d just done a whole bunch of site upgrades, so I figured it was just a temporary wordpress bug.
However, several updates later, and it’s still there – and on a couple of different sites, so I thought I’d investigate a bit further.
After much digging around, I figured out what it was.
Adblock Plus blocks the javascript because it creates a popup window called “tinyMCEPopup” – and of course, the anything called “popup” is evil.
To fix this, ensure that Adblock Plus is set to show in the status bar. (Tools | Adblock Plus Preferences – then when the window opens, under Options, check that “show in status bar” is ticked)
Then you’ll get a little red ABP stop sign in your status bar, you can then right click on that, and select “Disable on [yourdomain].com.” When you do that, it’ll go grey, showing that your domain has been whitelisted, & Adblock Plus will stop screwing with it.
Once the ABP icon is grey, you’ll need to refresh the page, but after that, everything will be hunky dory.
New Twitter Minimaliser
Twitter recently forced everybody over onto what they’ve dubbed “New Twitter.”
It’s got more functionality than the old version – which translates to “a lot more visual clutter.”
I’d been avoiding it for the most part, simply because I like clean, simple, straightforward. When I’m using Twitter on the web, I want to read tweets and send tweets. Nothing else.
Now I have no choice (if I’m using web-based Twitter), I thought I’d do something about it.
Thus, I present to you! The New Twitter Minimaliser.
This is a GreaseMonkey script, which means it works if you have the GreaseMonkey Add-on (follow that to get it) for Firefox, or if you run Chrome (where a lot of GreaseMonkey scripts run natively).
The New Twitter Minimaliser does the following:
Removes:
- Recommended Users
- Trends
- User Recommendations
- The “Witty Definition”
- Ability to do new style RTs (one click & all done)
Adds:
- Old Style RT button (where you quote the user & add your comment)
It also shrinks the dashboard on the side, and makes the main text area much larger. Ie, focusing the screen real estate on where it’s most useful.
It doesn’t screw with any of the code on the page (just the css) so it can’t add any new bugs. It’s also carefully optimised so it works very well on 1024×768 screens.
Oddly, now I’ve been running this script for a while, I actually prefer New Twitter to the old version. It’s much cleaner & snappier. Functionality wise it’s a bit of a wash – some things are easier, some things are harder.
Now, if I could just figure out how to get New Twitter to show me incoming DMs only (like old Twitter did, rather than one mushed up list), I’d be a super happy camper.
02.19.11
Gmail: Storage quota exceeded. Why this happens
I helped a friend out yesterday with her Gmail account. It was at 23% (or something) but email to her was bouncing with a “storage quota exceeded” message.
I googled the hell out of this but couldn’t find the reason, so I thought I’d put it up here for the world.
There are three possible reasons this happens (“storage quota exceeded” bounces when you don’t appear to have exceeded your quota):
- You’re forwarding email through/to another Gmail account that has exceeded its quota
- One of your filters is redirecting your mail through a full Gmail account
- You’re using Google Apps to manage all email for your domain, and your Google Apps domain email account is full.
Taking these in turn.
Number 1, check in your Gmail account under “Mail settings” | “Forwarding and POP/IMAP” if you have any forwarding going on. If you do (and it’s to another Gmail account), check that account hasn’t used up its quota.
Number 2, check in your Gmail account under “Mail settings” | “Filters” if any of your filters send mail via another Gmail account. If so, check if it’s used up its quota.
Number 3 (this is what happened to my friend), go to Google apps (that link will currently take you mostly there, but if you’re using the free version, click on the “Google apps (Free)” link at the bottom left, under “Solutions”. Alternatively, just google the damn site (ie, Google Google Apps. Yes, this is somewhat ridiculous). Google tends to shuffle things around pretty regularly.
ANYWAY.
- Sign in to Google Apps – remember, this will be your Google Apps login/password, not your Gmail login/password.
- Scroll down to “Email”
- Click on the link underneath the Icon, not the Email icon itself (this goes somewhere else)
This will take you to the inbox for your domain. All email to any accounts on your domain gets copied here (I guess so system administrators can see what everybody’s been up to). What this means is – even if you’re keeping your normal Gmail account nice & low, this domain-wide account will be getting filled up.
THIS is where the “storage quota exceeded” bounce messages are coming from.
Confusingly, there’s nothing in the email headers that will tell you this. You just have to magically know (Thanks Google!) No matter, I’m telling you, so now you know anyway.
From this domain-wide Gmail you can delete all the Spam. Do searches for things like “has:attachment”, then click “select all”, and “select all conversations that match this search”, then click Delete. Or, if you’re feeling wild & crazy, just select everything & delete it. Up to you.
Finally, remember to click on “Bin” (it’s what Trash is called in the Google Apps version of Gmail) and “Delete forever”. This’ll free up all the space, and stop your incoming emails bouncing.
You’re quite safe deleting anything you like from this account. It’s completely separate from your normal Gmail account.
Voila. You’re welcome. Be excellent to each other!
Firefox Redirecting Seemingly Randomly
This is another of the “this is so obscure I couldn’t find anything on Google about it, so I’m writing the canonical post” issues.
I just had an bug that drove me batty. I was trying to access a webserver on localhost, my laptop. Firefox was redirecting to my production (ie not the laptop) server, but other browsers weren’t. Browsing to ‘localhost’ or 127.0.0.1 worked perfectly, it was just using the laptop name that caused the problem.
Straight away this rules out: the local webserver (eg .htaccess rules), dns or hosts files screw ups. I also switched OFF the local server & same issue occurred.
Of course, it took quite some time to nut my way through the above diagnostics.
Turns out it was the Firefox awesome bar being not so awesome.
Months ago, I’d redirected the laptop name (not ‘localhost’ but the actual name) to the production server via the local hosts file, to assist with a debugging issue. Of course, once I finished debugging I changed the hosts file back again.
I only recently installed a webserver on the laptop, & it seems that when I’d browsed to the laptop sufficient number of times, Firefox obviously juggled its internal preferences & then started using an older, cached version of the name->ip resolution. This added to the confusion, of course. It worked perfectly, then with no relevant changes at all suddenly started redirecting.
The solution? View history, & delete all references to the local server. Voila, incorrect ip resolution gets removed & redirection stops.
Crazy. Hopefully this saves you some time.
11.8.09WordPress 2.8.5 dashboard not working in Firefox
I’ve been playing with an install of WordPress 2.8.5.. & along with a plethora of other bugs & issues, there was one in particular that stood out.
I couldn’t use the dashboard under Firefox.
It took a little searching, but I found the issue. Here’s how to solve it:
In your [blog root]/wp-admin/load-script.php file, around line 618, there is a line thus (this is all on one line, it’s just wrapping when displayed here):
echo “<script type=’text/javascript’ src=’” . esc_attr($src) . “‘></script>\n”;
change it to:
echo “<script type=’text/javascript’ src=’” . $src . “‘></script>\n”;
and around line 687 there is a line:
echo “<link rel=’stylesheet’ href=’” . esc_attr($href) . “‘ type=’text/css’ media=’all’ />\n”;
change it to:
echo “<link rel=’stylesheet’ href=’” . $href . “‘ type=’text/css’ media=’all’ />\n”;
So what’s going on here?
The lines before those two create urls for loading scripts & stylesheets, respectively.
Those urls are then getting run through esc_attr, which turns characters like & into strings like &
Often this is useful, but not here. What it means is that the html that is getting expressed in the page (you can see this if you pull up the dashboard & do a view source) looks like this:
<script type=’text/javascript’ src=’http://[site]/wp-admin/load-scripts.php?c=1&load=jquery,utils,quicktags&ver=b64ae9a301a545332f1fcd4c6c5351b4′></script>
and
<link rel=’stylesheet’ href=’http://[site]/wp-admin/load-styles.php?c=1&dir=ltr&load=dashboard,plugin-install,global,wp-admin &ver=6403d4cb3e6353f406fd43f1b0373ec2′ type=’text/css’ media=’all’ />
Which basically meant that the files weren’t getting loaded at all, the dashboard was looking like complete crap, and, well, not working at all.
The slight changes above simply remove that encoding, resulting in the correct urls:
<script type=’text/javascript’ src=’http://[site]/wp-admin/load-scripts.php?c=1&load=jquery,utils,quicktags&ver=b64ae9a301a545332f1fcd4c6c5351b4′></script>
and
11.8.09<link rel=’stylesheet’ href=’http://[site]/wp-admin/load-styles.php?c=1&dir=ltr&load=dashboard,plugin-install,global,wp-admin &ver=6403d4cb3e6353f406fd43f1b0373ec2′ type=’text/css’ media=’all’ />
Firefox 3.5.5 screwy characters appearing
There’s something that’s bugged me ever since I upgraded to Firefox 3. Certain pages that used to work perfectly in Firefox 2 suddenly didn’t.
Instead there would be a mess on the page – lots of square boxes the size of characters with text inside them. Like this 
or maybe this 
Typically this would be some kind of character encoding issue ( the server/browser specifying/requesting UTF-8 instead of ISO-8859-1 etc), or having Auto-Detect universal set off in Firefox – and most sites around the net propose this as a solution (oh, & also recommend partial reinstalls of your O/S).
Uhh, no.
It’s actually a compression issue.
If you’re having this problem, the resolution is this:
Enter into the address bar
about:config
in the Filter textbox below, type
network.http.accept-encoding
You can also just start typing “accept-encoding” until it appears on the screen.
Double click the network.http.accept-encoding entry.
Now, on my browser, it was set to
gzip,deflate;q=0.9,compress;q=0.7
but should have been
gzip,deflate
So, type that into the box & hit OK, then restart your browser (just make sure you close all your windows too)
Voila, you can now surf the web without having to constantly switch back to IE.
10.23.09Twitter OAuth Invalid Signature on friendships/create
This is a public service announcement.
I’ve been doing a bunch of work with Twitter recently & came across this problem.
When trying to do a friendships/create, I get back “OAuth Invalid Signature.”
I’m using Tweetsharp v0.15 preview (an excellent product, btw), but I don’t think this is a Tweetsharp issue, it’s a Twitter issue. People are really scratching their heads about it.
The Tweetsharp guys proposed a solution here, but that didn’t help me. In fact, the more I googled, the more erroneous solutions I found.
Here’s my setup. TwitCleaner (the app) has a consumer keys & secret. It would then get an access token/secret for the user, & use that token/secret to make the user follow @TheTwitCleaner. This is done so we can DM the user when their report is done. We encourage people to unfollow again (if they want to) once they get their report DM.
Anyway, pretty simple. We have valid OAuth token/secret from the user, so that’s not a problem.
We’re just trying to make the user follow @TheTwitCleaner, should be simple, right? No.
I wasted several hours on this. Among the solutions proposed (& wrong) were:
- You can’t use a consumer key/secret to follow the user those keys are associated with (ie, TwitCleaner the app has key/secret, but it’s associated with @TheTwitCleaner the Twitter account)
- The OAuth information is incorrect
- The request had to be made over https, not http (not something I have control over with TweetSharp, as far as I can tell)
- That because I was passing in Client information when making the request, that was gumming things up.
Well guess what? It was none of those.
Know what fixed it?
Passing in the username to follow in lower case.
I kid you not.
Now, @TheTwitCleaner is in Twitter with that combination of upper/lower case, so I was passing it exactly as stored. But no, apparently befriend (Twitter API friendships/create) needs lower case in order to work reliably.
So now you know. Hope that saves you some pain.
03.30.09The Importance Of Pipes
There’s a very subtle, often overlooked thing in Unix, the pipeline, or | character (often just called a pipe).
This is perhaps the most important thing in the entire operating system, with the possible exception of the “everything is a file” concept.
In case you’re unfamiliar (or didn’t feel like reading the wiki page above), here’s the basic concept:
A pipe allows you to link the output of one program to the input of another.
eg foo | bar – this takes the output from foo, and feeds whatever-it-is into bar – rather than, say, having to point bar at a specific file to make it do anything useful.
Why are pipes so awesome?
Well, the following reasons:
- Each program only has to do one thing, & do it well
- As such, development of those programs can be split up – even to the point where a thousand people can independently write a thousand programs, & they’ll all still be useful
- Each of those programs is very simple, thus faster to develop, easier to debug, etc
- Extremely complex behaviour can be created by linking different programs together in different ways
- None of that higher level behaviour has to be pre-thought or designed for
So, Unix has ended up with a ton of small but powerful programs. For example:
- ls – lists a directory
- cat – displays stuff
- sort – sorts stuff
- grep – finds things in stuff
- tr – translates stuff (eg, upper to lower case)
- wc – counts words or lines
- less – pauses stuff, allowing forwards & backwards scrolling
I’ve been deliberately vague with the descriptions. Why? Because ‘stuff’ can mean a file – if we specify it, or, it can mean whatever we pass in by putting a pipe in front of it.
So here’s an example. The file we’ll use is /usr/share/dict/words – ie, the dictionary.
cat /usr/share/dict/words
displays the dictionary
cat /usr/share/dict/words | grep eft
displays dict, but only shows words with ‘eft’ in them
cat /usr/share/dict/words | grep eft | sort
displays ‘eft’ words, sorted
cat /usr/share/dict/words | grep eft | sort | less
displays sorted ‘eft’ words, but paused so we can see what the hell we’ve got before it scrolls madly off the screen
cat /usr/share/dict/words | grep eft | grep -ve ‘^[A-Z]‘ | sort | less
displays paused sorted ‘eft’ words, but removes any that start with capital letters (ie, all the Proper Nouns)
cat /usr/share/dict/words | grep eft | grep -ve ‘^[A-Z]‘ | wc -l
gives us the count of how many non proper-noun ‘eft’ words there are in the dictionary (in the huge british english dictionary? 149, since I know you’re curious)
So there’s an additional benefit which is probably obvious. Debugging a complex set of interactions with pipes is incredibly straightforward. You can simply build up what you think you need, experimenting a little at each stage, & viewing the output. When it looks like what you want, you just remove the output-to-screen, and voila!
For the end-users, this means that the operating cost of using the system in a complex manner is drastically reduced.
What would happen without pipes? You’d end up with monolithic programs for every imaginable combination of user need. Ie, an unmitigated disaster. You can see elements of this in, umm, ‘certain other’ operating systems. *cough*
Most importantly of all, there is a meta benefit. A combination of all of the above benefits.
Pipes enable incredibly complex higher level behaviours to emerge without being designed in. It’s a spontaneous emergent behaviour of the system. There’s no onus on the system development programmers to be demi-gods, all they need to do is tackle one simple problem at a time – display a file, sort a file, and so on. The system as a whole benefits exponentially from every small piece of added functionality, as pipes then enable them to be used in every possible permutation.
It’s as if an anthill full of differently talented ants was suddenly building space ships.
Perhaps a better bits-vs-atoms metaphor is of money. Specifically the exchange of goods (atoms) for money, allows the conversion of those atoms into other atoms, via money. In the same way, pipes allows different programs to seamlessly interact via streamed data, in infinitely variable ways.
You don’t need to know how to make a car, since you can do what you’re good at, get paid, & exchange that money for a car. Or a boat. Or a computer. Society as a whole is vastly better off as each person can specialize & everybody benefits. Think how basic our world would be if we only had things that everybody knew how to build or do. Same thing with computers & pipes.
What seems like an almost ridiculously simple concept, pipes, has allowed an unimaginably sophisticated system to emerge from simple, relatively easily built pieces.
It’s not quite the holy grail of systems design, but it’s bloody close.
12.16.08A Nifty Non-Replacing Selection Algorithm
Algorithms are awesome fun, so I was super pleased when my little bro asked me to help him with a toy problem he had.
The description is this: It’s a secret santa chooser. A group of people, where each person has to be matched up with one other person, but not themselves.
He’s setup an array that has an id for each person.
His initial shot was something like this (pseudo, obviously):
foreach $array as $key => $subarr {
do {
// $count is set to count($array)
$var = rand(0, $count)
} while $var != $key and $var isn't already assigned
$array[$key][$assign] = $var
}
Initially he was mostly concerned that rand would get called a lot of times (it’s inefficient in the language he’s using).
However, there’s a ton of neat (non-obvious) problems with this algorithm:
- By the time we’re trying to match the last person, we’ll be calling rand (on average) N-1 times
- As a result, it’s inefficient as hell ( O(3N+1)/2)? )
- There is a small chance that on the last call we’ll actually lock – since we won’t have a non-dupe to match with
- Not obvious above, but he also considered recreating the array on every iteration of the loop *wince*
Add to this some interesting aspects of the language – immutable arrays (ie, there’s no inbuilt linked lists, so you can’t del from the middle of an array/list) & it becomes an interesting problem.
The key trick was to have two arrays:
One, 2-dimensional array (first dim holding keys, second the matches)
and one 1-dimensional array (which will only hold keys, in order).
Let’s call the first one “$list” and the second “$valid”.
The trick is this – $valid holds a list of all remaining valid keys, in the first N positions of the array, where initially N = $valid length. Both $list & $valid are initially loaded with all keys, in order.
So, to pick a valid key, we just select $valid[rand(N)] and make sure it’s not equal to the key we’re assigning to.
Then, we do two things:
- Swap the item at position rand(N) (which we just selected) with the Nth item in the $valid array, &
- Decrement N ($key_to_process).
This has the neat effect of ensuring that the item we just selected is always at position N+1. So, next time we rand(N), since N is now one smaller, we can be sure it’s impossible to re-select the just selected item.
Put another way, by the time we finish, $valid will still hold all the keys, just in reverse order that we selected them.
It also means we don’t have to do any array creation. There’s still a 1/N chance that we’ll self-select of course, but there’s no simple way of avoiding that.
Note that below we don’t do the swap (since really, why bother with two extra lines of code?) we simply ensure that position rand(N) (ie, $key_no) now holds the key we didn’t select – ie, the one that is just off the top of the selectable area.
Oh, and in this rand implementation rand(0, N) includes both 0 AND N (most only go 0->N-1 inclusive).
$valid = array_keys($list);
$key_to_process = count($valid) - 1;
do {
$key_no = rand(0, $key_to_process);
if ($key_to_process != $valid[$key_no]) {
$list[$key_to_process][2] = $valid[$key_no];
$valid[$key_no] = $valid[$key_to_process];
$key_to_process--;
}
# deal with the horrid edge case where the last
# $list key is equal to the last available
# $valid key
if ($key_to_process == 0 and $valid[0] == 0) {
$key_no = rand(1, count($list) - 1);
$list[0][2] = $key_no;
$list[$key_no][2] = 0;
$key_to_process--;
}
} while ($key_to_process >= 0);
Without the edge-case code, this results in a super fast, nice slick little 10 or so line algorithm (depending on how/if you count {}’s 
Elegant, I dig it.