Code by Kevin

QuickWho 3.0: When to rewrite an app

I've just released version 3.0 of QuickWho, my application for obtaining registration information for Internet domains. This new version is the most thorough rewrite of an application that I've ever done, so I wanted to discuss a bit about the new version and its changes.

Earlier versions of QuickWho were basically GUI wrappers, or front-ends, for the command line "whois" tool that ships with Mac OS X. A user would type in a domain name in the search field, QuickWho would send the query to whois, and then display the text output in the main window. I tried, with the user interface, to add some features that were not directly present in the command-line tool, such as the ability to select a specific whois server to query; the ability to call QuickWho via AppleScript; and the ability to save and print the textual output for later review.

However, despite these additional features, at its foundation QuickWho remained a simple wrapper for the whois program that offered little functionality not found in the command-line tool or in other GUI applications (such as Apple's own Network Utility, which ships for free with OS X). It simply didn't offer much value for the money I was charging for it, even after I lowered the price to $12.95. Reviews of the program, and its sales, reflected this lack of value.

As a result, I decided to make the new version of QuickWho substantially different from the older ones. It would offer more information than a simple dump of whois output. It would be easier to use and better integrated into OS X so it could be incorporated into different workflows. After looking at a number of possible features and technologies to support these features, here is the core mix of the new QuickWho:

Distills basic whois data to an at-a-glance list, with both domain information and geographic information about the domain. This is done through a mix of approaches: the Python pywhois library, for basic information about the domain; the "dig" command-line program, to get the main IP address for the domain; and the pwhois domain information server, which retrieves geographic information about the domain (city/state/country, latitude, longitute) based on the IP address. The raw whois data can also be displayed; this can be configured in the preferences. The result is that more precise, accurate whois data is delivered.
Displays the text data as rich text with clickable URL's for domain names and e-mail addresses, to facilitate e-mailing or displaying domain information in a browser.
Displays map information from the domain, based on Yahoo! web services and the PyYahooMaps library. This can be useful for gathering more information about the domain.
More robust AppleScript support and complete integration with the OS X Services menu, so QuickWho can both return whois data to other applications and access the functionality of other applications from the Services menu.

I've come to re-evaluate my long-standing belief that you should never rewrite an application from scratch, which Joel Spolsky eloquently articulated. If a program is at a dead end and cannot do what you want it to, you may have no choice but to rewrite large portions of it, even the entire thing. With QuickWho, I rewrote major portions of the user interface (removing the tree view, developing and customizing the rich text view, adding the image display) and completely re-implemented the whois query, as well as the alogorithms to print the data (since it now included an image as well as text).

With these changes, I'm restoring the price of QuickWho to its previous level of $24.95. I believe the program now does far more than the simple command-line whois tool, and previous versions of QuickWho: it offers a richer array of information about specific domains, in an easier-to-read format.

If you need to do frequent searches of domain information via "whois," then QuickWho 3.0 is a useful tool that can increase your productivity. It's definitely worth a look.

Simplicity vs. power

A big part of my design philosophy as a Mac developer is to harness the underlying power of the Unix command-line tools that reside beneath the Mac's shiny Aqua surface. This means that I attempt to expose the functionality of these tools with an easy-to-use graphical interface. As I do this, however, I am often faced with a conflict between simplicity--making the tool intuitive to use through the GUI--and power--allowing the user to access as much of the tool's functionality as possible through the GUI.

That conflict is not an easy one to negotiate. If you don't expose enough of the program's functionality, you run the risk of "dumbing it down"--making it less useful than it should be. A GUI tool that provides only a simple wrapper to the tool's most basic functionality doesn't offer a lot of value to the user. On the other hand, if the GUI tool attempts to access every single command-line switch from the program, you run the risk of burying the user in an avalanche of configuration options and preference settings, and overwhelming all but the most advanced users--who many not need the convenience of a GUI tool at all.

Sometimes finding the right balance between simplicity and power is straightforward. In both PortAuthority and Phynchronicity, the command-line tools that power my applications offer a managable range of functions--installing, updating, deleting, searching, and getting information about Unix software--that can easily be encompassed in a GUI. In the case of both of these programs, the GUI itself handles some functions (especially searching the list of software packages) and dispatches other functions directly to the command-line tool as required. My programs offer a more pleasant user experience over the command-line tool without sacrificing any essential or advanced functionality.

In other cases it's harder to find the right balance between simplicity and power. This conflict is especially acute in my network monitoring tool, PacketStream. PacketStream is a user interface for tcpdump, an enormously powerful but complex network monitoring tool. The documentation for PacketStream outlines a stunning range of options, which, in the hands of a power user, results in the ability to monitor network traffic with incredible precision. It's hard for a GUI to capture all of that power without becoming equally complex to use. (Some GUI tools for tcpdump, such as WireShark, are unabashedly complicated to use.) The challenge for me is to identify the most common uses for tcpdump and provide access to a range of functions in PacketStream, without turning the program into something as complicated as tcpdump itself.

It's still a work in progress, and I've been guided by my own instincts and user input. The earliest version of PacketStream just provided a simple wrapper for tcpdump with no options beyond which network interface it listened to (Airport, Ethernet, etc.). Over the past couple of years, I've added the ability to listen on specific network ports for certain types of traffic (such as http, e-mail, etc.), and to display the contents of packets; these are command-line switches to tcpdump options. I've also added, at the GUI level, the ability to save and print network output, and to call PacketStream via AppleScript and the Mac Services menu. These aren't hooks into tcpdump itself, but they do make the program more integrated into the Mac's GUI environment, and allow it to cooperate with other programs. Finally, I've also added some basic documentation on tcpdump itself, to assist users in understanding how PacketStream works under the hood.

I've also removed some features, or at least moved them out of the GUI, in the interest of making the program simpler without sacrificing power. Adding Keychain support allows you to type a password once and then store the password in the system keychain: some users have complained about having to type a password everything they launch a search. I've also added functionality that automatically sets the correct network interface to listen on. In these cases there's a lot of code working to make the program more effortless to the user, with less configuration required; I've been able to increase the program's simplicity with no adverse effect on its power.

Certainly my design decisions with PacketStream aren't for everyone. One review of the program, undoubtedly by a power user of tcpdump, complained that PacketStream is too simple, little more than a worthless GUI wrapper for tcpdump's most basic functions. I don't agree with this assessment, as there is a considerable amount of work in PacketStream to make accessing tcpdump seem effortless, but there it's certainly true that PacketStream doesn't provide access to every feature of tcpdump. Nonetheless, PacketStream is one of my better-selling programs, so at least some users are finding that it provides a good mix of simplicity and power.

My long-term goal for PacketStream is to improve both its simplicity and its power, by making the interface more and more effortless to use, even as I gradually expose more of tcpdump's functionality. Making power seem effortless is the design philosophy of most good Mac software, and I will continue to move PacketStream--and all my other programs--in that direction.

Packaging vs. developing, or mainstream vs. cutting-edge

As a developer primarily using scripting languages (Tcl/Tk and Python), I have multiple options for setting up my development environment: I can use the versions of the languages bundled with OS X; I can use one of several freely available binary installers for the languages; or I can build my own.

The drawback of using the bundled versions with OS X is that they tend to be out of date; Apple only updates its scripting languages with a new release of OS X. As a result, the version of Tcl/Tk available on my machine is 8.4, which is a couple of years out of date.

Binary installers--from ActiveState for Tcl and Python, or the official Python website--are very convenient, but you trade that convenience for control over your development environment. It's difficult to do non-standard or cutting-edge stuff with the standard pre-packaged distribution. My work with Tk-Cocoa and 64-bit, for instance, is too cutting-edge for these distributions; in fact, a lot of my work with Tk-Cocoa and extensions is being incorporated into the next generation of ActiveState's Tcl/Tk installer.

As a result, I'm left to the final option of the developer--"rolling my own," or building my own stuff from source. That's actually fine in most cases. I have complete control over my environment, and I'm able to focus on what I need to do. The drawback is that if I want to incorporate something outside of my usual development environment, I have a lot of work to do because I have to build it myself.

Recently, though, I found myself turning to pre-packaged distributions for a project--and greatly appreciating the convenience they offer.

I have packaged up a Mac build of a popular Python-based card game, PySolFC. PySolFC is a complex package that builds on many components, including the Python PyGame library and the Python Image Library (PIL)--themselves both highly complex Python extension libraries that also build on many external libraries. I've tried to build both PyGame and PIL myself from source many times, in order to integrate them into my custom Python builds, but my efforts haven't worked: both PyGame and PIL have too many moving parts, and something always breaks. I tried again, this time using MacPorts to build the dependencies, but that didn't work either: some of the MacPorts packages were out of date or were incompatible with each other.

Finally, after several hours of frustration, I decided to try the binary installers from ActiveState, both Python and Tcl/Tk. I moved my own builds out of the way and installed the ActiveState packages. ActiveState also provides various extension packages for Python and Tcl, and I used this method to install PIL. Finally, I used a binary installer of PyGame provided by the PyGame developers. Then, I ran the build of PySolFC using the py2app tool. The build worked out of the box. Total time spent using the binary tools: about an hour.

In this case, I definitely appreciate the convenience that the binary packages offer. The complexities of building a library like PIL become someone else's headache (just as these are my headaches if it's my own software that I'm working on). I wanted a working build of PySolFC; the binary packages, which provide the foundation for PySolFC, make it easy to build.

This process has given me some definite insight into the differences between being a developer of my own software and a packager of someone else's. In my own software, I want to push the limits of my toolset. That's why I'm working with Tk-Cocoa now, before it is mainstream, and also why I'm working with 64-bit applications. When you are working on the leading edge, sometimes things are difficult: for instance, I have to use the older, less capable bundlebuilder tool to wrap my 64-bit Python applications, because py2app doesn't yet handle 64-bit programs gracefully. (py2app is more powerful than bundlebuilder, scanning your application and mapping out dependencies as well as building application bundles, and as a result requires more changes to integrate successfully with 64-bit builds; bundlebuilder simply copies code libraries that you specify into an application bundle, and knows nothing about different kinds of builds.) The current maintainer of py2app, Ronald Oussoren, is also the maintainer of Python on the Mac and the maintainer of the PyObjC Cocoa-Python library. Needless to say, he's stretched pretty thin, and hasn't had time to update py2app in some time.

As an application developer, I want to push the limits, but as an application packager, I just want something that works, that has been thoroughly tested and debugged. Using the ActiveState binary installers for Python and Tcl/Tk, and py2app, I was able to efficiently build PySolFC. The built program doesn't reflect the cutting edge of Tcl/Tk or Python--it's not built on Cocoa, nor is it 64-bit capable--but it works great, and just as importantly, it was easy to build. I definitely appreciate the convenience offered by the binary installers!

Finally, since I release a great deal of my own work as open-source, I'm helping to contribute to the next generation of stable binary packages. ActiveState's beta build of Tk-Cocoa now includes some of my libraries. When this beta version (actually Tk 8.6) is finally released in the next year or so, hopefully the cutting-edge advances it includes will become more mainstream.

Tk-Cocoa is the best cross-platform toolkit for the Mac

I'd like to make a bold claim: Tk-Cocoa is now, in terms of its platform-specific integration, the best cross-platform toolkit for OS X.

With the release of tclservices, Tk-Cocoa now has a range of core features and extensions that no other cross-platform GUI toolkit can match. In other words, Tk-Cocoa provides fuller access to Mac-specific technologies--the features and user experience that help define the Mac--than any other toolkit.

Here's how I support this claim:

Services: With the tclservices package, Tk-Cocoa allows Tk applications to be both providers and consumers of services via the Services menu. No other toolkit--not wxWidgets, not Qt, not Gtk, not Java--supports this. Java allows Mac applications to access the Services menu, but provides no simple mechanism for providing them.
AppleScript: With its built-in AppleScript support via the "do script" command, Tk-Cocoa allows any Tk application to provide AppleScript support that goes beyond opening documents, launching the application, and so on. No other toolkit--not wxWidgets, not Qt, not Gtk, not Java--supports this.
Mac-specific window styles: Tk-Cocoa provides, via its core features and extension packages, access to such features as sheets and drawers. Qt provides full access to Mac-specific window styles, while Java provides access to sheets. Neither wxWidgets nor Gtk provide this.
Displaying Mac-native icons: Tk-Cocoa provides, via core features and extension packages, the ability to display Mac-native icons in an aplication. Qt and Java support this; wxWidgets and Gtk do not.

Does this mean that Tk-Cocoa is superior to these other toolkits? No, certainly not. Tk still lacks some significant features, most notably platform-native printing and a modern HTML library like WebKit. But if you're looking for platform-native integration, no other cross-platform toolkit offers as much specific support for these features as Tk-Cocoa.

I hope this brief discussion may persuade other developers to take a closer look at what Tk-Cocoa provides.

Tclservices 1.0

I've released version 1.0 of tclservices, a Tcl/Tk library that provides access to the Mac's Services menu, allowing Tcl/Tk applications built against the Cocoa frameworks (Tk-Cocoa) to function as both providers and consumers of services. (Tk-Carbon is not supported.) With tclservices, Tcl/Tk applications can send data to other applications via the Services menu, and can also provide functionality that can be accessed from the Services menu. This package adds yet another aspect of platform-native integration that can improve Tk applications on the Mac.

PacketStream 3.3

I've released version 3.3 of PacketStream, my network monitoring tool for OS X. This is a fairly major update. PacketStream now supports AppleScript and Services (both as a provider of a service, and able to access the Services menu); it also supports Keychain integration, which means that it's no longer necessary to type in your password each time you launch a new network scan. The other major new feature is a preference item to display the actual contents of packet data in the PacketStream data view.

All in all, this release puts more monitoring power at your fingertips, and also greatly improves the application's ease-of-use and integration with OS X. I strongly recommend that you upgrade, and if you aren't currently a registered user, it's a great time to give PacketStream a try.

Reconsidering "good enough"

Sometimes, good enough isn't good enough.

A few months ago I suggested that perfection wasn't the ideal goal in software; instead I championed "good enough," a strategy that lets you get a product out the door that does most of what needs to be done.

"Good enough" is indeed useful for shipping products, but I've come to the conclusion that perfection isn't such a bad goal after all--if you define "perfection" as a long-term goal that you strive for over multiple product releases. Even if you don't reach perfection, you still improve.

The reason I'm re-considering the goal of "good enough" is that I'm still quite unhappy with some of the additions I've made to my programming toolkit--specifically, integration of Tcl/Tk with drag-and-drop on the Mac, and accessing the Services menu from Tcl/Tk applications. You can drag files and text to my Tk applications, and they provide a service that can be accessed from the Services menu. But that's a one-way street; currently, you can't drag items from a Tk application to other applications, and you can't access the Services menu from my Tk programs. This feels incomplete to me.

So, over the past month or so, I've been working on improvements in these areas. I've gotten a working implementation of Tk applications as a "drag source" committed to TkDND's source code archive, and I've completely rewritten my Tcl Services package to properly integrate with Cocoa's Services API, both in sending data to Tk applications and accessing the Services menu from Tk applications; in the case of the Tcl Services package, it's a rare instance where a complete re-write makes sense. In fact, after a bit more stress-testing of the Tcl Services package in my own programs, I plan to add it to my growing list of Tcl extensions that provide integration of Tcl/Tk with various aspects of the Cocoa API.

I still don't feel compelled to revisit AppleScript support and printing support in my programs; there, the current implementation is truly "good enough," in that it satisfies all my current and projected use cases, and where a re-write would represent a huge amount of work with only a marginal improvement in the functionality offered by my current packages.

Look for these new improvements in new releases of my apps in the near future.

Manpower, PacketStream, QuickWho

I've released minor updates to Manpower, PacketStream, and QuickWho. All of these updates incorporate the library bug fix that I outlined here. If you use any of these programs, this is a recommended update.

Windowlist 1.3

I've done another update of my windowlist package, which implements a Mac-standard "window" menu in Tcl/Tk applications. This version, 1.3, fixes some additional bugs in the package that cropped up when I attempted to integrate it with my macdrawer package.

Because all of my applications make use of the windowlist package, look for another round of bug fix updates shortly.

Is Tcl/Tk dying?

A hardy perennial among Tcl/Tk developers is hand-wringing over the language and toolkit's standing relative to other languages and toolkits. At one time Tcl/Tk was one of the most widely-used programming combinations around, but in recent years it seems to have lost some popularity and mindshare among programmers and companies. Tcl/Tk developer Mark Roseman has posted an amusing satire about this question. David Welton has also recently posted a more serious consideration of this question. His analysis, which focuses on both the technical and cultural reasons for this erosion in popularity, is astute.

Still, even after acknowledging the relative decline of Tcl's mindshare and community size, I still have to believe that it's little more than an inconvenience--not a fatal liability--because the core language and toolkit are still being actively developed.

Make no mistake: Tcl/Tk's small community and mindshare are definitely an inconvenience. In Perl or Python, if you're looking for a library or extension package, it's more likely that someone, somewhere has done it already, because their communities are larger. In Tcl/Tk, you often have to do it yourself. In some instances, the functionality I'm looking for is quite difficult to implement (such as SSH support) or is already fully available in another language (such as RSS parsing); in these cases, I may turn to a language that already supports them, such as Python. And even when I do undertake an extension project, such as adding drawers and sheets, I'm often completely on my own because no one else in the community has the expertise or time to offer insight or assistance. Most of my recent posts on the Tcl-Mac mailing list go unanswered because they involve rather complex questions about the integration of Tk and its underlying Cocoa implementation. In fact, it's not an exaggeration to say that the number of active developers of the Tcl-Mac community (contributing to the language's core or with widely-used extensions) can be counted on one hand.

But these problems are an inconvenience--not a showstopper. I am usually able, eventually, to work out any problems I encounter in developing Tk extensions. And, more importantly, Tcl and Tk are getting development where it counts most. While the releases and improvements don't come as frequently as some other languages, they do come--and there are real improvements. Of course, many of these improvements are undertaken by the scrappy core team of Tcl/Tk developers. Some of them, however, are the result of corporate sponsorship, which contradicts those who say that Tcl's mindshare is withering.

In fact, I am heavily dependent on the Tcl/Tk improvement sponsored by a single company: Apple. Apple has twice sponsored ports of Tk to run on top of its native GUI frameworks: first Carbon in 2001, and then Cocoa in 2009. Even though the core Tk-Mac community is small even by Tcl/Tk standards, Tk itself is so widely used on the Mac (by Python, Ruby, etc.) that Apple wanted it as a first-class cross-platform GUI library. That support by Apple, especially by sponsoring a Cocoa version of Tk, ensures the toolkit's future on the platform and is the major reason I continue to develop my Mac-only apps in Tk instead of another toolkit.

I feel very confident in Tcl/Tk's future, and am not troubled by its small community. It would be nice if it had a larger community, but as long as the language continues to evolve and grow, then I will continue to stick with it.