Code by Kevin

Code by Kevin, Programming, code, business, and other pursuits

Your Host
Kevin Walzer, software developer.


Subscribe to RSS Feed
Get a syndicated feed of my weblog.



September 2014
Sun Mon Tue Wed Thu Fri Sat

Privacy Policy

Site design: Skeleton


Wed, 24 Sep 2014

Private API's and open-source projects

I've been doing a great deal of wrestling lately with a paradox: the impact of including private Apple Application Programming Interfaces (API's) in an open-source project.

For the past few years I've been the chief maintainer of the Tk GUI toolkit on the Mac. I've been a Tcl/Tk developer for a decade, started coding my own platform-native Tcl/Tk extensions in C and Objective-C about five years ago, and took over maintainer duties in 2011 after the author of the Cocoa port of Tk, Daniel Steffen, was hired by Apple. Daniel had been the lead maintainer of Tk on OS X for several years before I assumed the role.

One of the key parts of the Cocoa port of Tk was its use of private, undocumented Cocoa API's for rendering of application windows and widgets; Daniel added this into Tk to help make sure that Tk rendered graphics quickly and smoothly. He borrowed the technique, and likely a lot of code, from WebKit, the Apple-sponsored, open-source HTML framework that powers the Safari, Chrome and other web browsers. This code provides the WebKit developers, and Tk developers, a great deal of precision and low-level control over the drawing of GUI elements, more control than is typically afforded by the Cocoa frameworks. I can't speak for how WebKit works, but Tk's overall architecture requires a lot of low-level control over the layout of a GUI. On Windows, and on the older Mac Carbon frameworks, Tk's design lines up nicely with the design philosophy of those toolkits; it's a very different situation with Cocoa.

There's a pretty significant problem with including private API's: Apple strongly discourages it, with good reason. Private API's are generally considered by Apple to be internal to the operating system, and are subject to change and even removal. It's an inherently unstable situation, and Apple goes beyond simply discouraging the use of such code; any application calling into private API's or frameworks is rejected from the Mac App Store (and also the iOS store for mobile apps). This means that Apple will reject apps that bundle code from WebKit; the same is true for apps that bundle code from Tk-Cocoa. The only workaround is to link to the (often older) frameworks that are shipped with OS X; private API's are OK there because they are installed by Apple.

I've been increasingly uncomfortable with Tk's use of private API's, but it wasn't until this year that I felt knowledgable enough to try to remove them. Eventually, that's what I did. I was tired of not being able to ship up-to-date Tk code in my own applications in the Mac App Store, and more importantly, I felt that it was a bad idea to have an open-source library such as Tk resting on such a fragile foundation.

And now, having stripped out the private API calls, I can see why they were included.

I must be candid: removing the private API's did result in a degradation of Tk's drawing performance under Cocoa. Rendering that was snappy and accurate became glitchy, slow, and laggy, with weird artifacts like buttons scrolling outside their container window and scrollbars appearing in two different places when a window was remapped. Sometimes widgets would not remap after a window resized, and in a few instances too much resizing would cause a crash.

A lot of the work I've done over the past couple of months, in consultation with a couple of contributors, has been to try and mitigate the worst effects of removing the private API calls. The button and scrollbar issues have been fixed, and some improvement in rendering with window resizing and resizing of child windows is now in place. For instance, I've added some code to skip over some drawing operations in window resizing, which has smoothed things out a bit. Basic user interfaces, such as the ones in the Tk demo, render very well with little discernible loss of performance. Stability seems fine.

Unfortunately, there remains some laggy drawing in more complex interfaces. I've spent a lot of hours over the past several weeks becoming acquainted with Tk-Cocoa's drawing code, and I regret that I can't wring out much more improvement here. Those private API's are undocumented and I don't fully understand how they worked or what they actually did, but they did add a lot of low-level magic to drawing Tk widgets. If getting the best performance for graphic rendering is the goal, including those bits is absolutely the right call, and I fully understand why Daniel Steffen included them. If an Apple-sponsored open-source project (WebKit) includes this code, then it's entirely reasonable for another Apple-sponsored open-source project (Tk-Cocoa) to follow suit.

From a policy standpoint, however, such design is untenable, because Apple is so strictly enforcing against the deployment of code that makes use of such design.

It's hard for me to get past the absurdity of Apple's position here. It's simply baffling why one of the largest open-source projects they sponsor--WebKit--violates platform protocols by using private API's, and apps directly bundling such code can't be deployed on the platform's major distribution channel, the Mac App Store. Wouldn't it be better for Apple to open up these private API's, make them public, and allow third-party developers to use them if necessary? WebKit's use of these API's dates back to the earliest days of the project; I found commit messages from 2002 that report their inclusion. Tk would certainly benefit if use of those API's could be made legal. If a platform vendor's private API is used in a vendor-sponsored open-source project, how truly private is the API? If Apple is going to be consistent here, shouldn't WebKit remove these private API calls, and find another way to render browser windows in a smooth, crisp fashion?

Past a certain point, griping is pointless; this is where we are. And I don't want to be too critical of Apple; Apple has greatly benefited the Tcl/Tk community by sponsoring Daniel Steffen's work on the Cocoa port of Tk--I doubt it would have been written otherwise. No other major open-source GUI toolkit was the beneficiary of such largesse. Still, it's a sad situation that complying with Apple's guidelines has resulted in the measurable degradation of the GUI toolkit whose port Apple sponsored. It may be a necessary compromise, but it's hard to be happy about it.

[/general] permanent link

Mon, 15 Sep 2014

Tkmacicon 1.2

I've released version 1.2 of my tkmacicon package for Tcl/Tk on Mac OS X.

The tkmacicon package provides the data necessary to platform-native icons on OS X as Tk images. The ::tkmacicon::geticonfrompath command takes three arguments: a file path, width, and height. The ::tkmacicon::geticonfromtype command takes three arguments: a file extension, width, and height. These commands provide the raw image data necessary to render a Mac icon as a native Tk image.

This release represents a significant API change from earlier versions, which wrote the image data to a PNG file and then read the file into a Tk image. Obtaining the data without the overhead of file writes and reads makes the package significantly faster.

For more information, see

[/software] permanent link


I've released version 1.0 of my progressdock package for Tk on Mac OS X.

The progressdock package draws a nice progress bar over a Tk/Mac application's Dock icon. It is especially useful for showing download/uploads, and similar activities. The package is based on the UKDockProgressIndicator class by Uli Kusterer.

For more information on the progressdock package, see

[/software] permanent link

Sat, 06 Sep 2014

Deployment of Perl and Ruby Tk apps on the Mac: Coming along

Over the past few years I've experimented with both Ruby and Perl as development languages for desktop apps on the Mac. Both are robust programming languages, have superb library support, and have excellent bindings for my GUI toolkit of choice. However, neither language is used much on the Mac for development of desktop apps. Their achilles' heel is deployment; in contrast to Python or Tcl/Tk, neither language has a standard, widely-used mechanism for bundling up code into a single standalone Mac application package that can be run without issue.

I've spent a lot of time over the past year or so investigating the deployment issue, periodically pulling out a couple of simple Ruby or Perl projects that I play with as part of the process of learning these languages as desktop app tools. And I'm happy to report that I've made good progress and have prototype apps running as standalone apps on the Mac.

In keeping with Perl's "there's more than one way to do it" spirit, I've actually found several ways to deploy an app. The one I've been working with the most is the Perl Archiving Toolkit and its pp module, which bundles up Perl code into a standalone executable than can then be run on another system. This works quite well, actually, and one could do the extra work of deploying the executable inside a Mac application bundle structure (a Mac app bundle, though it looks like a single file on the Mac, is actually a special directory structure). However, this mode has limitations, particularly with linking to the Tcl/Tk frameworks. The pp module has no knowledge of Mac application structure and no easy way to use standard Mac mechanisms for finding a Tcl/Tk installation bundled with the application. This proved to be a serious, and surprisingly difficult-to-solve, issue.

As a result, I decided to look elsewhere, and eventually found a much better solution in Perl's library ecosystem: Mac::QuckBundle. The QuickBundle library incorporates yet another library, mac-perl-wrapper, and does two things: it creates a proper Mac application structure for the deployment of Perl code, and automates the process so that an app can be built using a single configuration file. With this structure in place, it is much for me easier to do some additional configuration so that Perl can find the correct Tcl/Tk libraries in the application package.

Mac::QuickBundle is a near-perfect tool for my needs here, quite analogous to py2app or cx_Freeze in the Python world. I'm rather surprised I hadn't heard of it before, but I think this is reflective of the fact that Perl does not have a large constituency in the Mac desktop development community. Still, I'm glad the code was out there.

Ruby has proven to be a much more complex language to work with. Despite all my research, I've never found a general-purpose tool for deploying Ruby apps on the Mac. What tools exist are either Windows-only or limited to deep Ruby/Cocoa integration, which doesn't meet my needs. Still, after more work with Ruby and studying how the Perl and Python apps work, I've put together the basic steps to deploy a Ruby/Tk app on the Mac:

  1. Create a small stub launcher in C that embeds Ruby. That seems to be the secret sauce to nearly all the tools I've studied; they start as small applications written in C that then launch an embedded interpreter. This isn't surprising, actually, and simplifies the basic structure of the application package.
  2. Use MacPorts to build Ruby and build Ruby with an "--enable-load-relative" flag. There are dozens of ways to build Ruby on the Mac; it is actually not easy to do. Of all the ways I've tried, MacPorts seems to have the simplest, most complete support for building Ruby. I submitted a patch to MacPorts to allow Ruby to build under a "load-relative" flag, which means that its search path for its core libraries is not hard-coded (the default mode) but rather is encoded to be relative to the main Ruby executable. Building "load-relative" means that the Ruby installation can be relocated to another location, such as an application bundle, and will still work as long as the overall installation structure is preserved. (Otherwise the deployed Ruby will look in a fixed location, such as /opt/local, for all of its supporting code, which won't work if nothing is installed there.)
  3. Use MacPorts's search mechanisms to find the locations of the appropriate libraries and copy them into the application bundle. Ruby does not seem to have any mechanisms for searching and identifying code that an application depends on (Perl has its "scandeps" library and Python has its "modulefinder" library). However, MacPorts itself provides tools for searching out its installations, such as pkg-config, and these tools make it possible to build and link the application launcher, and find and copy over the Ruby installation, seamlessly.

Right now my work with Ruby has been distilled into a shell script that creates my builds; I'm not sure if it can be generalized into a general-purpose application builder for Ruby. Ruby still has too many quirks to be as easily deployable as Python or Tcl/Tk, for instance. But I will likely post the script and some documentation at some point in the future, when it's been further tested and I have a releasable app to use with it.

I'm pleased that I've been able to make progress with deployment of Perl and Ruby apps on the Mac; this work will allow me to access the considerable power of Perl and Ruby for new desktop applications.

[/software] permanent link

Sparkle back in business

A few months ago I replaced the popular Sparkle update mechanism for the Mac in my apps with a hand-rolled mechanism that used some of Sparkle's ideas and data format, but which was simpler to configure in my own code. This was motivated by the fact that Sparkle seemed to have fallen into obsolescence, having been unmaintained for a few years.

I'm pleased to report that development on Sparkle has resumed, with a new project page. Sparkle now has a community of developers working on it, and it seems to have received several much-needed updates. While I don't plan to go back to using Sparkle in my own programs, I'm very pleased to see that this vital tool show signs of life on the Mac.

[/software] permanent link

Wed, 11 Jun 2014

Sweeping back into town

I've brought TextSweep, my search-and-replace app for the Mac, out of storage and back into active development. A couple of months ago I had announced its discontinuation because it seemed horribly broken: freezing up whenever the user selected a file to display. I blamed insoluble bugs in its underlying GUI framework, Tk, for the problem.

As it turns out, the bug was in my own code, so embarrassingly simple to fix that I still can't believe it. Essentially, if no search term in the application were defined (i.e. if the search field was blank) then the app would lock up; if a search term was defined, it would run as normal. Fixing this is literally one line of code. Unfortunately, finding the source of the bug turned out to be fiendishly hard (I ran across the actual error essentially by accident), and kept the app from being usable for months.

Ah, well. Now that I've found the source of the error, I have released version 2.1 of TextSweep and submitted the same version to the Mac App Store for review and inclusion. I'm still not 100% satisfied with the app and want to refine its features further, but now, at least, it runs. That has to come before anything else.

[/software] permanent link

Sun, 04 May 2014

PortAuthority 6.2

I've released version 6.2 of PortAuthority, my GUI for MacPorts. This release fixes a bug in updating ports and includes a couple of minor internal code improvements.

[/software] permanent link

Wed, 16 Apr 2014

Swept away: TextSweep

I've decided to discontinue my TextSweep search-and-replace app. I've had a large number of problems getting it to run in a stable fashion on Mavericks, it hasn't sold anything this year (owing to its lack of functionality, I suspect), and it has never worked as well as I'd like even if it does launch correctly. Rather than invest time in rewriting it from scratch, I've decided to scrap it. I have released updates to several other apps this month, so I will continue to work on those.

[/software] permanent link

Sun, 13 Apr 2014

PortAuthority 6.1, PacketStream 6.1

I've released updates to both PortAuthority and PacketStream. These updates include the improved update mechanism based on Sparkle that I'm adding to each of my apps, and also a workaround for a serious system-level bug in Mavericks.

The bug in Mavericks--which affected these apps but which was not my doing--revolved around running AppleScripts with elevated user privileges, which is a requirement for some of the operations that PortAuthority and PacketStream perform (installing software, listening to network data). I had turned to AppleScript to provide a better, more native implementation of this process than older versions of my apps supported. Unfortunately, the AppleScript bug in the most recent update of Mavericks, 10.9.2, caused the AppleScript to hang; thus, my apps making use of this functionality never completed their operations and never displayed data. In short, they stopped working. Developers call this type of bug a "showstopper," and it's as serious as you can get.

Given that this was Apple's bug rather than something in my own code, my options for dealing with it were limited. I filed a bug report with Apple in hopes that a new update of OS X might resolve the issue, but no action has been taken to address the issue as of yet. I was reluctant to move away from using AppleScript because it provided a much better user experience than my older methods: native dialog, behavior more consistent with other apps, better performance. However, none of that matters if the app doesn't work.

As a result, I've reluctantly re-introduced my older approach to running privileged operations. Instead of running an AppleScript command with administrator privileges, I now presenting a password dialog to the user, then pipe that password to the command line "sudo" tool under the hood. "Sudo" does the same thing as my AppleScript method did, just without the fancy native dialog. Some developers suggest this method is a little less secure than the native method that AppleScript implements. However, even if that's true, it has the virtue of actually working. The new versions of these apps work as expected.

I took the opportunity to simplify the password code a bit, and I've retained the improved method of reading data into the apps that I implemented with the AppleScript approach; thus, the apps haven't lost anything in terms of their performance and responsiveness. (Another reason I went the AppleScript route was that my old password code was sluggish and could cause the apps to lock up at unpredictable times. I don't think that will happen here.)

In any event, I don't plan to go back to using AppleScript. It's highly unlikely that "sudo"--which is a long-stable, core Unix command--will ever run into the same kind of system-level bugs that AppleScript has.

If you've recently downloaded these apps, and wondered why they didn't work, please give the new versions a try.

[/software] permanent link

Thu, 10 Apr 2014

NameFind updates and replacement for Sparkle

I've released a couple of updates to NameFind in the last week. The first update, 7.1, fixed some UI glitches on Mavericks: icon images were diplaying in the UI in a jagged manner. The images also slowed down scrolling in the data view. I ultimately decided to get rid of icons altogether in the table view and instead display them in a separate info window, if the user calls that up.

The second update, 7.2, released today, completely overhauls the app's update mechanism. For several years I had used my own system, but last year I decided to add Sparkle support, the popular framework for app updates that is still used by hundreds of apps that are downloaded from outside the Mac App Store. My own homegrown updating system has had some hard-to-track bugs in recent versions of my apps, and I just decided to abandon it in favor of Sparkle.

Unfortunately, Sparkle itself has proven to be somewhat unstable on Mavericks; it often fails to fully update an app, crashing and requiring a manual relaunch of the updated app before the new version is visible. Just as badly, Sparkle's source code page at Github states that Sparkle is now unmaintained by its developer--which means he is no longer working on it. (I believe he is now an Apple employee, whch precludes work on outside projects, even popular ones.) Unfortunately, because Sparkle is now unmaintained, there is little chance that bugs in it will be addressed going forward.

Given that both my old and new update mechanisms are problematic, how have I decided to proceed? By implementing a hybrid of the old and new.

I like Sparkle's "appcast" approach for broadcasting updates; it requires the developer to upload an XML file to a website, similar to an RSS feed, which other sites can download. (Some Mac software listing sites, such as MacUpdate, use this mechanism to track app releases.) This is better than my old approach, which could not be parsed by other systems. So I opted to keep Sparkle's server-side mechanism, and replace its app-level code with my own. This required me to do two things: fix the bug in my old code that kept updates from installing correctly (it was a stupid, stupid error in the way I archived the app update file for download), and rewrite some of the code to present a user interface similar to Sparkle's, which I also liked.

Thus, this approach is the best of the old and new: it uses an industry-standard mechanism (appcast) to announce app updates, but better integrates with my own apps by implementing a Sparkle-like interface and workflow in my own code.

If you use the Sparkle update in the current release of NameFind, you will see the same old bugs, but going forward, you'll see the new Sparkle-like update approach, which should work a lot better.

I don't regret working with Sparkle; spending time with its internals, even if I couldn't fix them, allowed me to understand it well enough to implement something very similar, and Sparkle's server-side "appcast" mechanism remains intact in my toolbox. The new hybrid approach provides a smooth upgrade experience; I have been remiss in not paying attention as I should have to this part of the user experience of my non-Mac App Store apps. I will continue to focus on this going foward.

[/software] permanent link