Code by Kevin

Code by Kevin, Programming, code, business, and other pursuits

Your Host
Kevin Walzer, software developer.


Subscribe to RSS Feed
Get a syndicated feed of my weblog.



Privacy Policy

Site design: Skeleton


Wed, 24 Sep 2014

Private API's and open-source projects

I've been doing a great deal of wrestling lately with a paradox: the impact of including private Apple Application Programming Interfaces (API's) in an open-source project.

For the past few years I've been the chief maintainer of the Tk GUI toolkit on the Mac. I've been a Tcl/Tk developer for a decade, started coding my own platform-native Tcl/Tk extensions in C and Objective-C about five years ago, and took over maintainer duties in 2011 after the author of the Cocoa port of Tk, Daniel Steffen, was hired by Apple. Daniel had been the lead maintainer of Tk on OS X for several years before I assumed the role.

One of the key parts of the Cocoa port of Tk was its use of private, undocumented Cocoa API's for rendering of application windows and widgets; Daniel added this into Tk to help make sure that Tk rendered graphics quickly and smoothly. He borrowed the technique, and likely a lot of code, from WebKit, the Apple-sponsored, open-source HTML framework that powers the Safari, Chrome and other web browsers. This code provides the WebKit developers, and Tk developers, a great deal of precision and low-level control over the drawing of GUI elements, more control than is typically afforded by the Cocoa frameworks. I can't speak for how WebKit works, but Tk's overall architecture requires a lot of low-level control over the layout of a GUI. On Windows, and on the older Mac Carbon frameworks, Tk's design lines up nicely with the design philosophy of those toolkits; it's a very different situation with Cocoa.

There's a pretty significant problem with including private API's: Apple strongly discourages it, with good reason. Private API's are generally considered by Apple to be internal to the operating system, and are subject to change and even removal. It's an inherently unstable situation, and Apple goes beyond simply discouraging the use of such code; any application calling into private API's or frameworks is rejected from the Mac App Store (and also the iOS store for mobile apps). This means that Apple will reject apps that bundle code from WebKit; the same is true for apps that bundle code from Tk-Cocoa. The only workaround is to link to the (often older) frameworks that are shipped with OS X; private API's are OK there because they are installed by Apple.

I've been increasingly uncomfortable with Tk's use of private API's, but it wasn't until this year that I felt knowledgable enough to try to remove them. Eventually, that's what I did. I was tired of not being able to ship up-to-date Tk code in my own applications in the Mac App Store, and more importantly, I felt that it was a bad idea to have an open-source library such as Tk resting on such a fragile foundation.

And now, having stripped out the private API calls, I can see why they were included.

I must be candid: removing the private API's did result in a degradation of Tk's drawing performance under Cocoa. Rendering that was snappy and accurate became glitchy, slow, and laggy, with weird artifacts like buttons scrolling outside their container window and scrollbars appearing in two different places when a window was remapped. Sometimes widgets would not remap after a window resized, and in a few instances too much resizing would cause a crash.

A lot of the work I've done over the past couple of months, in consultation with a couple of contributors, has been to try and mitigate the worst effects of removing the private API calls. The button and scrollbar issues have been fixed, and some improvement in rendering with window resizing and resizing of child windows is now in place. For instance, I've added some code to skip over some drawing operations in window resizing, which has smoothed things out a bit. Basic user interfaces, such as the ones in the Tk demo, render very well with little discernible loss of performance. Stability seems fine.

Unfortunately, there remains some laggy drawing in more complex interfaces. I've spent a lot of hours over the past several weeks becoming acquainted with Tk-Cocoa's drawing code, and I regret that I can't wring out much more improvement here. Those private API's are undocumented and I don't fully understand how they worked or what they actually did, but they did add a lot of low-level magic to drawing Tk widgets. If getting the best performance for graphic rendering is the goal, including those bits is absolutely the right call, and I fully understand why Daniel Steffen included them. If an Apple-sponsored open-source project (WebKit) includes this code, then it's entirely reasonable for another Apple-sponsored open-source project (Tk-Cocoa) to follow suit.

From a policy standpoint, however, such design is untenable, because Apple is so strictly enforcing against the deployment of code that makes use of such design.

It's hard for me to get past the absurdity of Apple's position here. It's simply baffling why one of the largest open-source projects they sponsor--WebKit--violates platform protocols by using private API's, and apps directly bundling such code can't be deployed on the platform's major distribution channel, the Mac App Store. Wouldn't it be better for Apple to open up these private API's, make them public, and allow third-party developers to use them if necessary? WebKit's use of these API's dates back to the earliest days of the project; I found commit messages from 2002 that report their inclusion. Tk would certainly benefit if use of those API's could be made legal. If a platform vendor's private API is used in a vendor-sponsored open-source project, how truly private is the API? If Apple is going to be consistent here, shouldn't WebKit remove these private API calls, and find another way to render browser windows in a smooth, crisp fashion?

Past a certain point, griping is pointless; this is where we are. And I don't want to be too critical of Apple; Apple has greatly benefited the Tcl/Tk community by sponsoring Daniel Steffen's work on the Cocoa port of Tk--I doubt it would have been written otherwise. No other major open-source GUI toolkit was the beneficiary of such largesse. Still, it's a sad situation that complying with Apple's guidelines has resulted in the measurable degradation of the GUI toolkit whose port Apple sponsored. It may be a necessary compromise, but it's hard to be happy about it.

[/general] permanent link