Code by Kevin, Programming, code, business, and other pursuits
Kevin Walzer, software developer.
Subscribe to RSS Feed
Get a syndicated feed of my weblog.
Site design: Skeleton
Tk was initially ported to Cocoa in 2009 by Daniel Steffen, who had maintained Tk for several years. Apple sponsored the port of Tk to Cocoa after it deprecated its Carbon frameworks, which had previously been the foundation for Tk. The Cocoa port was a top-to-bottom rewrite, essentially cut from whole cloth rather than a simple update or port. Daniel Steffen had to create an entirely new code architecture for Tk to work with Cocoa, which has a significantly different, higher-level approach to GUI programming than Tk, based on a lower-level model derived from Unix's Xlib library (and which had strong parallels with Carbon). The resulting Cocoa-based Tk was a remarkable piece of design, and brought many advantages in speed and system integration.
After Tk-Cocoa was released in the summer of 2009, Daniel did a major round of updates and bug fixes to smooth out some rough edges in the initial release, and then left Tcl/Tk development to join Apple full-time. A bit later, after releasing some Tcl/Tk extensions that integrated with parts of Cocoa and contributing some patches to the Tk core, I took over as maintainer.
Over the next couple of years, as users gained more experience with Tk-Cocoa, some limitations of the system became apparent. Apart from the standard-issue bugs and glitches that are part of any software project, and which were relatively easy to fix, the differing way Tk and Cocoa handled user input and processing of events caused a variety of issues. Applications would occasionally freeze, and stop responding to user input, because they would become overloaded with events to process. There would be occasional issues with drawing and flickering. These issues were structural, not simply glitches that can be fixed with a few lines of code. The complexity involved would have required a major redesign of Cocoa and Tk's integration, something Daniel Steffen was precluded from doing as an Apple employee (Apple's developers are not allowed to participate in open source projects in most cases); in any case it was not even entirely clear if the integration could be designed differently.
Another issue also arose, not technical but one of policy: Tk-Cocoa, at a deep level, made use of private, undocumented Apple API's to improve drawing performance, using methods borrowed from Apple's open-source WebKit project. This practice, while always discouraged by Apple, took on greater importance when Apple introduced the Mac App Store, which stated that apps making use of private API's would be rejected. This was a serious issues for developers who, like me, wanted to distribute Tcl/Tk apps through the Mac App Store; the only workaround at the time was to link my applications to the somewhat buggy version of Tk-Cocoa that Apple shipped with OS X. (System libraries are allowed to use private API's.)
This was the situation for a couple of years: a lot of low-level frustration with the structural limitations of Tk-Cocoa, no clear path to fixing them, just incremental fixes of bugs, and a handful of Tcl/Tk apps in the Mac App Store linking to an aging version of Tk that did not include any improvements. In the summer of 2014, however, after working with Marc Culler on some patches he submitted to improve how images are rendered in Tk/Cocoa (some OS updates had broken Tk's previous mechanism), I decided to take another look at the deeper issues in Tk.
I found, to my surprise, that Tk would work without the private API calls--so I removed them altogether. I also saw, however, that Tk worked much better with them included. Their removal opened a Pandora's box of bugs and very serious drawing issues: vast amounts of flickering, ghostly images of buttons and scrollbars appearing at random whenever a window was resized or changed in any way. Except for Apple's policy about private API's, which presented a major obstacle to using Tcl/Tk in via its premier distribution channel, I would rather have left the private API's in. They made Tk/Cocoa run much better. But because of Apple's policies, they had to go--and their removal left a huge mess behind.
After doing some more work with Tk without the private API's, the issues started to sort themselves out into some predictable patterns, which made it possible to chart a plan to address them. These patterns included:
The common denominator between these bugs seemed to be that Tk-Cocoa had trouble managing a complex hierarchy of Cocoa views in a window. Tk-Cocoa's design involves using a single NSView in an NSWindow to render the layout of a Tk interface. It handles widgets drawn by Tk that are not directly derived from Cocoa widgets, such as listboxes, just fine. But NSButton and NSScroller widgets (themselves based on NSView) present added complexity that Tk seems unable to manage easily without the private API's.
The solution to these issues focused on reducing the complexity of what Tk-Cocoa had to draw:
The window-resize code I developed on my own, with a lot of trial and error and lots of time on Google. The HITheme code, however, was heavily based on work that others had done in past years--a good thing, because that was a much more complicated task.
The button code, both for regular pushbuttons and popup menubuttons, was based on a patch developed a decade ago by Revar Desmera, a polyglot developer who has done a number of projects in Tcl/Tk. Revar's patch, which added support for the HITheme API to Tk buttons, was never adopted, but it was still sitting in Tk's code tracker and provided the perfect foundation. I had to tweak and update some things, but that work only took a week or two. I am grateful to Revar for his initial work.
The scrolling code was the hardest piece of this puzzle. There was no existing HITheme scrolling implementation for me to tweak, so I had to start from scratch; I worked on the scrolling code for about a month with only modest progress, borrowing some code from Tk's older Carbon implementation and its Unix implementation, which is the foundation for both the Mac and Windows ports of Tk. I got the scrolling code to a point where it would draw the scrollbar, and nothing more. Finally, after rummaging around in Tk's source tree, I found an old patch from Donal Fellows, one of Tcl/Tk's core developers, which fixed a bug in the Mac's other HITheme-derived widgets (scale and progress bar) that seemed suspiciously like the scrollbar issue: the widgets had to apply some kind of multiplier to set their values correctly, and thus redraw. (The math was, and is, over my head.) Once I got that placed into the scrollbar code, things progressed quickly. The final piece of the puzzle was to get the scrollbar to respond to being dragged by the mouse; my initial work here focused on bringing a lot of the old Carbon code back into Cocoa, and updating it somehow. The final solution was beautifully simple and elegant. Based on an observation from Jim Ingham, Tk's original maintainer on the Mac, that the HITheme API would allow Tk/Mac's scrollbar code to become more like its Unix version, that's what I decided to do: throw out all the old Carbon code, import mostly Unix code, and just let the HITheme scrollbar be redrawn by the native API's, not moved and driven by them. Bingo.
I've committed all these updates to Tk's main line of development (trunk) and its still-supported 8.5 version this week, and they are ready to be tested and used. Here is the difference these changes make in Tk-Cocoa:
Let me be clear: Tk-Cocoa remains Cocoa-based. I have left most of Daniel Steffen's brilliant architecture unchanged: Tk's app structure is still based on the NSApp API, it still has the same complex event loop integration, and it still draws into an NSWindow containing an NSView. There still may be bugs owing to the complexity that cannot be changed. However, by making these changes to Tk-Cocoa's drawing code, I have removed some of the complexity that was there before, and brought Tk a bit closer to the level of abstraction it had under Carbon, and retains under both X11 (Unix) and Windows, where it is most performant. As a result, I feel this is a significant evolution of Tk-Cocoa, and an improvement, and I do not feel embarrassed in calling it Tk-Cocoa 2.0.
To all Tk/Mac developers and users out there: I hope you find these changes helpful. And to the developers named here and to those who have provided encouragement and feedback: thank you. While at times I have complained that I work alone, this is not true: I have relied on the contributions of many, both of code and feedback, and I am grateful.Tue, 16 Dec 2014
I've released version 1.1 of TkWebView, my Tcl package that allows a developer to add a native Cocoa WebView/WebKit widget to a Tcl/Tk application, using the same rendering engine that powers Safari.
TkWebView is available under a BSD-style license.Thu, 04 Dec 2014
I'm pleased to announce the initial release of ruby2app, a tool for bundling Ruby scripts into standalone Mac applications:
ruby2app is a simple command-line tool, inspired by similar Mac tools like py2app for Python, that bundles a Ruby script with the Ruby runtime into a standalone Mac application. It reads a basic configuration file to find the appropriate Ruby installation, compiles a small C program that embeds the Ruby interpreter, copies the entire Ruby installation into the application bundle, and re-links the relevant libraries with the main executable. Ruby must be compiled with the "--enable-load-relative" flag for ruby2app to work.
At present the tool is primarily geared for Ruby-Tk applications, since Tk is the best-supported open-source GUI framework with Ruby bindings on the Mac in 2014, and (not coincidentally) is the one I use. But I invite others to submit patches to add support for other UI frameworks as well; Qt support might be a good candidate. Support for RubyMotion is outside the scope of ruby2app, as RubyMotion presumably has its own toolchain.
I am still in the process of learning Ruby, and in fact wrote this tool to be certain that I could deploy a Ruby-Tk application in a convenient manner on the Mac desktop. There are no other current tools that provide basic support for deployment of an open-source Mac application written in Ruby. Ruby-Cocoa and MacRuby are effectively dead and their tools (standaloneify.rb) are obsolete, RubyMotion has its own toolchain, and crate and rubyscript2exe do not support current versions of Ruby.
I do have a decade of experience writing Mac applications in Python, Perl and Tcl using the Tk toolkit, so I adapted some ideas from bundling tools for those languages in creating ruby2app. ruby2app is written in Bash, and thus should be easy to read and modify as required, and should fit in well to any application building toolchain.
ruby2app is available under the MIT license, and feedback is welcome.Wed, 26 Nov 2014
This is a significant announcement because it means that all private API's have been removed from Tcl/Tk, as verified by Apple's own scanning tools, and therefore apps using Tcl/Tk can be submitted to the MAS using up-to-date versions of Tcl/Tk and without linking to Apple's own outdated system installation. As I noted previously, I spent several weeks scrubbing the private API calls, forbidden by Apple, from Tcl/Tk. One reason for the lag between the release of FileMorph 2.0 and its acceptance in the MAS was that there was still a bit of private cruft that I had overlooked, which has since been addressed.
Going forward, I plan to keep my apps as up-to-date as possible with Tcl/Tk, and not link to Apple's libraries any more.Wed, 12 Nov 2014
I've release FileMorph 2.0, an app I created in Tcl/Tk a few years ago to provide batch modification of file names and attributes, such the last date accessed. My intent in creating the program was to offer an easier-to-use tool than other file renaming programs, which tend to overwhelm with their options, and the command-line, which is powerful but obscure. I withdrew FileMorph from sales a year or so ago to focus on some other projects, but decided to return to it this fall as part of my effort to add Perl as a development language to my toolkit: FileMorph is an ideal app to implement in Perl because its functions play right into Perl's strengths, with is text and file management.
I'm pleased with the results overall. Perl has proven to be a pleasant language to develop in, and its bindings to the Tk toolkit (via its Tkx module) and the availability of the Mac::QuickBundle module make the development of a full-featured, GUI-based, desktop app feasible.
There were a few hiccups in the development process that led to it taking longer than expected. First, there was the usual assortment of changes in Apple's development tools and requirements for submitting to the Mac App Store, which caused some delay. Also, there were a few unexpected snags in Perl's interaction with Tcl under the hood: occasionally, when running a Tcl/Tk function from Perl, I would get obscure errors like "sh: -c: line 0: unexpected EOF while looking for matching `''". In Perl code, this typically means that there is a typographical error, a missing quote mark that causes execution of code to stop; but when the error occurs in the context of a Tcl procedure that Perl is calling, the actual issue is anyone's guess. I observed this when Perl (via Tcl) was downloading a file from the Internet, and parsing XML (part of the application update machinery I've implemented). The only solution was to continually tweak the code until the error no longer occurred, even though it wasn't clear to me where the error was.
I'm using this version of FileMorph as the basis for the next generation of my development platform, as well, which means, among other things, that I am now moving from linking to the Mac-installed system Tcl/Tk to shipping my own, more up-to-date versions, stripped of the private Cocoa calls that previously caused rejection by the Mac App Store. This app also includes some UI refinements that better fit in with OS X 10.10.
FileMorph is now ready for sale at my website and is undergoing review at the Mac App Store. If you do batch renaming of files, it's worth a look.Mon, 13 Oct 2014
I've written about my efforts to add Ruby to my desktop development toolkit, particularly to make it feasible to deploy standalone Ruby-Tk apps on the Mac. My efforts with Ruby deployment had paid off with a viable approach, and I was feeling confident about using Ruby as a language for a planned project.
Imagine my frustration, then, when I discovered that Ruby did not support the latest version of Tk, 8.6; numerous developers and users were reporting crashes. While this may be expected when a language like Tcl/Tk does a major new release, Tk 8.6 has been out for nearly two years, since December 2012. It would seem that two years is more than enough time to work through whatever bugs and incompatibilities might arise, but apparently the issues were rather complex.
I haven't yet migrated my Tcl/Tk apps to 8.6, mainly because Tk's use of private frameworks on the Mac prevented me from using a version other than the one Apple ships with the operating system. Now that I have removed that code from Tk, migrating my apps to Tk 8.6 is going to be part of the next major releases of my products. Tcl/Tk 8.5 is likely to reach end-of-life status soon, meaning that it will no longer be supported or updated by the Tcl/Tk developers. As a result, if I want to include Ruby in my projects, it is vital that Ruby support 8.6.
Happily, after I posted a query on the Ruby-Talk mailing list about Tk 8.6, Ruby-Tk's maintainer, Hidetoshi Nagai, committed a patch to Ruby's development branch that adds "experimental" support for Tk 8.6. I rebuilt Ruby against Tk 8.6 and tested it against a complex Ruby-Tk application, and it seemed to run just fine--no crashes. That is terrific news, because it provides a foundation for full support for Tk 8.6 in Ruby, and it means I can add Ruby--a beautiful language with a rich library--to my app toolbox. I am very pleased, and encourage others to give Ruby-Tk 8.6 a try when the next version of Ruby is released. Thanks to Hidetoshi Nagai for being an extremely responsive developer.Wed, 24 Sep 2014
I've been doing a great deal of wrestling lately with a paradox: the impact of including private Apple Application Programming Interfaces (API's) in an open-source project.
For the past few years I've been the chief maintainer of the Tk GUI toolkit on the Mac. I've been a Tcl/Tk developer for a decade, started coding my own platform-native Tcl/Tk extensions in C and Objective-C about five years ago, and took over maintainer duties in 2011 after the author of the Cocoa port of Tk, Daniel Steffen, was hired by Apple. Daniel had been the lead maintainer of Tk on OS X for several years before I assumed the role.
One of the key parts of the Cocoa port of Tk was its use of private, undocumented Cocoa API's for rendering of application windows and widgets; Daniel added this into Tk to help make sure that Tk rendered graphics quickly and smoothly. He borrowed the technique, and likely a lot of code, from WebKit, the Apple-sponsored, open-source HTML framework that powers the Safari, Chrome and other web browsers. This code provides the WebKit developers, and Tk developers, a great deal of precision and low-level control over the drawing of GUI elements, more control than is typically afforded by the Cocoa frameworks. I can't speak for how WebKit works, but Tk's overall architecture requires a lot of low-level control over the layout of a GUI. On Windows, and on the older Mac Carbon frameworks, Tk's design lines up nicely with the design philosophy of those toolkits; it's a very different situation with Cocoa.
There's a pretty significant problem with including private API's: Apple strongly discourages it, with good reason. Private API's are generally considered by Apple to be internal to the operating system, and are subject to change and even removal. It's an inherently unstable situation, and Apple goes beyond simply discouraging the use of such code; any application calling into private API's or frameworks is rejected from the Mac App Store (and also the iOS store for mobile apps). This means that Apple will reject apps that bundle code from WebKit; the same is true for apps that bundle code from Tk-Cocoa. The only workaround is to link to the (often older) frameworks that are shipped with OS X; private API's are OK there because they are installed by Apple.
I've been increasingly uncomfortable with Tk's use of private API's, but it wasn't until this year that I felt knowledgable enough to try to remove them. Eventually, that's what I did. I was tired of not being able to ship up-to-date Tk code in my own applications in the Mac App Store, and more importantly, I felt that it was a bad idea to have an open-source library such as Tk resting on such a fragile foundation.
And now, having stripped out the private API calls, I can see why they were included.
I must be candid: removing the private API's did result in a degradation of Tk's drawing performance under Cocoa. Rendering that was snappy and accurate became glitchy, slow, and laggy, with weird artifacts like buttons scrolling outside their container window and scrollbars appearing in two different places when a window was remapped. Sometimes widgets would not remap after a window resized, and in a few instances too much resizing would cause a crash.
A lot of the work I've done over the past couple of months, in consultation with a couple of contributors, has been to try and mitigate the worst effects of removing the private API calls. The button and scrollbar issues have been fixed, and some improvement in rendering with window resizing and resizing of child windows is now in place. For instance, I've added some code to skip over some drawing operations in window resizing, which has smoothed things out a bit. Basic user interfaces, such as the ones in the Tk demo, render very well with little discernible loss of performance. Stability seems fine.
Unfortunately, there remains some laggy drawing in more complex interfaces. I've spent a lot of hours over the past several weeks becoming acquainted with Tk-Cocoa's drawing code, and I regret that I can't wring out much more improvement here. Those private API's are undocumented and I don't fully understand how they worked or what they actually did, but they did add a lot of low-level magic to drawing Tk widgets. If getting the best performance for graphic rendering is the goal, including those bits is absolutely the right call, and I fully understand why Daniel Steffen included them. If an Apple-sponsored open-source project (WebKit) includes this code, then it's entirely reasonable for another Apple-sponsored open-source project (Tk-Cocoa) to follow suit.
From a policy standpoint, however, such design is untenable, because Apple is so strictly enforcing against the deployment of code that makes use of such design.
It's hard for me to get past the absurdity of Apple's position here. It's simply baffling why one of the largest open-source projects they sponsor--WebKit--violates platform protocols by using private API's, and apps directly bundling such code can't be deployed on the platform's major distribution channel, the Mac App Store. Wouldn't it be better for Apple to open up these private API's, make them public, and allow third-party developers to use them if necessary? WebKit's use of these API's dates back to the earliest days of the project; I found commit messages from 2002 that report their inclusion. Tk would certainly benefit if use of those API's could be made legal. If a platform vendor's private API is used in a vendor-sponsored open-source project, how truly private is the API? If Apple is going to be consistent here, shouldn't WebKit remove these private API calls, and find another way to render browser windows in a smooth, crisp fashion?
Past a certain point, griping is pointless; this is where we are. And I don't want to be too critical of Apple; Apple has greatly benefited the Tcl/Tk community by sponsoring Daniel Steffen's work on the Cocoa port of Tk--I doubt it would have been written otherwise. No other major open-source GUI toolkit was the beneficiary of such largesse. Still, it's a sad situation that complying with Apple's guidelines has resulted in the measurable degradation of the GUI toolkit whose port Apple sponsored. It may be a necessary compromise, but it's hard to be happy about it.Mon, 15 Sep 2014
I've released version 1.2 of my tkmacicon package for Tcl/Tk on Mac OS X.
The tkmacicon package provides the data necessary to platform-native icons on OS X as Tk images. The ::tkmacicon::geticonfrompath command takes three arguments: a file path, width, and height. The ::tkmacicon::geticonfromtype command takes three arguments: a file extension, width, and height. These commands provide the raw image data necessary to render a Mac icon as a native Tk image.
This release represents a significant API change from earlier versions, which wrote the image data to a PNG file and then read the file into a Tk image. Obtaining the data without the overhead of file writes and reads makes the package significantly faster.
For more information, see http://opensource.codebykevin.com/native.html#tkmacicon.
I've released version 1.0 of my progressdock package for Tk on Mac OS X.
The progressdock package draws a nice progress bar over a Tk/Mac application's Dock icon. It is especially useful for showing download/uploads, and similar activities. The package is based on the UKDockProgressIndicator class by Uli Kusterer.
For more information on the progressdock package, see http://opensource.codebykevin.com/native.html#progressdock.Sat, 06 Sep 2014
Over the past few years I've experimented with both Ruby and Perl as development languages for desktop apps on the Mac. Both are robust programming languages, have superb library support, and have excellent bindings for my GUI toolkit of choice. However, neither language is used much on the Mac for development of desktop apps. Their achilles' heel is deployment; in contrast to Python or Tcl/Tk, neither language has a standard, widely-used mechanism for bundling up code into a single standalone Mac application package that can be run without issue.
I've spent a lot of time over the past year or so investigating the deployment issue, periodically pulling out a couple of simple Ruby or Perl projects that I play with as part of the process of learning these languages as desktop app tools. And I'm happy to report that I've made good progress and have prototype apps running as standalone apps on the Mac.
In keeping with Perl's "there's more than one way to do it" spirit, I've actually found several ways to deploy an app. The one I've been working with the most is the Perl Archiving Toolkit and its pp module, which bundles up Perl code into a standalone executable than can then be run on another system. This works quite well, actually, and one could do the extra work of deploying the executable inside a Mac application bundle structure (a Mac app bundle, though it looks like a single file on the Mac, is actually a special directory structure). However, this mode has limitations, particularly with linking to the Tcl/Tk frameworks. The pp module has no knowledge of Mac application structure and no easy way to use standard Mac mechanisms for finding a Tcl/Tk installation bundled with the application. This proved to be a serious, and surprisingly difficult-to-solve, issue.
As a result, I decided to look elsewhere, and eventually found a much better solution in Perl's library ecosystem: Mac::QuckBundle. The QuickBundle library incorporates yet another library, mac-perl-wrapper, and does two things: it creates a proper Mac application structure for the deployment of Perl code, and automates the process so that an app can be built using a single configuration file. With this structure in place, it is much for me easier to do some additional configuration so that Perl can find the correct Tcl/Tk libraries in the application package.
Mac::QuickBundle is a near-perfect tool for my needs here, quite analogous to py2app or cx_Freeze in the Python world. I'm rather surprised I hadn't heard of it before, but I think this is reflective of the fact that Perl does not have a large constituency in the Mac desktop development community. Still, I'm glad the code was out there.
Ruby has proven to be a much more complex language to work with. Despite all my research, I've never found a general-purpose tool for deploying Ruby apps on the Mac. What tools exist are either Windows-only or limited to deep Ruby/Cocoa integration, which doesn't meet my needs. Still, after more work with Ruby and studying how the Perl and Python apps work, I've put together the basic steps to deploy a Ruby/Tk app on the Mac:
Right now my work with Ruby has been distilled into a shell script that creates my builds; I'm not sure if it can be generalized into a general-purpose application builder for Ruby. Ruby still has too many quirks to be as easily deployable as Python or Tcl/Tk, for instance. But I will likely post the script and some documentation at some point in the future, when it's been further tested and I have a releasable app to use with it.
I'm pleased that I've been able to make progress with deployment of Perl and Ruby apps on the Mac; this work will allow me to access the considerable power of Perl and Ruby for new desktop applications.