Code by Kevin, Programming, code, business, and other pursuits
Kevin Walzer, software developer.
Subscribe to RSS Feed
Get a syndicated feed of my weblog.
Site design: Skeleton
I've released version 2.6 of FileMorph, my file modification app for Mac and Windows. This version focuses mainly on Windows, with a substantially re-worked deployment structure that ensures smooth launching.
Apps written in Perl, FileMorph's main development language, are not easy to deploy on Windows. I prefer to use open-source tools rather than commercial toolkits like PerlApp from ActiveState, so I had deployed the last version using Par/PP, which creates standalone Perl application with all libraries bundled in a single executable file. Based on my research and testing, PAR/pp works fine for simple Perl/Windows applications that don't require a lot of polish.
However, for larger, more complex applications with multiple dependencies and professional-level polish--such as a custom icon, and standard application data such as version information contained in the executable--PAR/pp is seriously broken on Windows. Adding a custom icon and standard application text data is a non-trivial process, requires tools to actually hack the executable, and runs the serious risk of corrupting the executable to the point where it will not run. In my case, I got the executable to run, but at the cost of an extremely long startup time (over a minute) with a blank console popping up while everything loaded. After the application started, it still had issues: the default PAR icon still leaked out in places such as the Windows taskbar. While the app was entirely functional, it seemed rough, unpolished, and unprofessional.
This was unacceptable. I had held off on doing much marketing and promotion of FileMorph on Windows because it offered such a poor user experience out of the box. Something had to change.
After doing some additional research, I found a different approach that offered great promise: a simple stub executable that linked into Perl via its embedding API and did nothing but start the application. (The approach is outlined here: http://perl-node-interface.blogspot.com/2011/03/deploy-perl-application-on-windows.html.) I took interest in this approach because I've used something like it in my experimental ruby2app deployment tool for Ruby apps on OS X. I took the basic sample code for the Perl project, modified it to fit my application structure, and after some trial and error, got things working well. FileMorph now starts right up and, while the Windows download installer is much larger than before (60 megabytes vs. 20 megabytes), that's an acceptable trade-off for an application that runs well out of the box.
I commend this approach to any developer who is working with Perl on Windows; I am very pleased with it. While FileMorph is a commercial application, its source code is available under an open-source license at http://fossil.codebykevin.com/fossil.cgi/filemorph/timeline, so feel free to take a look. (As an aside, I also use a terrific module for deploying the application on the Mac: Mac::QuickBundle.)
And, of course, if you are an end user looking for a handy file modification tool for Windows, FileMorph may just do the trick--feel free to download and play with it. A one-time cost for a license includes free upgrades for life. It's worth your attention.Sun, 11 Oct 2015
After nearly four months since my last blog entry I'm pleased to announce the release of FileMorph 2.4.0. In terms of new features, this version is a minor update except for one thing--it now runs on Windows.
This isn't my first foray into Windows. When I tried it before, I went with an application that doesn't sell much, and I also spent longer porting it to Windows than I did trying to market and sell it--in short, I gave up pretty quickly. In this second go-round, I am planning on sticking with Windows for the long haul, because I need a second large market apart from the Mac outside the Mac App Store.
Of course, developing for Windows is not a simple process, and I had to re-learn several things about the platform; most of the last four months has been spent learning the particulars of Windows, and of re-working my development stack to support Windows. This means adding a lot of Windows details not just to a single app but to the entire suite of libraries I use to provide functionality to my applications: installation, user data, platform integration, and more. That work, at least, will be reduced with the next app I target for Windows.
I won't bore you with the technical differences between Windows and Mac--they are considerable, but not insurmountable. It is an interesting challenge to develop for a different platform, and I hope to learn more about marketing for that platform as well.Sun, 16 Aug 2015
These releases are the result of a two-month change toward looser enforcement of user registration, i.e. paying for a license to keep using the application after a thirty-day demo. I decided to make the releases nagware (they would pop up an alert if no serial number were found) but not otherwise limit their functionality, even after the 30-day demo period ended. The result was striking: without the requirement to pay, no one paid. I've had no sales over the past couple of months even with the usual number of downloads. Reviewing the rather large drop in sales revenue, I recalled this change I made to my registration code and concluded that was likely the cause. As a result, I've re-enabled the stricter registration code, and demos will once again expire after 30 days. Users who have been using the apps for the past couple of months without registering will eventually have to pay for a license if they want to keep using the apps. (You only pay once, by the way! Upgrades are provided free for life.)
Registration code is one of the duller parts of an application--a lot of housekeeping code, and surprisingly complicated, even with a fairly basic registration scheme such as the one I use. In the past my code has been prone to unexpected bugs that caused the code to freeze my application. There are a lot of scenarios a developer has to test for--the existence of a license or not, measuring how long a demo has run, and so on. It's not fun. To be honest, I eliminated the demo period because I wanted to remove the headaches of keeping the code tested and consistent. However, that code provides an essential function--ensuring that customers who continue to use my application actually pay for the work I've done. In fact, it's the most important function my code provides, from a financial perspective, especially now that I'm moving my applications out of the Mac App Store.
It's a bit disappointing that I have to use a stricter registration scheme to ensure that my work provides a bit of revenue, but I guess it's human nature to avoid paying for something if you don't have to. My code doesn't take elaborate measures to enforce registration, but it's strict enough to keep honest users honest, and that allows my apps to provide a bit of financial support for my business and family.Wed, 22 Jul 2015
The major feature powering new releases of my commercial apps today is a new open-source Tcl project I'm introducing: aem 1.0, which provides a lightweight mechanism for Tcl applications to respond to arbitrary Apple Events sent via the AppleScript programming language, Apple's system scripting language.
aem (short for "Apple Event Manager") is not the first Tcl package to provide an interface to the Apple Event mechanism. Jon Guyer's TclAE package provides a comprehensive interface to the AEM framework; it is powerful and stable. However, it is essentially a thin, low-level wrapper over a complex framework, and inherits that complexity; as a result it is difficult to use, especially if being called from another language with Tcl/Tk bindings. Additionally, it is not actively maintained these days. As a result, I wanted to move in another direction with AppleScript support in Tcl.
The aem package achieves this goal. It implements only a small subset of TclAE's functionality, the ability to respond to arbitrary Apple Events. This allows a Tcl/Tk application to be fully scriptable with AppleScript, in a manner similar to Cocoa applications written in Objective-C. (Other aspects of TclAE's functionality, such as the ability to build individual Apple Events and send them to other applications, can be achieved in a simpler fashion by using the TclApplescript package, which allows in-line execution of AppleScript code, or by executing AppleScript code using the "osascript" command-line tool.)
In addition to being a simpler implementation, the aem package also offers a cleaner, more straightforward design: it directly maps an AppleScript command with a parameter to a Tcl command with a parameter, for execution by the Tcl interpreter. The TclAE package inherits the cumbersome structure of the C-level Apple Event Manager API, which requires it to set up Tcl commands with "event" and "reply" parameters that have no relation to the Tcl code actually being executed, and which also require it to use C function calls to actually retrieve and dispose the relevant parameter from the AppleScript command. The aem package, being higher-level, delegates all that machinery to the underlying C code and does not expose it to the script level. In addition to making Tcl support for AppleScript easier to implement at the script level, this approach also makes it much easier to call aem from other scripting languages with Tcl/Tk bindings. (FileMorph, written in Perl, handles Apple Event support just fine with aem, but could not interface correctly with TclAE because of the parameter machinery.)
I've been hacking away at AppleScript support in my apps for a long time, and have been trying to find a workable solution for supporting Apple Events across the various scripting languages I use. In addition to the difficulties presented by TclAE, Perl's support has been broken for some years (see Mac::Carbon), and the appscript library for Python, Ruby and Objective-C is now actively unmaintained by its author (he discourages its use). With aem, I feel I finally have a library that will support the entire range of my applications, and I'm very proud of it.
I've released version 7.2 of PortAuthority, my GUI for the MacPorts Unix software installation system. The major feature of this release is streamlined under-the-hood support for AppleScript and some minor bug fixes. As always, this release is a free upgrade for registered users, and I encourage you to give it a try if you are a MacPorts user--PortAuthority is the only stable, actively-developed GUI client for the project.
I've released version 2.2 of FileMorph, my file modification tool for Mac OS X. The major feature of this release is a significant upgrade in AppleScript support, moving from a bare-bones implementation to a full-fledged AppleScript API, with genuine AppleScript commands that can be called from scripts and other applications. If you are looking for a scriptable application that can do batch modification and renaming of files and directories, with an intuitive UI, FileMorph may well fit the bill.Mon, 13 Jul 2015
Hot on the heels of Manpower'srecent upgrade, I've released version 6.1, which features some significant fixes in how Manpower registers itself as the default viewer for man pages on OS X (crashes were a regular occurrence), and also a significantly improved mechanism for responding to Apple Events. The two libraries powering these features are ones I plan to formally announce as open-source components in the near future, after a bit more testing and work on them.
As always, Manpower is a free upgrade to registered users. This also includes users of the Mac App Store version; I've been in contact with several people to whom I've provided free serials in recognition of the discontinuation of the Mac App Store version. Feel free to give the new version a try.Sat, 04 Jul 2015
I've released version 6.0 of Manpower, my Mac program for browing Unix system documentation, or man pages. As I've discussed recently, I've decided to withdraw this app from the Mac App Store, and so it will be sold only from my website.
This version of Manpower includes a UI refresh and some general refinements in performance. I feel it is the nicest app on OS X for viewing Unix documentation, and is well worth your time if you need to understand how something on your system works under the hood.
As always, it's a free upgrade for registered users. Give it a try.Tue, 30 Jun 2015
Now that I've begun that work, I find my heart simply isn't in it.
My heart is with the Mac.
I've done enough work on Windows and Linux to know that I can certainly develop for those platforms, but there's a reason I stay with the Mac--even using a cross-platform toolkit. It is by far the best environment for developers. I absolutely love developing for the Mac, even in spite of Apple's many restrictions, even as I move out of the Mac App Store for good.
If my work on the Mac isn't very rewarding financially, it can at least be rewarding in terms of enjoyment.
Reading Brent Simmons' recent blog entry, "Love," crystallized this for me:
Write the apps you want to write in your free time and out of love for the platform and for those specific apps. Take risks. Make those apps interesting and different. Don't play it safe. If you're not expecting money, you have nothing to lose.
I'll keep open-sourcing key components of my apps, and of course I will continue to do my essential open-source work as maintainer of Tk on the Mac. But I will also continue to do my own commercial work, making what money I can, and making the best apps I can.
You always come back to what you love.Sun, 28 Jun 2015
I am undertaking some fairly significant changes to my apps.
First, I have decided to open-source all my code, while still charging for pre-built binaries. I'm a strong proponent of open source code, and this move has been in the back of my mind for years. I have nothing to hide in my code. Others may find the code, not just the apps, useful. So I am in the process of establishing open-source repositories of all my code at http://fossil.codebykevin.com, using the elegant, superb Fossil source code management tool developed by D. Richard Hipp, author of the hugely popular SQLite database. I am first making available various libraries and packages I use in all of my applications, and then will follow with the source code to the apps themselves.
Second, I will be taking my applications cross-platform whenever it makes sense to do so. I would like my apps to be as widely used as possible, and this means that I would like to get them on more platforms than just the Mac. In practical terms this means I will be re-working the applications' code a bit to account for differences in platforms (Windows and Unix/Linux have different keyboard shortcuts than the Mac, for instance). I expect these differences to be fairly minor. The larger challenge here will be to set up processes for building and deploying the applications on each platform. (I expect to provide pre-built packages for Windows and Mac, but because of the diversity of platforms and deployment systems in the Unix world, I will only provide source code there.)
Third, these changes will require not just some modification of my applications' source code to fit in reasonably on each platform, but a simplification of the code design. In practice, this means removing code that is not essential to the application's functionality, or re-implmenting code in a simpler, more cross-platform way. I have a lot of Mac-specific code in my applications that requires compilation and which provides features that does not exist on other platforms (the Mac's Services API, for instance); I also have complex platform-specific code that can be implemented in a simpler, script-based cross-platform mode (printing files, for instance); I also have platform-specific code that expands functionality already provided by Tcl/Tk's core API's (application scripting) without much benefit. In deciding to keep or abandon native API's, I am going to look at whether the functionality is essential to the app and if it cannot be achieved in a similar way using a simpler approach. If the answer to both questions is yes, then the native functionally will stay. Otherwise, the functionality will be removed or re-implemented.
You may wonder about the business case for open-sourcing my code. Will this allow anyone to download my apps for free without paying for them? Well, in theory, yes; anyone can download and patch my apps for their needs. But building my apps is a non-trivial process--most people are not software developers--and I don't expect that many people are going to be inclined to bang away at the steps needed to patch and build my apps. There is still a huge convenience factor in providing an easy-to-install package from the get-go. Given that I am not currently seeing huge sales on the Mac, the upside to expanding my distribution to other platforms far exceeds the risk I am undertaking by open-sourcing my code.
Concurrent with this change, I plan to phase out my involvement with the Mac App Store. Like many other developers have found, my sales in the app store have dwindled to almost nothing, and the increasing technical restrictions on apps deployed there have simply become too large a headache to deal with. (See this discussion for more detail.) I still plan to remain a paid member of Apple's developer network, because of other benefits it provides (documentation, code-signing certificate, etc.). But I also plan to pursue other platforms as well.Wed, 13 May 2015
I've released version 2.1 of FileMorph, my tool for batch operations on files (renaming, changing modification dates, etc). This release is the result of a larger-than-anticipated effort to add AppleScript support for the application, and will be released on my website only--not in the Mac App Store.
I won't bore you with the technical details of why it took months to get AppleScript support worked out. Perl used to have very good support for AppleScript, but the Mac::Carbon extension library that, among other things, enabled AppleScript integration has not been updated for several years by its maintainer, Chris Nandor. (Apple's deprecation of Carbon makes updating the Mac::Carbon package a daunting task, as huge swaths of it no longer work and will need to be rewritten or removed.) My knowledge of Perl's low-level code to integrate natively with OS X is not good, and so I was not able to take the task on myself. As a result, I fell back on using Tcl's built-in support for executing Tcl code from AppleScript, substituting Perl commands for Tcl in this case. The result is a basic, but adequate, package of AppleScript support that can be used to automate FileMorph's functions, a mark of a polished and professional Mac application.
Unfortunately, I did run into some insurmountable issues with AppleScript and the Mac sandbox, which limits the operations an application can support. While in theory the sandbox should not cause any problems with AppleScript, in this case it did, perhaps because my application requires access to any directory on the user's hard-drive--a big no-no with the sandbox. Proceeding with a Mac App Store submission would require removing AppleScript support, a compromise I am unwilling to make.
I've loudly pulled my applications from the Mac App Store before, also because of issues with the sandbox, and subsequently tried to make things work with the sandbox to get the apps released in the MAS. I don't see that happening this time with FileMorph. Not only are the technical issue more intractable, but the business case for the MAS makes less sense than it used to: my sales have declined in the MAS to the point where it will hardly matter to my bottom line whether I sell there or not. Given the work required to maintain a sandboxed version of my app, the financial return does not seem worth it. If I have to maintain two versions of my app, it had better make sense financially. (I've even given thought to porting FileMorph to Windows, though no decision has been made there.)
I'm not going to make any sweeping announcements or policies about my apps in the MAS: they will be determined on a case-by-case basis. But the continued technical hurdles, combined with diminishing financial returns, make it hard to justify the work required to keep that sales channel open.Sat, 21 Mar 2015
I've just committed another big batch of patches to Tk-Cocoa, submitted by Marc Culler, that address some significant lingering issues with memory management, zombie windows not being removed, and event loop integration between Tk and Cocoa. Not only have Marc's patches made Tk run smoother and faster, they have also simplified the code in many places by removing outdated API calls and improving the comments on what the code does.
Marc submitted these updates after many hours of work in the interest of getting the visualization application he co-develops, SnapPy, running better with Tk-Cocoa: it's a large, complex application that really pushes Tk hard, and showed a lot of the bugs that persisted in Tk-Cocoa even after the recent updates. After these were done, he reported that SnapPy runs beautifully. My own testing shows that Tk is running better than ever, and some of the old reports that illustrated showstopper bugs simply don't display issues anymore, at all.
I wanted to highlight Marc's contributions because they go beyond fixing a specific bug or implementing a specific feature, as valuable as such contributions are; he has helped to substantially re-design Tk-Cocoa's implementation and documentation to be simpler, faster, and easier to work with. He has essentially served as co-author of this version of Tk-Cocoa, and I am enormously grateful that he has invested the time and patience to learn the relevant API's and propose changes that don't just fix specific issues, but substantially improve the whole. In gratitude for his contributions, I've credited him as one of the main authors of Tk on the Mac, along with Daniel Steffen, Jim Ingham, and Ian Reid, whose work has made Tk possible on Mac OS X.
Now that SnapPy is ready for a new release soon, I don't expect Marc to remain as busy with submitting patches: he can get back to his day job as a mathematics professor at the University of Illinois-Chicago. Now I truly believe, in no small part because of his work, that Tk-Cocoa is now the stable, fast, simple toolkit that all of its users need it to be on the Mac--and I encourage anyone who is still hesitant about moving to Tk-Cocoa to take the plunge.
Thanks, Marc!Fri, 06 Mar 2015
As this roundup at Michael Tsai's blog shows, an increasing number of Mac and iOS developers are having a hard time making a go of indie development. Even the developers of Vesper, considered the best notes app for the iPhone, have been unable to turn it into anything other than a part-time gig, and this is truly shocking considering the talent behind the app: Brent Simmons, a pioneer of indie Mac developers, who took a full-time gig at Omni; John Gruber, one of the leading Mac bloggers (and who, to be fair, likely doesn't need the income, as his blog generates upwards of $450,000 a year in advertising revenue); and Dave Wiskus, a talented app designer/UI specialist.
One culprit is the iOS and Mac App Store's drive for low app prices. As a result, some developers are raising prices. If lowering prices doesn't move the needle on sales, then raising prices certainly is worth a try. It's one reason I've continued to maintain a $30 price point for my own apps. If demand is relatively constant, then higher prices means more revenue.
Still, one thing that many developers do not seem to have admitted: there simply may not be room in the market for so many indie developers. Even if I concede Gus Mueller's cranky point that maybe my apps just suck, and Michael Tsai's observation that "fake is drawing a bunch of pixels into a plain NSView so that it looks like a text field or toolbar" (and to which I readily plead guilty), I think it's still true that the Mac and iOS app markets have changed in fundamental ways. When Brent Simmons has to take a full-time gig, what does that mean for the rest of us?
In other words, I'm afraid that for many of us, "sustainable software" may be less a strategy than a pipe dream.
As I work on updating my apps to take advantage of the improvements to Tk-Cocoa that I've overseen, I realize I am moving in a different direction than I did a few years ago: more Tk, less Cocoa.
What I mean by this is that I am gradually adjusting the balance my apps strike between native elements and aspects that don't hook directly into native API's. This process has been going on for a while, especially with the user interface of my apps; I've moved away from trying to use as many native Cocoa elements as possible at the user level. This gives me more flexibility in making changes when necessary, and also simplifies the development process, because by default Tk is easier to work with than Cocoa. However, I've also started doing this at a lower level in my apps, and also within Tk's internals, as the maintainer of the toolkit on the Mac.
One of the big architectural changes of Tk that I oversaw was removing a lot of Tk's direct use of Cocoa-based widgets, such as NSButton. Tk's drawing model simply can't handle multiple NSView-based widgets in a single window; it functions best when it draws everything itself, within a single NSView. Fortunately there is a Mac API to draw elements with a native appearance, HITheme; this allowed me to hand drawing of these widgets (buttons, scrollbars) over to Tk and still retain the native appearance. The result is more stable and faster drawing. Nonetheless, the balance here is more Tk, less Cocoa.
Another example of this is how I choose to implement Mac-styled features in my apps. In recent years I've gone deep into Cocoa to integrate these elements into my apps. Sometimes that works well, and sometimes it does not; for instance, my use of the 10.7 "fullscreen" API is not very stable, and crashed constantly when I was updating PortAuthority. (It had required a huge amount of code at the Objective-C level to work with Tk, and the result was not very satisfactory.) As a result, I re-worked my "fullscreen" Tk package to hook into Cocoa at only a couple of points--to set the fullscreen button in the window--but to otherwise pass management of the fullscreen transitions to Tk. (In technical terms, I wrote a category on NSWindow that overrides the "toggleFullScreen" method call to emit a Tk event, to which I could bind Tk's standard call to "wm attributes $w -fullscreen.")
Going forward, the question I will ask in going with a Tk-based or Cocoa-based approach will depend on a few factors: how smoothly does the Cocoa feature integrate with my apps, and how robust is a Tk-based alternative? The answer will likely vary on a case-by-case basis. For instance, my "macsheet" package, which displays Tk windows as modal sheet windows, works reasonably well in its current implementation, which uses Objective-C at a low level to set some basic properties, and uses a lot of management at the Tk script level to control the display of the window. It would be possible to implement a pure-Tk approach to the sheet window, but it's not necessary. As an alternative, while I've developed a package that wraps a Cocoa WebView to display HTML content in an app, I'm likely going to keep using a pure-Tk approach: the Tk script code I have works fine, and integrates well into my apps. I use the same approach for my app updating mechanism, which implements a Sparkle-style UI in Tk (while, in fact, hooking into Sparkle's XML appcast format under the hood).
Bottom line: it's sometimes necessary to hook into Cocoa at a low level, and at those times my apps will do so. But Tk offers considerable strengths in terms of simplicity, and at those times I am going to favor Tk.Tue, 03 Mar 2015
I've released version 7.0 of PortAuthority, my GUI for the MacPorts Unix software management system. PortAuthority is the oldest and, these days, only actively-maintained UI client for MacPorts. Its goal is to make using MacPorts easier.
This version of PortAuthority benefits from the many improvements to Tk/Cocoa that I've been working on in recent months, and is generally more responsive and refined. It also restores native system dialogs for running operations with elevated privileges, such as installing a software port. Finally, this release adds support for Mac system notifications if Growl is not installed.
PortAuthority costs $30 for a new installation, and lifetime upgrades are free to registered users. It's worth a try.Mon, 02 Mar 2015
I've released version 1.2 of my "fullscreen" package for Tk-Cocoa.
This package adds native a "fullscreen" button to Tk windows on Mac OS X 10.7 and later. The ::fullscreen::fullscreen command implements a Cocoa-native fullscreen button that, when pressed, will move the window to fullscreen status. The window can be restored to its previous state by clicking the "resize" icon in the application menu. This command is implemented internally by overriding the native Cocoa fullscreen API to generate a "ToggleFullScreen" virtual event, which is then passed to Tcl to map the fullscreen status to the window via the "wm attributes $w -fullscreen" command. This ensures smoother integration between Tk and Cocoa by working with a standard Tk mechanism, but does not work identically to fully-native Cocoa fullscreen implementations. Earlier implementations of the fullscreen attempted to provide a fully-native Cocoa implementation, but it was highly-complex and prone to crashing.
More information: http://opensource.codebykevin.com/native.html#fullscreenFri, 30 Jan 2015
Tk was initially ported to Cocoa in 2009 by Daniel Steffen, who had maintained Tk for several years. Apple sponsored the port of Tk to Cocoa after it deprecated its Carbon frameworks, which had previously been the foundation for Tk. The Cocoa port was a top-to-bottom rewrite, essentially cut from whole cloth rather than a simple update or port. Daniel Steffen had to create an entirely new code architecture for Tk to work with Cocoa, which has a significantly different, higher-level approach to GUI programming than Tk, based on a lower-level model derived from Unix's Xlib library (and which had strong parallels with Carbon). The resulting Cocoa-based Tk was a remarkable piece of design, and brought many advantages in speed and system integration.
After Tk-Cocoa was released in the summer of 2009, Daniel did a major round of updates and bug fixes to smooth out some rough edges in the initial release, and then left Tcl/Tk development to join Apple full-time. A bit later, after releasing some Tcl/Tk extensions that integrated with parts of Cocoa and contributing some patches to the Tk core, I took over as maintainer.
Over the next couple of years, as users gained more experience with Tk-Cocoa, some limitations of the system became apparent. Apart from the standard-issue bugs and glitches that are part of any software project, and which were relatively easy to fix, the differing way Tk and Cocoa handled user input and processing of events caused a variety of issues. Applications would occasionally freeze, and stop responding to user input, because they would become overloaded with events to process. There would be occasional issues with drawing and flickering. These issues were structural, not simply glitches that can be fixed with a few lines of code. The complexity involved would have required a major redesign of Cocoa and Tk's integration, something Daniel Steffen was precluded from doing as an Apple employee (Apple's developers are not allowed to participate in open source projects in most cases); in any case it was not even entirely clear if the integration could be designed differently.
Another issue also arose, not technical but one of policy: Tk-Cocoa, at a deep level, made use of private, undocumented Apple API's to improve drawing performance, using methods borrowed from Apple's open-source WebKit project. This practice, while always discouraged by Apple, took on greater importance when Apple introduced the Mac App Store, which stated that apps making use of private API's would be rejected. This was a serious issues for developers who, like me, wanted to distribute Tcl/Tk apps through the Mac App Store; the only workaround at the time was to link my applications to the somewhat buggy version of Tk-Cocoa that Apple shipped with OS X. (System libraries are allowed to use private API's.)
This was the situation for a couple of years: a lot of low-level frustration with the structural limitations of Tk-Cocoa, no clear path to fixing them, just incremental fixes of bugs, and a handful of Tcl/Tk apps in the Mac App Store linking to an aging version of Tk that did not include any improvements. In the summer of 2014, however, after working with Marc Culler on some patches he submitted to improve how images are rendered in Tk/Cocoa (some OS updates had broken Tk's previous mechanism), I decided to take another look at the deeper issues in Tk.
I found, to my surprise, that Tk would work without the private API calls--so I removed them altogether. I also saw, however, that Tk worked much better with them included. Their removal opened a Pandora's box of bugs and very serious drawing issues: vast amounts of flickering, ghostly images of buttons and scrollbars appearing at random whenever a window was resized or changed in any way. Except for Apple's policy about private API's, which presented a major obstacle to using Tcl/Tk in via its premier distribution channel, I would rather have left the private API's in. They made Tk/Cocoa run much better. But because of Apple's policies, they had to go--and their removal left a huge mess behind.
After doing some more work with Tk without the private API's, the issues started to sort themselves out into some predictable patterns, which made it possible to chart a plan to address them. These patterns included:
The common denominator between these bugs seemed to be that Tk-Cocoa had trouble managing a complex hierarchy of Cocoa views in a window. Tk-Cocoa's design involves using a single NSView in an NSWindow to render the layout of a Tk interface. It handles widgets drawn by Tk that are not directly derived from Cocoa widgets, such as listboxes, just fine. But NSButton and NSScroller widgets (themselves based on NSView) present added complexity that Tk seems unable to manage easily without the private API's.
The solution to these issues focused on reducing the complexity of what Tk-Cocoa had to draw:
The window-resize code I developed on my own, with a lot of trial and error and lots of time on Google. The HITheme code, however, was heavily based on work that others had done in past years--a good thing, because that was a much more complicated task.
The button code, both for regular pushbuttons and popup menubuttons, was based on a patch developed a decade ago by Revar Desmera, a polyglot developer who has done a number of projects in Tcl/Tk. Revar's patch, which added support for the HITheme API to Tk buttons, was never adopted, but it was still sitting in Tk's code tracker and provided the perfect foundation. I had to tweak and update some things, but that work only took a week or two. I am grateful to Revar for his initial work.
The scrolling code was the hardest piece of this puzzle. There was no existing HITheme scrolling implementation for me to tweak, so I had to start from scratch; I worked on the scrolling code for about a month with only modest progress, borrowing some code from Tk's older Carbon implementation and its Unix implementation, which is the foundation for both the Mac and Windows ports of Tk. I got the scrolling code to a point where it would draw the scrollbar, and nothing more. Finally, after rummaging around in Tk's source tree, I found an old patch from Donal Fellows, one of Tcl/Tk's core developers, which fixed a bug in the Mac's other HITheme-derived widgets (scale and progress bar) that seemed suspiciously like the scrollbar issue: the widgets had to apply some kind of multiplier to set their values correctly, and thus redraw. (The math was, and is, over my head.) Once I got that placed into the scrollbar code, things progressed quickly. The final piece of the puzzle was to get the scrollbar to respond to being dragged by the mouse; my initial work here focused on bringing a lot of the old Carbon code back into Cocoa, and updating it somehow. The final solution was beautifully simple and elegant. Based on an observation from Jim Ingham, Tk's original maintainer on the Mac, that the HITheme API would allow Tk/Mac's scrollbar code to become more like its Unix version, that's what I decided to do: throw out all the old Carbon code, import mostly Unix code, and just let the HITheme scrollbar be redrawn by the native API's, not moved and driven by them. Bingo.
I've committed all these updates to Tk's main line of development (trunk) and its still-supported 8.5 version this week, and they are ready to be tested and used. Here is the difference these changes make in Tk-Cocoa:
Let me be clear: Tk-Cocoa remains Cocoa-based. I have left most of Daniel Steffen's brilliant architecture unchanged: Tk's app structure is still based on the NSApp API, it still has the same complex event loop integration, and it still draws into an NSWindow containing an NSView. There still may be bugs owing to the complexity that cannot be changed. However, by making these changes to Tk-Cocoa's drawing code, I have removed some of the complexity that was there before, and brought Tk a bit closer to the level of abstraction it had under Carbon, and retains under both X11 (Unix) and Windows, where it is most performant. As a result, I feel this is a significant evolution of Tk-Cocoa, and an improvement, and I do not feel embarrassed in calling it Tk-Cocoa 2.0.
To all Tk/Mac developers and users out there: I hope you find these changes helpful. And to the developers named here and to those who have provided encouragement and feedback: thank you. While at times I have complained that I work alone, this is not true: I have relied on the contributions of many, both of code and feedback, and I am grateful.