Code by Kevin, Programming, code, business, and other pursuits
Kevin Walzer, software developer.
Subscribe to RSS Feed
Get a syndicated feed of my weblog.
Site design: Skeleton
I've released version 2.1 of FileMorph, my tool for batch operations on files (renaming, changing modification dates, etc). This release is the result of a larger-than-anticipated effort to add AppleScript support for the application, and will be released on my website only--not in the Mac App Store.
I won't bore you with the technical details of why it took months to get AppleScript support worked out. Perl used to have very good support for AppleScript, but the Mac::Carbon extension library that, among other things, enabled AppleScript integration has not been updated for several years by its maintainer, Chris Nandor. (Apple's deprecation of Carbon makes updating the Mac::Carbon package a daunting task, as huge swaths of it no longer work and will need to be rewritten or removed.) My knowledge of Perl's low-level code to integrate natively with OS X is not good, and so I was not able to take the task on myself. As a result, I fell back on using Tcl's built-in support for executing Tcl code from AppleScript, substituting Perl commands for Tcl in this case. The result is a basic, but adequate, package of AppleScript support that can be used to automate FileMorph's functions, a mark of a polished and professional Mac application.
Unfortunately, I did run into some insurmountable issues with AppleScript and the Mac sandbox, which limits the operations an application can support. While in theory the sandbox should not cause any problems with AppleScript, in this case it did, perhaps because my application requires access to any directory on the user's hard-drive--a big no-no with the sandbox. Proceeding with a Mac App Store submission would require removing AppleScript support, a compromise I am unwilling to make.
I've loudly pulled my applications from the Mac App Store before, also because of issues with the sandbox, and subsequently tried to make things work with the sandbox to get the apps released in the MAS. I don't see that happening this time with FileMorph. Not only are the technical issue more intractable, but the business case for the MAS makes less sense than it used to: my sales have declined in the MAS to the point where it will hardly matter to my bottom line whether I sell there or not. Given the work required to maintain a sandboxed version of my app, the financial return does not seem worth it. If I have to maintain two versions of my app, it had better make sense financially. (I've even given thought to porting FileMorph to Windows, though no decision has been made there.)
I'm not going to make any sweeping announcements or policies about my apps in the MAS: they will be determined on a case-by-case basis. But the continued technical hurdles, combined with diminishing financial returns, make it hard to justify the work required to keep that sales channel open.Sat, 21 Mar 2015
I've just committed another big batch of patches to Tk-Cocoa, submitted by Marc Culler, that address some significant lingering issues with memory management, zombie windows not being removed, and event loop integration between Tk and Cocoa. Not only have Marc's patches made Tk run smoother and faster, they have also simplified the code in many places by removing outdated API calls and improving the comments on what the code does.
Marc submitted these updates after many hours of work in the interest of getting the visualization application he co-develops, SnapPy, running better with Tk-Cocoa: it's a large, complex application that really pushes Tk hard, and showed a lot of the bugs that persisted in Tk-Cocoa even after the recent updates. After these were done, he reported that SnapPy runs beautifully. My own testing shows that Tk is running better than ever, and some of the old reports that illustrated showstopper bugs simply don't display issues anymore, at all.
I wanted to highlight Marc's contributions because they go beyond fixing a specific bug or implementing a specific feature, as valuable as such contributions are; he has helped to substantially re-design Tk-Cocoa's implementation and documentation to be simpler, faster, and easier to work with. He has essentially served as co-author of this version of Tk-Cocoa, and I am enormously grateful that he has invested the time and patience to learn the relevant API's and propose changes that don't just fix specific issues, but substantially improve the whole. In gratitude for his contributions, I've credited him as one of the main authors of Tk on the Mac, along with Daniel Steffen, Jim Ingham, and Ian Reid, whose work has made Tk possible on Mac OS X.
Now that SnapPy is ready for a new release soon, I don't expect Marc to remain as busy with submitting patches: he can get back to his day job as a mathematics professor at the University of Illinois-Chicago. Now I truly believe, in no small part because of his work, that Tk-Cocoa is now the stable, fast, simple toolkit that all of its users need it to be on the Mac--and I encourage anyone who is still hesitant about moving to Tk-Cocoa to take the plunge.
Thanks, Marc!Fri, 06 Mar 2015
As this roundup at Michael Tsai's blog shows, an increasing number of Mac and iOS developers are having a hard time making a go of indie development. Even the developers of Vesper, considered the best notes app for the iPhone, have been unable to turn it into anything other than a part-time gig, and this is truly shocking considering the talent behind the app: Brent Simmons, a pioneer of indie Mac developers, who took a full-time gig at Omni; John Gruber, one of the leading Mac bloggers (and who, to be fair, likely doesn't need the income, as his blog generates upwards of $450,000 a year in advertising revenue); and Dave Wiskus, a talented app designer/UI specialist.
One culprit is the iOS and Mac App Store's drive for low app prices. As a result, some developers are raising prices. If lowering prices doesn't move the needle on sales, then raising prices certainly is worth a try. It's one reason I've continued to maintain a $30 price point for my own apps. If demand is relatively constant, then higher prices means more revenue.
Still, one thing that many developers do not seem to have admitted: there simply may not be room in the market for so many indie developers. Even if I concede Gus Mueller's cranky point that maybe my apps just suck, and Michael Tsai's observation that "fake is drawing a bunch of pixels into a plain NSView so that it looks like a text field or toolbar" (and to which I readily plead guilty), I think it's still true that the Mac and iOS app markets have changed in fundamental ways. When Brent Simmons has to take a full-time gig, what does that mean for the rest of us?
In other words, I'm afraid that for many of us, "sustainable software" may be less a strategy than a pipe dream.
As I work on updating my apps to take advantage of the improvements to Tk-Cocoa that I've overseen, I realize I am moving in a different direction than I did a few years ago: more Tk, less Cocoa.
What I mean by this is that I am gradually adjusting the balance my apps strike between native elements and aspects that don't hook directly into native API's. This process has been going on for a while, especially with the user interface of my apps; I've moved away from trying to use as many native Cocoa elements as possible at the user level. This gives me more flexibility in making changes when necessary, and also simplifies the development process, because by default Tk is easier to work with than Cocoa. However, I've also started doing this at a lower level in my apps, and also within Tk's internals, as the maintainer of the toolkit on the Mac.
One of the big architectural changes of Tk that I oversaw was removing a lot of Tk's direct use of Cocoa-based widgets, such as NSButton. Tk's drawing model simply can't handle multiple NSView-based widgets in a single window; it functions best when it draws everything itself, within a single NSView. Fortunately there is a Mac API to draw elements with a native appearance, HITheme; this allowed me to hand drawing of these widgets (buttons, scrollbars) over to Tk and still retain the native appearance. The result is more stable and faster drawing. Nonetheless, the balance here is more Tk, less Cocoa.
Another example of this is how I choose to implement Mac-styled features in my apps. In recent years I've gone deep into Cocoa to integrate these elements into my apps. Sometimes that works well, and sometimes it does not; for instance, my use of the 10.7 "fullscreen" API is not very stable, and crashed constantly when I was updating PortAuthority. (It had required a huge amount of code at the Objective-C level to work with Tk, and the result was not very satisfactory.) As a result, I re-worked my "fullscreen" Tk package to hook into Cocoa at only a couple of points--to set the fullscreen button in the window--but to otherwise pass management of the fullscreen transitions to Tk. (In technical terms, I wrote a category on NSWindow that overrides the "toggleFullScreen" method call to emit a Tk event, to which I could bind Tk's standard call to "wm attributes $w -fullscreen.")
Going forward, the question I will ask in going with a Tk-based or Cocoa-based approach will depend on a few factors: how smoothly does the Cocoa feature integrate with my apps, and how robust is a Tk-based alternative? The answer will likely vary on a case-by-case basis. For instance, my "macsheet" package, which displays Tk windows as modal sheet windows, works reasonably well in its current implementation, which uses Objective-C at a low level to set some basic properties, and uses a lot of management at the Tk script level to control the display of the window. It would be possible to implement a pure-Tk approach to the sheet window, but it's not necessary. As an alternative, while I've developed a package that wraps a Cocoa WebView to display HTML content in an app, I'm likely going to keep using a pure-Tk approach: the Tk script code I have works fine, and integrates well into my apps. I use the same approach for my app updating mechanism, which implements a Sparkle-style UI in Tk (while, in fact, hooking into Sparkle's XML appcast format under the hood).
Bottom line: it's sometimes necessary to hook into Cocoa at a low level, and at those times my apps will do so. But Tk offers considerable strengths in terms of simplicity, and at those times I am going to favor Tk.Tue, 03 Mar 2015
I've released version 7.0 of PortAuthority, my GUI for the MacPorts Unix software management system. PortAuthority is the oldest and, these days, only actively-maintained UI client for MacPorts. Its goal is to make using MacPorts easier.
This version of PortAuthority benefits from the many improvements to Tk/Cocoa that I've been working on in recent months, and is generally more responsive and refined. It also restores native system dialogs for running operations with elevated privileges, such as installing a software port. Finally, this release adds support for Mac system notifications if Growl is not installed.
PortAuthority costs $30 for a new installation, and lifetime upgrades are free to registered users. It's worth a try.Mon, 02 Mar 2015
I've released version 1.2 of my "fullscreen" package for Tk-Cocoa.
This package adds native a "fullscreen" button to Tk windows on Mac OS X 10.7 and later. The ::fullscreen::fullscreen command implements a Cocoa-native fullscreen button that, when pressed, will move the window to fullscreen status. The window can be restored to its previous state by clicking the "resize" icon in the application menu. This command is implemented internally by overriding the native Cocoa fullscreen API to generate a "ToggleFullScreen" virtual event, which is then passed to Tcl to map the fullscreen status to the window via the "wm attributes $w -fullscreen" command. This ensures smoother integration between Tk and Cocoa by working with a standard Tk mechanism, but does not work identically to fully-native Cocoa fullscreen implementations. Earlier implementations of the fullscreen attempted to provide a fully-native Cocoa implementation, but it was highly-complex and prone to crashing.
More information: http://opensource.codebykevin.com/native.html#fullscreenFri, 30 Jan 2015
Tk was initially ported to Cocoa in 2009 by Daniel Steffen, who had maintained Tk for several years. Apple sponsored the port of Tk to Cocoa after it deprecated its Carbon frameworks, which had previously been the foundation for Tk. The Cocoa port was a top-to-bottom rewrite, essentially cut from whole cloth rather than a simple update or port. Daniel Steffen had to create an entirely new code architecture for Tk to work with Cocoa, which has a significantly different, higher-level approach to GUI programming than Tk, based on a lower-level model derived from Unix's Xlib library (and which had strong parallels with Carbon). The resulting Cocoa-based Tk was a remarkable piece of design, and brought many advantages in speed and system integration.
After Tk-Cocoa was released in the summer of 2009, Daniel did a major round of updates and bug fixes to smooth out some rough edges in the initial release, and then left Tcl/Tk development to join Apple full-time. A bit later, after releasing some Tcl/Tk extensions that integrated with parts of Cocoa and contributing some patches to the Tk core, I took over as maintainer.
Over the next couple of years, as users gained more experience with Tk-Cocoa, some limitations of the system became apparent. Apart from the standard-issue bugs and glitches that are part of any software project, and which were relatively easy to fix, the differing way Tk and Cocoa handled user input and processing of events caused a variety of issues. Applications would occasionally freeze, and stop responding to user input, because they would become overloaded with events to process. There would be occasional issues with drawing and flickering. These issues were structural, not simply glitches that can be fixed with a few lines of code. The complexity involved would have required a major redesign of Cocoa and Tk's integration, something Daniel Steffen was precluded from doing as an Apple employee (Apple's developers are not allowed to participate in open source projects in most cases); in any case it was not even entirely clear if the integration could be designed differently.
Another issue also arose, not technical but one of policy: Tk-Cocoa, at a deep level, made use of private, undocumented Apple API's to improve drawing performance, using methods borrowed from Apple's open-source WebKit project. This practice, while always discouraged by Apple, took on greater importance when Apple introduced the Mac App Store, which stated that apps making use of private API's would be rejected. This was a serious issues for developers who, like me, wanted to distribute Tcl/Tk apps through the Mac App Store; the only workaround at the time was to link my applications to the somewhat buggy version of Tk-Cocoa that Apple shipped with OS X. (System libraries are allowed to use private API's.)
This was the situation for a couple of years: a lot of low-level frustration with the structural limitations of Tk-Cocoa, no clear path to fixing them, just incremental fixes of bugs, and a handful of Tcl/Tk apps in the Mac App Store linking to an aging version of Tk that did not include any improvements. In the summer of 2014, however, after working with Marc Culler on some patches he submitted to improve how images are rendered in Tk/Cocoa (some OS updates had broken Tk's previous mechanism), I decided to take another look at the deeper issues in Tk.
I found, to my surprise, that Tk would work without the private API calls--so I removed them altogether. I also saw, however, that Tk worked much better with them included. Their removal opened a Pandora's box of bugs and very serious drawing issues: vast amounts of flickering, ghostly images of buttons and scrollbars appearing at random whenever a window was resized or changed in any way. Except for Apple's policy about private API's, which presented a major obstacle to using Tcl/Tk in via its premier distribution channel, I would rather have left the private API's in. They made Tk/Cocoa run much better. But because of Apple's policies, they had to go--and their removal left a huge mess behind.
After doing some more work with Tk without the private API's, the issues started to sort themselves out into some predictable patterns, which made it possible to chart a plan to address them. These patterns included:
The common denominator between these bugs seemed to be that Tk-Cocoa had trouble managing a complex hierarchy of Cocoa views in a window. Tk-Cocoa's design involves using a single NSView in an NSWindow to render the layout of a Tk interface. It handles widgets drawn by Tk that are not directly derived from Cocoa widgets, such as listboxes, just fine. But NSButton and NSScroller widgets (themselves based on NSView) present added complexity that Tk seems unable to manage easily without the private API's.
The solution to these issues focused on reducing the complexity of what Tk-Cocoa had to draw:
The window-resize code I developed on my own, with a lot of trial and error and lots of time on Google. The HITheme code, however, was heavily based on work that others had done in past years--a good thing, because that was a much more complicated task.
The button code, both for regular pushbuttons and popup menubuttons, was based on a patch developed a decade ago by Revar Desmera, a polyglot developer who has done a number of projects in Tcl/Tk. Revar's patch, which added support for the HITheme API to Tk buttons, was never adopted, but it was still sitting in Tk's code tracker and provided the perfect foundation. I had to tweak and update some things, but that work only took a week or two. I am grateful to Revar for his initial work.
The scrolling code was the hardest piece of this puzzle. There was no existing HITheme scrolling implementation for me to tweak, so I had to start from scratch; I worked on the scrolling code for about a month with only modest progress, borrowing some code from Tk's older Carbon implementation and its Unix implementation, which is the foundation for both the Mac and Windows ports of Tk. I got the scrolling code to a point where it would draw the scrollbar, and nothing more. Finally, after rummaging around in Tk's source tree, I found an old patch from Donal Fellows, one of Tcl/Tk's core developers, which fixed a bug in the Mac's other HITheme-derived widgets (scale and progress bar) that seemed suspiciously like the scrollbar issue: the widgets had to apply some kind of multiplier to set their values correctly, and thus redraw. (The math was, and is, over my head.) Once I got that placed into the scrollbar code, things progressed quickly. The final piece of the puzzle was to get the scrollbar to respond to being dragged by the mouse; my initial work here focused on bringing a lot of the old Carbon code back into Cocoa, and updating it somehow. The final solution was beautifully simple and elegant. Based on an observation from Jim Ingham, Tk's original maintainer on the Mac, that the HITheme API would allow Tk/Mac's scrollbar code to become more like its Unix version, that's what I decided to do: throw out all the old Carbon code, import mostly Unix code, and just let the HITheme scrollbar be redrawn by the native API's, not moved and driven by them. Bingo.
I've committed all these updates to Tk's main line of development (trunk) and its still-supported 8.5 version this week, and they are ready to be tested and used. Here is the difference these changes make in Tk-Cocoa:
Let me be clear: Tk-Cocoa remains Cocoa-based. I have left most of Daniel Steffen's brilliant architecture unchanged: Tk's app structure is still based on the NSApp API, it still has the same complex event loop integration, and it still draws into an NSWindow containing an NSView. There still may be bugs owing to the complexity that cannot be changed. However, by making these changes to Tk-Cocoa's drawing code, I have removed some of the complexity that was there before, and brought Tk a bit closer to the level of abstraction it had under Carbon, and retains under both X11 (Unix) and Windows, where it is most performant. As a result, I feel this is a significant evolution of Tk-Cocoa, and an improvement, and I do not feel embarrassed in calling it Tk-Cocoa 2.0.
To all Tk/Mac developers and users out there: I hope you find these changes helpful. And to the developers named here and to those who have provided encouragement and feedback: thank you. While at times I have complained that I work alone, this is not true: I have relied on the contributions of many, both of code and feedback, and I am grateful.Tue, 16 Dec 2014
I've released version 1.1 of TkWebView, my Tcl package that allows a developer to add a native Cocoa WebView/WebKit widget to a Tcl/Tk application, using the same rendering engine that powers Safari.
TkWebView is available under a BSD-style license.Thu, 04 Dec 2014
I'm pleased to announce the initial release of ruby2app, a tool for bundling Ruby scripts into standalone Mac applications:
ruby2app is a simple command-line tool, inspired by similar Mac tools like py2app for Python, that bundles a Ruby script with the Ruby runtime into a standalone Mac application. It reads a basic configuration file to find the appropriate Ruby installation, compiles a small C program that embeds the Ruby interpreter, copies the entire Ruby installation into the application bundle, and re-links the relevant libraries with the main executable. Ruby must be compiled with the "--enable-load-relative" flag for ruby2app to work.
At present the tool is primarily geared for Ruby-Tk applications, since Tk is the best-supported open-source GUI framework with Ruby bindings on the Mac in 2014, and (not coincidentally) is the one I use. But I invite others to submit patches to add support for other UI frameworks as well; Qt support might be a good candidate. Support for RubyMotion is outside the scope of ruby2app, as RubyMotion presumably has its own toolchain.
I am still in the process of learning Ruby, and in fact wrote this tool to be certain that I could deploy a Ruby-Tk application in a convenient manner on the Mac desktop. There are no other current tools that provide basic support for deployment of an open-source Mac application written in Ruby. Ruby-Cocoa and MacRuby are effectively dead and their tools (standaloneify.rb) are obsolete, RubyMotion has its own toolchain, and crate and rubyscript2exe do not support current versions of Ruby.
I do have a decade of experience writing Mac applications in Python, Perl and Tcl using the Tk toolkit, so I adapted some ideas from bundling tools for those languages in creating ruby2app. ruby2app is written in Bash, and thus should be easy to read and modify as required, and should fit in well to any application building toolchain.
ruby2app is available under the MIT license, and feedback is welcome.Wed, 26 Nov 2014
This is a significant announcement because it means that all private API's have been removed from Tcl/Tk, as verified by Apple's own scanning tools, and therefore apps using Tcl/Tk can be submitted to the MAS using up-to-date versions of Tcl/Tk and without linking to Apple's own outdated system installation. As I noted previously, I spent several weeks scrubbing the private API calls, forbidden by Apple, from Tcl/Tk. One reason for the lag between the release of FileMorph 2.0 and its acceptance in the MAS was that there was still a bit of private cruft that I had overlooked, which has since been addressed.
Going forward, I plan to keep my apps as up-to-date as possible with Tcl/Tk, and not link to Apple's libraries any more.