Code by Kevin, Programming, code, business, and other pursuits
Kevin Walzer, software developer.
Subscribe to RSS Feed
Get a syndicated feed of my weblog.
Site design: Skeleton
I've released version 6.0 of Manpower, my Mac program for browing Unix system documentation, or man pages. As I've discussed recently, I've decided to withdraw this app from the Mac App Store, and so it will be sold only from my website.
This version of Manpower includes a UI refresh and some general refinements in performance. I feel it is the nicest app on OS X for viewing Unix documentation, and is well worth your time if you need to understand how something on your system works under the hood.
As always, it's a free upgrade for registered users. Give it a try.Tue, 30 Jun 2015
Now that I've begun that work, I find my heart simply isn't in it.
My heart is with the Mac.
I've done enough work on Windows and Linux to know that I can certainly develop for those platforms, but there's a reason I stay with the Mac--even using a cross-platform toolkit. It is by far the best environment for developers. I absolutely love developing for the Mac, even in spite of Apple's many restrictions, even as I move out of the Mac App Store for good.
If my work on the Mac isn't very rewarding financially, it can at least be rewarding in terms of enjoyment.
Reading Brent Simmons' recent blog entry, "Love," crystallized this for me:
Write the apps you want to write in your free time and out of love for the platform and for those specific apps. Take risks. Make those apps interesting and different. Don't play it safe. If you're not expecting money, you have nothing to lose.
I'll keep open-sourcing key components of my apps, and of course I will continue to do my essential open-source work as maintainer of Tk on the Mac. But I will also continue to do my own commercial work, making what money I can, and making the best apps I can.
You always come back to what you love.Sun, 28 Jun 2015
I am undertaking some fairly significant changes to my apps.
First, I have decided to open-source all my code, while still charging for pre-built binaries. I'm a strong proponent of open source code, and this move has been in the back of my mind for years. I have nothing to hide in my code. Others may find the code, not just the apps, useful. So I am in the process of establishing open-source repositories of all my code at http://fossil.codebykevin.com, using the elegant, superb Fossil source code management tool developed by D. Richard Hipp, author of the hugely popular SQLite database. I am first making available various libraries and packages I use in all of my applications, and then will follow with the source code to the apps themselves.
Second, I will be taking my applications cross-platform whenever it makes sense to do so. I would like my apps to be as widely used as possible, and this means that I would like to get them on more platforms than just the Mac. In practical terms this means I will be re-working the applications' code a bit to account for differences in platforms (Windows and Unix/Linux have different keyboard shortcuts than the Mac, for instance). I expect these differences to be fairly minor. The larger challenge here will be to set up processes for building and deploying the applications on each platform. (I expect to provide pre-built packages for Windows and Mac, but because of the diversity of platforms and deployment systems in the Unix world, I will only provide source code there.)
Third, these changes will require not just some modification of my applications' source code to fit in reasonably on each platform, but a simplification of the code design. In practice, this means removing code that is not essential to the application's functionality, or re-implmenting code in a simpler, more cross-platform way. I have a lot of Mac-specific code in my applications that requires compilation and which provides features that does not exist on other platforms (the Mac's Services API, for instance); I also have complex platform-specific code that can be implemented in a simpler, script-based cross-platform mode (printing files, for instance); I also have platform-specific code that expands functionality already provided by Tcl/Tk's core API's (application scripting) without much benefit. In deciding to keep or abandon native API's, I am going to look at whether the functionality is essential to the app and if it cannot be achieved in a similar way using a simpler approach. If the answer to both questions is yes, then the native functionally will stay. Otherwise, the functionality will be removed or re-implemented.
You may wonder about the business case for open-sourcing my code. Will this allow anyone to download my apps for free without paying for them? Well, in theory, yes; anyone can download and patch my apps for their needs. But building my apps is a non-trivial process--most people are not software developers--and I don't expect that many people are going to be inclined to bang away at the steps needed to patch and build my apps. There is still a huge convenience factor in providing an easy-to-install package from the get-go. Given that I am not currently seeing huge sales on the Mac, the upside to expanding my distribution to other platforms far exceeds the risk I am undertaking by open-sourcing my code.
Concurrent with this change, I plan to phase out my involvement with the Mac App Store. Like many other developers have found, my sales in the app store have dwindled to almost nothing, and the increasing technical restrictions on apps deployed there have simply become too large a headache to deal with. (See this discussion for more detail.) I still plan to remain a paid member of Apple's developer network, because of other benefits it provides (documentation, code-signing certificate, etc.). But I also plan to pursue other platforms as well.Wed, 13 May 2015
I've released version 2.1 of FileMorph, my tool for batch operations on files (renaming, changing modification dates, etc). This release is the result of a larger-than-anticipated effort to add AppleScript support for the application, and will be released on my website only--not in the Mac App Store.
I won't bore you with the technical details of why it took months to get AppleScript support worked out. Perl used to have very good support for AppleScript, but the Mac::Carbon extension library that, among other things, enabled AppleScript integration has not been updated for several years by its maintainer, Chris Nandor. (Apple's deprecation of Carbon makes updating the Mac::Carbon package a daunting task, as huge swaths of it no longer work and will need to be rewritten or removed.) My knowledge of Perl's low-level code to integrate natively with OS X is not good, and so I was not able to take the task on myself. As a result, I fell back on using Tcl's built-in support for executing Tcl code from AppleScript, substituting Perl commands for Tcl in this case. The result is a basic, but adequate, package of AppleScript support that can be used to automate FileMorph's functions, a mark of a polished and professional Mac application.
Unfortunately, I did run into some insurmountable issues with AppleScript and the Mac sandbox, which limits the operations an application can support. While in theory the sandbox should not cause any problems with AppleScript, in this case it did, perhaps because my application requires access to any directory on the user's hard-drive--a big no-no with the sandbox. Proceeding with a Mac App Store submission would require removing AppleScript support, a compromise I am unwilling to make.
I've loudly pulled my applications from the Mac App Store before, also because of issues with the sandbox, and subsequently tried to make things work with the sandbox to get the apps released in the MAS. I don't see that happening this time with FileMorph. Not only are the technical issue more intractable, but the business case for the MAS makes less sense than it used to: my sales have declined in the MAS to the point where it will hardly matter to my bottom line whether I sell there or not. Given the work required to maintain a sandboxed version of my app, the financial return does not seem worth it. If I have to maintain two versions of my app, it had better make sense financially. (I've even given thought to porting FileMorph to Windows, though no decision has been made there.)
I'm not going to make any sweeping announcements or policies about my apps in the MAS: they will be determined on a case-by-case basis. But the continued technical hurdles, combined with diminishing financial returns, make it hard to justify the work required to keep that sales channel open.Sat, 21 Mar 2015
I've just committed another big batch of patches to Tk-Cocoa, submitted by Marc Culler, that address some significant lingering issues with memory management, zombie windows not being removed, and event loop integration between Tk and Cocoa. Not only have Marc's patches made Tk run smoother and faster, they have also simplified the code in many places by removing outdated API calls and improving the comments on what the code does.
Marc submitted these updates after many hours of work in the interest of getting the visualization application he co-develops, SnapPy, running better with Tk-Cocoa: it's a large, complex application that really pushes Tk hard, and showed a lot of the bugs that persisted in Tk-Cocoa even after the recent updates. After these were done, he reported that SnapPy runs beautifully. My own testing shows that Tk is running better than ever, and some of the old reports that illustrated showstopper bugs simply don't display issues anymore, at all.
I wanted to highlight Marc's contributions because they go beyond fixing a specific bug or implementing a specific feature, as valuable as such contributions are; he has helped to substantially re-design Tk-Cocoa's implementation and documentation to be simpler, faster, and easier to work with. He has essentially served as co-author of this version of Tk-Cocoa, and I am enormously grateful that he has invested the time and patience to learn the relevant API's and propose changes that don't just fix specific issues, but substantially improve the whole. In gratitude for his contributions, I've credited him as one of the main authors of Tk on the Mac, along with Daniel Steffen, Jim Ingham, and Ian Reid, whose work has made Tk possible on Mac OS X.
Now that SnapPy is ready for a new release soon, I don't expect Marc to remain as busy with submitting patches: he can get back to his day job as a mathematics professor at the University of Illinois-Chicago. Now I truly believe, in no small part because of his work, that Tk-Cocoa is now the stable, fast, simple toolkit that all of its users need it to be on the Mac--and I encourage anyone who is still hesitant about moving to Tk-Cocoa to take the plunge.
Thanks, Marc!Fri, 06 Mar 2015
As this roundup at Michael Tsai's blog shows, an increasing number of Mac and iOS developers are having a hard time making a go of indie development. Even the developers of Vesper, considered the best notes app for the iPhone, have been unable to turn it into anything other than a part-time gig, and this is truly shocking considering the talent behind the app: Brent Simmons, a pioneer of indie Mac developers, who took a full-time gig at Omni; John Gruber, one of the leading Mac bloggers (and who, to be fair, likely doesn't need the income, as his blog generates upwards of $450,000 a year in advertising revenue); and Dave Wiskus, a talented app designer/UI specialist.
One culprit is the iOS and Mac App Store's drive for low app prices. As a result, some developers are raising prices. If lowering prices doesn't move the needle on sales, then raising prices certainly is worth a try. It's one reason I've continued to maintain a $30 price point for my own apps. If demand is relatively constant, then higher prices means more revenue.
Still, one thing that many developers do not seem to have admitted: there simply may not be room in the market for so many indie developers. Even if I concede Gus Mueller's cranky point that maybe my apps just suck, and Michael Tsai's observation that "fake is drawing a bunch of pixels into a plain NSView so that it looks like a text field or toolbar" (and to which I readily plead guilty), I think it's still true that the Mac and iOS app markets have changed in fundamental ways. When Brent Simmons has to take a full-time gig, what does that mean for the rest of us?
In other words, I'm afraid that for many of us, "sustainable software" may be less a strategy than a pipe dream.
As I work on updating my apps to take advantage of the improvements to Tk-Cocoa that I've overseen, I realize I am moving in a different direction than I did a few years ago: more Tk, less Cocoa.
What I mean by this is that I am gradually adjusting the balance my apps strike between native elements and aspects that don't hook directly into native API's. This process has been going on for a while, especially with the user interface of my apps; I've moved away from trying to use as many native Cocoa elements as possible at the user level. This gives me more flexibility in making changes when necessary, and also simplifies the development process, because by default Tk is easier to work with than Cocoa. However, I've also started doing this at a lower level in my apps, and also within Tk's internals, as the maintainer of the toolkit on the Mac.
One of the big architectural changes of Tk that I oversaw was removing a lot of Tk's direct use of Cocoa-based widgets, such as NSButton. Tk's drawing model simply can't handle multiple NSView-based widgets in a single window; it functions best when it draws everything itself, within a single NSView. Fortunately there is a Mac API to draw elements with a native appearance, HITheme; this allowed me to hand drawing of these widgets (buttons, scrollbars) over to Tk and still retain the native appearance. The result is more stable and faster drawing. Nonetheless, the balance here is more Tk, less Cocoa.
Another example of this is how I choose to implement Mac-styled features in my apps. In recent years I've gone deep into Cocoa to integrate these elements into my apps. Sometimes that works well, and sometimes it does not; for instance, my use of the 10.7 "fullscreen" API is not very stable, and crashed constantly when I was updating PortAuthority. (It had required a huge amount of code at the Objective-C level to work with Tk, and the result was not very satisfactory.) As a result, I re-worked my "fullscreen" Tk package to hook into Cocoa at only a couple of points--to set the fullscreen button in the window--but to otherwise pass management of the fullscreen transitions to Tk. (In technical terms, I wrote a category on NSWindow that overrides the "toggleFullScreen" method call to emit a Tk event, to which I could bind Tk's standard call to "wm attributes $w -fullscreen.")
Going forward, the question I will ask in going with a Tk-based or Cocoa-based approach will depend on a few factors: how smoothly does the Cocoa feature integrate with my apps, and how robust is a Tk-based alternative? The answer will likely vary on a case-by-case basis. For instance, my "macsheet" package, which displays Tk windows as modal sheet windows, works reasonably well in its current implementation, which uses Objective-C at a low level to set some basic properties, and uses a lot of management at the Tk script level to control the display of the window. It would be possible to implement a pure-Tk approach to the sheet window, but it's not necessary. As an alternative, while I've developed a package that wraps a Cocoa WebView to display HTML content in an app, I'm likely going to keep using a pure-Tk approach: the Tk script code I have works fine, and integrates well into my apps. I use the same approach for my app updating mechanism, which implements a Sparkle-style UI in Tk (while, in fact, hooking into Sparkle's XML appcast format under the hood).
Bottom line: it's sometimes necessary to hook into Cocoa at a low level, and at those times my apps will do so. But Tk offers considerable strengths in terms of simplicity, and at those times I am going to favor Tk.Tue, 03 Mar 2015
I've released version 7.0 of PortAuthority, my GUI for the MacPorts Unix software management system. PortAuthority is the oldest and, these days, only actively-maintained UI client for MacPorts. Its goal is to make using MacPorts easier.
This version of PortAuthority benefits from the many improvements to Tk/Cocoa that I've been working on in recent months, and is generally more responsive and refined. It also restores native system dialogs for running operations with elevated privileges, such as installing a software port. Finally, this release adds support for Mac system notifications if Growl is not installed.
PortAuthority costs $30 for a new installation, and lifetime upgrades are free to registered users. It's worth a try.Mon, 02 Mar 2015
I've released version 1.2 of my "fullscreen" package for Tk-Cocoa.
This package adds native a "fullscreen" button to Tk windows on Mac OS X 10.7 and later. The ::fullscreen::fullscreen command implements a Cocoa-native fullscreen button that, when pressed, will move the window to fullscreen status. The window can be restored to its previous state by clicking the "resize" icon in the application menu. This command is implemented internally by overriding the native Cocoa fullscreen API to generate a "ToggleFullScreen" virtual event, which is then passed to Tcl to map the fullscreen status to the window via the "wm attributes $w -fullscreen" command. This ensures smoother integration between Tk and Cocoa by working with a standard Tk mechanism, but does not work identically to fully-native Cocoa fullscreen implementations. Earlier implementations of the fullscreen attempted to provide a fully-native Cocoa implementation, but it was highly-complex and prone to crashing.
More information: http://opensource.codebykevin.com/native.html#fullscreenFri, 30 Jan 2015
Tk was initially ported to Cocoa in 2009 by Daniel Steffen, who had maintained Tk for several years. Apple sponsored the port of Tk to Cocoa after it deprecated its Carbon frameworks, which had previously been the foundation for Tk. The Cocoa port was a top-to-bottom rewrite, essentially cut from whole cloth rather than a simple update or port. Daniel Steffen had to create an entirely new code architecture for Tk to work with Cocoa, which has a significantly different, higher-level approach to GUI programming than Tk, based on a lower-level model derived from Unix's Xlib library (and which had strong parallels with Carbon). The resulting Cocoa-based Tk was a remarkable piece of design, and brought many advantages in speed and system integration.
After Tk-Cocoa was released in the summer of 2009, Daniel did a major round of updates and bug fixes to smooth out some rough edges in the initial release, and then left Tcl/Tk development to join Apple full-time. A bit later, after releasing some Tcl/Tk extensions that integrated with parts of Cocoa and contributing some patches to the Tk core, I took over as maintainer.
Over the next couple of years, as users gained more experience with Tk-Cocoa, some limitations of the system became apparent. Apart from the standard-issue bugs and glitches that are part of any software project, and which were relatively easy to fix, the differing way Tk and Cocoa handled user input and processing of events caused a variety of issues. Applications would occasionally freeze, and stop responding to user input, because they would become overloaded with events to process. There would be occasional issues with drawing and flickering. These issues were structural, not simply glitches that can be fixed with a few lines of code. The complexity involved would have required a major redesign of Cocoa and Tk's integration, something Daniel Steffen was precluded from doing as an Apple employee (Apple's developers are not allowed to participate in open source projects in most cases); in any case it was not even entirely clear if the integration could be designed differently.
Another issue also arose, not technical but one of policy: Tk-Cocoa, at a deep level, made use of private, undocumented Apple API's to improve drawing performance, using methods borrowed from Apple's open-source WebKit project. This practice, while always discouraged by Apple, took on greater importance when Apple introduced the Mac App Store, which stated that apps making use of private API's would be rejected. This was a serious issues for developers who, like me, wanted to distribute Tcl/Tk apps through the Mac App Store; the only workaround at the time was to link my applications to the somewhat buggy version of Tk-Cocoa that Apple shipped with OS X. (System libraries are allowed to use private API's.)
This was the situation for a couple of years: a lot of low-level frustration with the structural limitations of Tk-Cocoa, no clear path to fixing them, just incremental fixes of bugs, and a handful of Tcl/Tk apps in the Mac App Store linking to an aging version of Tk that did not include any improvements. In the summer of 2014, however, after working with Marc Culler on some patches he submitted to improve how images are rendered in Tk/Cocoa (some OS updates had broken Tk's previous mechanism), I decided to take another look at the deeper issues in Tk.
I found, to my surprise, that Tk would work without the private API calls--so I removed them altogether. I also saw, however, that Tk worked much better with them included. Their removal opened a Pandora's box of bugs and very serious drawing issues: vast amounts of flickering, ghostly images of buttons and scrollbars appearing at random whenever a window was resized or changed in any way. Except for Apple's policy about private API's, which presented a major obstacle to using Tcl/Tk in via its premier distribution channel, I would rather have left the private API's in. They made Tk/Cocoa run much better. But because of Apple's policies, they had to go--and their removal left a huge mess behind.
After doing some more work with Tk without the private API's, the issues started to sort themselves out into some predictable patterns, which made it possible to chart a plan to address them. These patterns included:
The common denominator between these bugs seemed to be that Tk-Cocoa had trouble managing a complex hierarchy of Cocoa views in a window. Tk-Cocoa's design involves using a single NSView in an NSWindow to render the layout of a Tk interface. It handles widgets drawn by Tk that are not directly derived from Cocoa widgets, such as listboxes, just fine. But NSButton and NSScroller widgets (themselves based on NSView) present added complexity that Tk seems unable to manage easily without the private API's.
The solution to these issues focused on reducing the complexity of what Tk-Cocoa had to draw:
The window-resize code I developed on my own, with a lot of trial and error and lots of time on Google. The HITheme code, however, was heavily based on work that others had done in past years--a good thing, because that was a much more complicated task.
The button code, both for regular pushbuttons and popup menubuttons, was based on a patch developed a decade ago by Revar Desmera, a polyglot developer who has done a number of projects in Tcl/Tk. Revar's patch, which added support for the HITheme API to Tk buttons, was never adopted, but it was still sitting in Tk's code tracker and provided the perfect foundation. I had to tweak and update some things, but that work only took a week or two. I am grateful to Revar for his initial work.
The scrolling code was the hardest piece of this puzzle. There was no existing HITheme scrolling implementation for me to tweak, so I had to start from scratch; I worked on the scrolling code for about a month with only modest progress, borrowing some code from Tk's older Carbon implementation and its Unix implementation, which is the foundation for both the Mac and Windows ports of Tk. I got the scrolling code to a point where it would draw the scrollbar, and nothing more. Finally, after rummaging around in Tk's source tree, I found an old patch from Donal Fellows, one of Tcl/Tk's core developers, which fixed a bug in the Mac's other HITheme-derived widgets (scale and progress bar) that seemed suspiciously like the scrollbar issue: the widgets had to apply some kind of multiplier to set their values correctly, and thus redraw. (The math was, and is, over my head.) Once I got that placed into the scrollbar code, things progressed quickly. The final piece of the puzzle was to get the scrollbar to respond to being dragged by the mouse; my initial work here focused on bringing a lot of the old Carbon code back into Cocoa, and updating it somehow. The final solution was beautifully simple and elegant. Based on an observation from Jim Ingham, Tk's original maintainer on the Mac, that the HITheme API would allow Tk/Mac's scrollbar code to become more like its Unix version, that's what I decided to do: throw out all the old Carbon code, import mostly Unix code, and just let the HITheme scrollbar be redrawn by the native API's, not moved and driven by them. Bingo.
I've committed all these updates to Tk's main line of development (trunk) and its still-supported 8.5 version this week, and they are ready to be tested and used. Here is the difference these changes make in Tk-Cocoa:
Let me be clear: Tk-Cocoa remains Cocoa-based. I have left most of Daniel Steffen's brilliant architecture unchanged: Tk's app structure is still based on the NSApp API, it still has the same complex event loop integration, and it still draws into an NSWindow containing an NSView. There still may be bugs owing to the complexity that cannot be changed. However, by making these changes to Tk-Cocoa's drawing code, I have removed some of the complexity that was there before, and brought Tk a bit closer to the level of abstraction it had under Carbon, and retains under both X11 (Unix) and Windows, where it is most performant. As a result, I feel this is a significant evolution of Tk-Cocoa, and an improvement, and I do not feel embarrassed in calling it Tk-Cocoa 2.0.
To all Tk/Mac developers and users out there: I hope you find these changes helpful. And to the developers named here and to those who have provided encouragement and feedback: thank you. While at times I have complained that I work alone, this is not true: I have relied on the contributions of many, both of code and feedback, and I am grateful.