Code by Kevin, Programming, code, business, and other pursuits
Kevin Walzer, software developer.
Subscribe to RSS Feed
Get a syndicated feed of my weblog.
Site design: Skeleton
I've released version 1.1 of TkWebView, my Tcl package that allows a developer to add a native Cocoa WebView/WebKit widget to a Tcl/Tk application, using the same rendering engine that powers Safari.
TkWebView is available under a BSD-style license.Thu, 04 Dec 2014
I'm pleased to announce the initial release of ruby2app, a tool for bundling Ruby scripts into standalone Mac applications:
ruby2app is a simple command-line tool, inspired by similar Mac tools like py2app for Python, that bundles a Ruby script with the Ruby runtime into a standalone Mac application. It reads a basic configuration file to find the appropriate Ruby installation, compiles a small C program that embeds the Ruby interpreter, copies the entire Ruby installation into the application bundle, and re-links the relevant libraries with the main executable. Ruby must be compiled with the "--enable-load-relative" flag for ruby2app to work.
At present the tool is primarily geared for Ruby-Tk applications, since Tk is the best-supported open-source GUI framework with Ruby bindings on the Mac in 2014, and (not coincidentally) is the one I use. But I invite others to submit patches to add support for other UI frameworks as well; Qt support might be a good candidate. Support for RubyMotion is outside the scope of ruby2app, as RubyMotion presumably has its own toolchain.
I am still in the process of learning Ruby, and in fact wrote this tool to be certain that I could deploy a Ruby-Tk application in a convenient manner on the Mac desktop. There are no other current tools that provide basic support for deployment of an open-source Mac application written in Ruby. Ruby-Cocoa and MacRuby are effectively dead and their tools (standaloneify.rb) are obsolete, RubyMotion has its own toolchain, and crate and rubyscript2exe do not support current versions of Ruby.
I do have a decade of experience writing Mac applications in Python, Perl and Tcl using the Tk toolkit, so I adapted some ideas from bundling tools for those languages in creating ruby2app. ruby2app is written in Bash, and thus should be easy to read and modify as required, and should fit in well to any application building toolchain.
ruby2app is available under the MIT license, and feedback is welcome.Wed, 26 Nov 2014
This is a significant announcement because it means that all private API's have been removed from Tcl/Tk, as verified by Apple's own scanning tools, and therefore apps using Tcl/Tk can be submitted to the MAS using up-to-date versions of Tcl/Tk and without linking to Apple's own outdated system installation. As I noted previously, I spent several weeks scrubbing the private API calls, forbidden by Apple, from Tcl/Tk. One reason for the lag between the release of FileMorph 2.0 and its acceptance in the MAS was that there was still a bit of private cruft that I had overlooked, which has since been addressed.
Going forward, I plan to keep my apps as up-to-date as possible with Tcl/Tk, and not link to Apple's libraries any more.Wed, 12 Nov 2014
I've release FileMorph 2.0, an app I created in Tcl/Tk a few years ago to provide batch modification of file names and attributes, such the last date accessed. My intent in creating the program was to offer an easier-to-use tool than other file renaming programs, which tend to overwhelm with their options, and the command-line, which is powerful but obscure. I withdrew FileMorph from sales a year or so ago to focus on some other projects, but decided to return to it this fall as part of my effort to add Perl as a development language to my toolkit: FileMorph is an ideal app to implement in Perl because its functions play right into Perl's strengths, with is text and file management.
I'm pleased with the results overall. Perl has proven to be a pleasant language to develop in, and its bindings to the Tk toolkit (via its Tkx module) and the availability of the Mac::QuickBundle module make the development of a full-featured, GUI-based, desktop app feasible.
There were a few hiccups in the development process that led to it taking longer than expected. First, there was the usual assortment of changes in Apple's development tools and requirements for submitting to the Mac App Store, which caused some delay. Also, there were a few unexpected snags in Perl's interaction with Tcl under the hood: occasionally, when running a Tcl/Tk function from Perl, I would get obscure errors like "sh: -c: line 0: unexpected EOF while looking for matching `''". In Perl code, this typically means that there is a typographical error, a missing quote mark that causes execution of code to stop; but when the error occurs in the context of a Tcl procedure that Perl is calling, the actual issue is anyone's guess. I observed this when Perl (via Tcl) was downloading a file from the Internet, and parsing XML (part of the application update machinery I've implemented). The only solution was to continually tweak the code until the error no longer occurred, even though it wasn't clear to me where the error was.
I'm using this version of FileMorph as the basis for the next generation of my development platform, as well, which means, among other things, that I am now moving from linking to the Mac-installed system Tcl/Tk to shipping my own, more up-to-date versions, stripped of the private Cocoa calls that previously caused rejection by the Mac App Store. This app also includes some UI refinements that better fit in with OS X 10.10.
FileMorph is now ready for sale at my website and is undergoing review at the Mac App Store. If you do batch renaming of files, it's worth a look.Mon, 13 Oct 2014
I've written about my efforts to add Ruby to my desktop development toolkit, particularly to make it feasible to deploy standalone Ruby-Tk apps on the Mac. My efforts with Ruby deployment had paid off with a viable approach, and I was feeling confident about using Ruby as a language for a planned project.
Imagine my frustration, then, when I discovered that Ruby did not support the latest version of Tk, 8.6; numerous developers and users were reporting crashes. While this may be expected when a language like Tcl/Tk does a major new release, Tk 8.6 has been out for nearly two years, since December 2012. It would seem that two years is more than enough time to work through whatever bugs and incompatibilities might arise, but apparently the issues were rather complex.
I haven't yet migrated my Tcl/Tk apps to 8.6, mainly because Tk's use of private frameworks on the Mac prevented me from using a version other than the one Apple ships with the operating system. Now that I have removed that code from Tk, migrating my apps to Tk 8.6 is going to be part of the next major releases of my products. Tcl/Tk 8.5 is likely to reach end-of-life status soon, meaning that it will no longer be supported or updated by the Tcl/Tk developers. As a result, if I want to include Ruby in my projects, it is vital that Ruby support 8.6.
Happily, after I posted a query on the Ruby-Talk mailing list about Tk 8.6, Ruby-Tk's maintainer, Hidetoshi Nagai, committed a patch to Ruby's development branch that adds "experimental" support for Tk 8.6. I rebuilt Ruby against Tk 8.6 and tested it against a complex Ruby-Tk application, and it seemed to run just fine--no crashes. That is terrific news, because it provides a foundation for full support for Tk 8.6 in Ruby, and it means I can add Ruby--a beautiful language with a rich library--to my app toolbox. I am very pleased, and encourage others to give Ruby-Tk 8.6 a try when the next version of Ruby is released. Thanks to Hidetoshi Nagai for being an extremely responsive developer.Wed, 24 Sep 2014
I've been doing a great deal of wrestling lately with a paradox: the impact of including private Apple Application Programming Interfaces (API's) in an open-source project.
For the past few years I've been the chief maintainer of the Tk GUI toolkit on the Mac. I've been a Tcl/Tk developer for a decade, started coding my own platform-native Tcl/Tk extensions in C and Objective-C about five years ago, and took over maintainer duties in 2011 after the author of the Cocoa port of Tk, Daniel Steffen, was hired by Apple. Daniel had been the lead maintainer of Tk on OS X for several years before I assumed the role.
One of the key parts of the Cocoa port of Tk was its use of private, undocumented Cocoa API's for rendering of application windows and widgets; Daniel added this into Tk to help make sure that Tk rendered graphics quickly and smoothly. He borrowed the technique, and likely a lot of code, from WebKit, the Apple-sponsored, open-source HTML framework that powers the Safari, Chrome and other web browsers. This code provides the WebKit developers, and Tk developers, a great deal of precision and low-level control over the drawing of GUI elements, more control than is typically afforded by the Cocoa frameworks. I can't speak for how WebKit works, but Tk's overall architecture requires a lot of low-level control over the layout of a GUI. On Windows, and on the older Mac Carbon frameworks, Tk's design lines up nicely with the design philosophy of those toolkits; it's a very different situation with Cocoa.
There's a pretty significant problem with including private API's: Apple strongly discourages it, with good reason. Private API's are generally considered by Apple to be internal to the operating system, and are subject to change and even removal. It's an inherently unstable situation, and Apple goes beyond simply discouraging the use of such code; any application calling into private API's or frameworks is rejected from the Mac App Store (and also the iOS store for mobile apps). This means that Apple will reject apps that bundle code from WebKit; the same is true for apps that bundle code from Tk-Cocoa. The only workaround is to link to the (often older) frameworks that are shipped with OS X; private API's are OK there because they are installed by Apple.
I've been increasingly uncomfortable with Tk's use of private API's, but it wasn't until this year that I felt knowledgable enough to try to remove them. Eventually, that's what I did. I was tired of not being able to ship up-to-date Tk code in my own applications in the Mac App Store, and more importantly, I felt that it was a bad idea to have an open-source library such as Tk resting on such a fragile foundation.
And now, having stripped out the private API calls, I can see why they were included.
I must be candid: removing the private API's did result in a degradation of Tk's drawing performance under Cocoa. Rendering that was snappy and accurate became glitchy, slow, and laggy, with weird artifacts like buttons scrolling outside their container window and scrollbars appearing in two different places when a window was remapped. Sometimes widgets would not remap after a window resized, and in a few instances too much resizing would cause a crash.
A lot of the work I've done over the past couple of months, in consultation with a couple of contributors, has been to try and mitigate the worst effects of removing the private API calls. The button and scrollbar issues have been fixed, and some improvement in rendering with window resizing and resizing of child windows is now in place. For instance, I've added some code to skip over some drawing operations in window resizing, which has smoothed things out a bit. Basic user interfaces, such as the ones in the Tk demo, render very well with little discernible loss of performance. Stability seems fine.
Unfortunately, there remains some laggy drawing in more complex interfaces. I've spent a lot of hours over the past several weeks becoming acquainted with Tk-Cocoa's drawing code, and I regret that I can't wring out much more improvement here. Those private API's are undocumented and I don't fully understand how they worked or what they actually did, but they did add a lot of low-level magic to drawing Tk widgets. If getting the best performance for graphic rendering is the goal, including those bits is absolutely the right call, and I fully understand why Daniel Steffen included them. If an Apple-sponsored open-source project (WebKit) includes this code, then it's entirely reasonable for another Apple-sponsored open-source project (Tk-Cocoa) to follow suit.
From a policy standpoint, however, such design is untenable, because Apple is so strictly enforcing against the deployment of code that makes use of such design.
It's hard for me to get past the absurdity of Apple's position here. It's simply baffling why one of the largest open-source projects they sponsor--WebKit--violates platform protocols by using private API's, and apps directly bundling such code can't be deployed on the platform's major distribution channel, the Mac App Store. Wouldn't it be better for Apple to open up these private API's, make them public, and allow third-party developers to use them if necessary? WebKit's use of these API's dates back to the earliest days of the project; I found commit messages from 2002 that report their inclusion. Tk would certainly benefit if use of those API's could be made legal. If a platform vendor's private API is used in a vendor-sponsored open-source project, how truly private is the API? If Apple is going to be consistent here, shouldn't WebKit remove these private API calls, and find another way to render browser windows in a smooth, crisp fashion?
Past a certain point, griping is pointless; this is where we are. And I don't want to be too critical of Apple; Apple has greatly benefited the Tcl/Tk community by sponsoring Daniel Steffen's work on the Cocoa port of Tk--I doubt it would have been written otherwise. No other major open-source GUI toolkit was the beneficiary of such largesse. Still, it's a sad situation that complying with Apple's guidelines has resulted in the measurable degradation of the GUI toolkit whose port Apple sponsored. It may be a necessary compromise, but it's hard to be happy about it.Mon, 15 Sep 2014
I've released version 1.2 of my tkmacicon package for Tcl/Tk on Mac OS X.
The tkmacicon package provides the data necessary to platform-native icons on OS X as Tk images. The ::tkmacicon::geticonfrompath command takes three arguments: a file path, width, and height. The ::tkmacicon::geticonfromtype command takes three arguments: a file extension, width, and height. These commands provide the raw image data necessary to render a Mac icon as a native Tk image.
This release represents a significant API change from earlier versions, which wrote the image data to a PNG file and then read the file into a Tk image. Obtaining the data without the overhead of file writes and reads makes the package significantly faster.
For more information, see http://opensource.codebykevin.com/native.html#tkmacicon.
I've released version 1.0 of my progressdock package for Tk on Mac OS X.
The progressdock package draws a nice progress bar over a Tk/Mac application's Dock icon. It is especially useful for showing download/uploads, and similar activities. The package is based on the UKDockProgressIndicator class by Uli Kusterer.
For more information on the progressdock package, see http://opensource.codebykevin.com/native.html#progressdock.Sat, 06 Sep 2014
Over the past few years I've experimented with both Ruby and Perl as development languages for desktop apps on the Mac. Both are robust programming languages, have superb library support, and have excellent bindings for my GUI toolkit of choice. However, neither language is used much on the Mac for development of desktop apps. Their achilles' heel is deployment; in contrast to Python or Tcl/Tk, neither language has a standard, widely-used mechanism for bundling up code into a single standalone Mac application package that can be run without issue.
I've spent a lot of time over the past year or so investigating the deployment issue, periodically pulling out a couple of simple Ruby or Perl projects that I play with as part of the process of learning these languages as desktop app tools. And I'm happy to report that I've made good progress and have prototype apps running as standalone apps on the Mac.
In keeping with Perl's "there's more than one way to do it" spirit, I've actually found several ways to deploy an app. The one I've been working with the most is the Perl Archiving Toolkit and its pp module, which bundles up Perl code into a standalone executable than can then be run on another system. This works quite well, actually, and one could do the extra work of deploying the executable inside a Mac application bundle structure (a Mac app bundle, though it looks like a single file on the Mac, is actually a special directory structure). However, this mode has limitations, particularly with linking to the Tcl/Tk frameworks. The pp module has no knowledge of Mac application structure and no easy way to use standard Mac mechanisms for finding a Tcl/Tk installation bundled with the application. This proved to be a serious, and surprisingly difficult-to-solve, issue.
As a result, I decided to look elsewhere, and eventually found a much better solution in Perl's library ecosystem: Mac::QuckBundle. The QuickBundle library incorporates yet another library, mac-perl-wrapper, and does two things: it creates a proper Mac application structure for the deployment of Perl code, and automates the process so that an app can be built using a single configuration file. With this structure in place, it is much for me easier to do some additional configuration so that Perl can find the correct Tcl/Tk libraries in the application package.
Mac::QuickBundle is a near-perfect tool for my needs here, quite analogous to py2app or cx_Freeze in the Python world. I'm rather surprised I hadn't heard of it before, but I think this is reflective of the fact that Perl does not have a large constituency in the Mac desktop development community. Still, I'm glad the code was out there.
Ruby has proven to be a much more complex language to work with. Despite all my research, I've never found a general-purpose tool for deploying Ruby apps on the Mac. What tools exist are either Windows-only or limited to deep Ruby/Cocoa integration, which doesn't meet my needs. Still, after more work with Ruby and studying how the Perl and Python apps work, I've put together the basic steps to deploy a Ruby/Tk app on the Mac:
Right now my work with Ruby has been distilled into a shell script that creates my builds; I'm not sure if it can be generalized into a general-purpose application builder for Ruby. Ruby still has too many quirks to be as easily deployable as Python or Tcl/Tk, for instance. But I will likely post the script and some documentation at some point in the future, when it's been further tested and I have a releasable app to use with it.
I'm pleased that I've been able to make progress with deployment of Perl and Ruby apps on the Mac; this work will allow me to access the considerable power of Perl and Ruby for new desktop applications.
A few months ago I replaced the popular Sparkle update mechanism for the Mac in my apps with a hand-rolled mechanism that used some of Sparkle's ideas and data format, but which was simpler to configure in my own code. This was motivated by the fact that Sparkle seemed to have fallen into obsolescence, having been unmaintained for a few years.
I'm pleased to report that development on Sparkle has resumed, with a new project page. Sparkle now has a community of developers working on it, and it seems to have received several much-needed updates. While I don't plan to go back to using Sparkle in my own programs, I'm very pleased to see that this vital tool show signs of life on the Mac.Wed, 11 Jun 2014
I've brought TextSweep, my search-and-replace app for the Mac, out of storage and back into active development. A couple of months ago I had announced its discontinuation because it seemed horribly broken: freezing up whenever the user selected a file to display. I blamed insoluble bugs in its underlying GUI framework, Tk, for the problem.
As it turns out, the bug was in my own code, so embarrassingly simple to fix that I still can't believe it. Essentially, if no search term in the application were defined (i.e. if the search field was blank) then the app would lock up; if a search term was defined, it would run as normal. Fixing this is literally one line of code. Unfortunately, finding the source of the bug turned out to be fiendishly hard (I ran across the actual error essentially by accident), and kept the app from being usable for months.
Ah, well. Now that I've found the source of the error, I have released version 2.1 of TextSweep and submitted the same version to the Mac App Store for review and inclusion. I'm still not 100% satisfied with the app and want to refine its features further, but now, at least, it runs. That has to come before anything else.Sun, 04 May 2014 Wed, 16 Apr 2014
I've decided to discontinue my TextSweep search-and-replace app. I've had a large number of problems getting it to run in a stable fashion on Mavericks, it hasn't sold anything this year (owing to its lack of functionality, I suspect), and it has never worked as well as I'd like even if it does launch correctly. Rather than invest time in rewriting it from scratch, I've decided to scrap it. I have released updates to several other apps this month, so I will continue to work on those.Sun, 13 Apr 2014
I've released updates to both PortAuthority and PacketStream. These updates include the improved update mechanism based on Sparkle that I'm adding to each of my apps, and also a workaround for a serious system-level bug in Mavericks.
The bug in Mavericks--which affected these apps but which was not my doing--revolved around running AppleScripts with elevated user privileges, which is a requirement for some of the operations that PortAuthority and PacketStream perform (installing software, listening to network data). I had turned to AppleScript to provide a better, more native implementation of this process than older versions of my apps supported. Unfortunately, the AppleScript bug in the most recent update of Mavericks, 10.9.2, caused the AppleScript to hang; thus, my apps making use of this functionality never completed their operations and never displayed data. In short, they stopped working. Developers call this type of bug a "showstopper," and it's as serious as you can get.
Given that this was Apple's bug rather than something in my own code, my options for dealing with it were limited. I filed a bug report with Apple in hopes that a new update of OS X might resolve the issue, but no action has been taken to address the issue as of yet. I was reluctant to move away from using AppleScript because it provided a much better user experience than my older methods: native dialog, behavior more consistent with other apps, better performance. However, none of that matters if the app doesn't work.
As a result, I've reluctantly re-introduced my older approach to running privileged operations. Instead of running an AppleScript command with administrator privileges, I now presenting a password dialog to the user, then pipe that password to the command line "sudo" tool under the hood. "Sudo" does the same thing as my AppleScript method did, just without the fancy native dialog. Some developers suggest this method is a little less secure than the native method that AppleScript implements. However, even if that's true, it has the virtue of actually working. The new versions of these apps work as expected.
I took the opportunity to simplify the password code a bit, and I've retained the improved method of reading data into the apps that I implemented with the AppleScript approach; thus, the apps haven't lost anything in terms of their performance and responsiveness. (Another reason I went the AppleScript route was that my old password code was sluggish and could cause the apps to lock up at unpredictable times. I don't think that will happen here.)
In any event, I don't plan to go back to using AppleScript. It's highly unlikely that "sudo"--which is a long-stable, core Unix command--will ever run into the same kind of system-level bugs that AppleScript has.
If you've recently downloaded these apps, and wondered why they didn't work, please give the new versions a try.Thu, 10 Apr 2014
I've released a couple of updates to NameFind in the last week. The first update, 7.1, fixed some UI glitches on Mavericks: icon images were diplaying in the UI in a jagged manner. The images also slowed down scrolling in the data view. I ultimately decided to get rid of icons altogether in the table view and instead display them in a separate info window, if the user calls that up.
The second update, 7.2, released today, completely overhauls the app's update mechanism. For several years I had used my own system, but last year I decided to add Sparkle support, the popular framework for app updates that is still used by hundreds of apps that are downloaded from outside the Mac App Store. My own homegrown updating system has had some hard-to-track bugs in recent versions of my apps, and I just decided to abandon it in favor of Sparkle.
Unfortunately, Sparkle itself has proven to be somewhat unstable on Mavericks; it often fails to fully update an app, crashing and requiring a manual relaunch of the updated app before the new version is visible. Just as badly, Sparkle's source code page at Github states that Sparkle is now unmaintained by its developer--which means he is no longer working on it. (I believe he is now an Apple employee, whch precludes work on outside projects, even popular ones.) Unfortunately, because Sparkle is now unmaintained, there is little chance that bugs in it will be addressed going forward.
Given that both my old and new update mechanisms are problematic, how have I decided to proceed? By implementing a hybrid of the old and new.
I like Sparkle's "appcast" approach for broadcasting updates; it requires the developer to upload an XML file to a website, similar to an RSS feed, which other sites can download. (Some Mac software listing sites, such as MacUpdate, use this mechanism to track app releases.) This is better than my old approach, which could not be parsed by other systems. So I opted to keep Sparkle's server-side mechanism, and replace its app-level code with my own. This required me to do two things: fix the bug in my old code that kept updates from installing correctly (it was a stupid, stupid error in the way I archived the app update file for download), and rewrite some of the code to present a user interface similar to Sparkle's, which I also liked.
Thus, this approach is the best of the old and new: it uses an industry-standard mechanism (appcast) to announce app updates, but better integrates with my own apps by implementing a Sparkle-like interface and workflow in my own code.
If you use the Sparkle update in the current release of NameFind, you will see the same old bugs, but going forward, you'll see the new Sparkle-like update approach, which should work a lot better.
I don't regret working with Sparkle; spending time with its internals, even if I couldn't fix them, allowed me to understand it well enough to implement something very similar, and Sparkle's server-side "appcast" mechanism remains intact in my toolbox. The new hybrid approach provides a smooth upgrade experience; I have been remiss in not paying attention as I should have to this part of the user experience of my non-Mac App Store apps. I will continue to focus on this going foward.Mon, 10 Mar 2014
I've released version 5.2 of Manpower, my man page viewer for Mac OS X. This version cleans up some of the user documentation and fixes a couple of recurring UI glitches that I thought I had licked in the last release. Please feel free to give it a try.Tue, 25 Feb 2014
I've released version 5.1 of Manpower, my man page viewer for OS X. This release includes a few bug fixes for UI glitches and rendering on Retina displays.
I've gotten some reports from users that OS X on Mavericks reports that the app is damaged, and advises trashing it. OS X's Gatekeeper library, which sets certain restrictions on apps, apparently thinks Manpower has been tampered with or otherwise corrupted. The app is signed with a valid developer certificate, which should pass through Gatekeeper's restrictions; I'm not certain why it's being flagged.
I have found a workaround. I was able to fix this by changing my Gatekeeper settings to allow apps downloaded from anywhere, and installing in /Applications. After launching Manpower, I then re-enabled more restricted settings in Gatekeeper and Manpower ran fine. So, if you download Manpower and run into this issue, give this workaround a try.
(Note that this workaround will not apply to the Mac App Store version of Manpower, which shouldn't display the issue at all. I've submitted version 5.1 to the MAS and hopefully it will be released soon.)Wed, 15 Jan 2014