Code by Kevin

About
Code by Kevin, Programming, code, business, and other pursuits

Your Host
Kevin Walzer, software developer.



Home

Subscribe to RSS Feed
Get a syndicated feed of my weblog.


Archives
2014
2013
2012
2011
2010
2009
2008
2007
2006

Categories
Business
Software
General

October 2014
Sun Mon Tue Wed Thu Fri Sat
     
 

Privacy Policy

Site design: Skeleton

 

Mon, 13 Oct 2014

Ruby and Tk 8.6

I've written about my efforts to add Ruby to my desktop development toolkit, particularly to make it feasible to deploy standalone Ruby-Tk apps on the Mac. My efforts with Ruby deployment had paid off with a viable approach, and I was feeling confident about using Ruby as a language for a planned project.

Imagine my frustration, then, when I discovered that Ruby did not support the latest version of Tk, 8.6; numerous developers and users were reporting crashes. While this may be expected when a language like Tcl/Tk does a major new release, Tk 8.6 has been out for nearly two years, since December 2012. It would seem that two years is more than enough time to work through whatever bugs and incompatibilities might arise, but apparently the issues were rather complex.

I haven't yet migrated my Tcl/Tk apps to 8.6, mainly because Tk's use of private frameworks on the Mac prevented me from using a version other than the one Apple ships with the operating system. Now that I have removed that code from Tk, migrating my apps to Tk 8.6 is going to be part of the next major releases of my products. Tcl/Tk 8.5 is likely to reach end-of-life status soon, meaning that it will no longer be supported or updated by the Tcl/Tk developers. As a result, if I want to include Ruby in my projects, it is vital that Ruby support 8.6.

Happily, after I posted a query on the Ruby-Talk mailing list about Tk 8.6, Ruby-Tk's maintainer, Hidetoshi Nagai, committed a patch to Ruby's development branch that adds "experimental" support for Tk 8.6. I rebuilt Ruby against Tk 8.6 and tested it against a complex Ruby-Tk application, and it seemed to run just fine--no crashes. That is terrific news, because it provides a foundation for full support for Tk 8.6 in Ruby, and it means I can add Ruby--a beautiful language with a rich library--to my app toolbox. I am very pleased, and encourage others to give Ruby-Tk 8.6 a try when the next version of Ruby is released. Thanks to Hidetoshi Nagai for being an extremely responsive developer.

[/software] permanent link

Wed, 24 Sep 2014

Private API's and open-source projects

I've been doing a great deal of wrestling lately with a paradox: the impact of including private Apple Application Programming Interfaces (API's) in an open-source project.

For the past few years I've been the chief maintainer of the Tk GUI toolkit on the Mac. I've been a Tcl/Tk developer for a decade, started coding my own platform-native Tcl/Tk extensions in C and Objective-C about five years ago, and took over maintainer duties in 2011 after the author of the Cocoa port of Tk, Daniel Steffen, was hired by Apple. Daniel had been the lead maintainer of Tk on OS X for several years before I assumed the role.

One of the key parts of the Cocoa port of Tk was its use of private, undocumented Cocoa API's for rendering of application windows and widgets; Daniel added this into Tk to help make sure that Tk rendered graphics quickly and smoothly. He borrowed the technique, and likely a lot of code, from WebKit, the Apple-sponsored, open-source HTML framework that powers the Safari, Chrome and other web browsers. This code provides the WebKit developers, and Tk developers, a great deal of precision and low-level control over the drawing of GUI elements, more control than is typically afforded by the Cocoa frameworks. I can't speak for how WebKit works, but Tk's overall architecture requires a lot of low-level control over the layout of a GUI. On Windows, and on the older Mac Carbon frameworks, Tk's design lines up nicely with the design philosophy of those toolkits; it's a very different situation with Cocoa.

There's a pretty significant problem with including private API's: Apple strongly discourages it, with good reason. Private API's are generally considered by Apple to be internal to the operating system, and are subject to change and even removal. It's an inherently unstable situation, and Apple goes beyond simply discouraging the use of such code; any application calling into private API's or frameworks is rejected from the Mac App Store (and also the iOS store for mobile apps). This means that Apple will reject apps that bundle code from WebKit; the same is true for apps that bundle code from Tk-Cocoa. The only workaround is to link to the (often older) frameworks that are shipped with OS X; private API's are OK there because they are installed by Apple.

I've been increasingly uncomfortable with Tk's use of private API's, but it wasn't until this year that I felt knowledgable enough to try to remove them. Eventually, that's what I did. I was tired of not being able to ship up-to-date Tk code in my own applications in the Mac App Store, and more importantly, I felt that it was a bad idea to have an open-source library such as Tk resting on such a fragile foundation.

And now, having stripped out the private API calls, I can see why they were included.

I must be candid: removing the private API's did result in a degradation of Tk's drawing performance under Cocoa. Rendering that was snappy and accurate became glitchy, slow, and laggy, with weird artifacts like buttons scrolling outside their container window and scrollbars appearing in two different places when a window was remapped. Sometimes widgets would not remap after a window resized, and in a few instances too much resizing would cause a crash.

A lot of the work I've done over the past couple of months, in consultation with a couple of contributors, has been to try and mitigate the worst effects of removing the private API calls. The button and scrollbar issues have been fixed, and some improvement in rendering with window resizing and resizing of child windows is now in place. For instance, I've added some code to skip over some drawing operations in window resizing, which has smoothed things out a bit. Basic user interfaces, such as the ones in the Tk demo, render very well with little discernible loss of performance. Stability seems fine.

Unfortunately, there remains some laggy drawing in more complex interfaces. I've spent a lot of hours over the past several weeks becoming acquainted with Tk-Cocoa's drawing code, and I regret that I can't wring out much more improvement here. Those private API's are undocumented and I don't fully understand how they worked or what they actually did, but they did add a lot of low-level magic to drawing Tk widgets. If getting the best performance for graphic rendering is the goal, including those bits is absolutely the right call, and I fully understand why Daniel Steffen included them. If an Apple-sponsored open-source project (WebKit) includes this code, then it's entirely reasonable for another Apple-sponsored open-source project (Tk-Cocoa) to follow suit.

From a policy standpoint, however, such design is untenable, because Apple is so strictly enforcing against the deployment of code that makes use of such design.

It's hard for me to get past the absurdity of Apple's position here. It's simply baffling why one of the largest open-source projects they sponsor--WebKit--violates platform protocols by using private API's, and apps directly bundling such code can't be deployed on the platform's major distribution channel, the Mac App Store. Wouldn't it be better for Apple to open up these private API's, make them public, and allow third-party developers to use them if necessary? WebKit's use of these API's dates back to the earliest days of the project; I found commit messages from 2002 that report their inclusion. Tk would certainly benefit if use of those API's could be made legal. If a platform vendor's private API is used in a vendor-sponsored open-source project, how truly private is the API? If Apple is going to be consistent here, shouldn't WebKit remove these private API calls, and find another way to render browser windows in a smooth, crisp fashion?

Past a certain point, griping is pointless; this is where we are. And I don't want to be too critical of Apple; Apple has greatly benefited the Tcl/Tk community by sponsoring Daniel Steffen's work on the Cocoa port of Tk--I doubt it would have been written otherwise. No other major open-source GUI toolkit was the beneficiary of such largesse. Still, it's a sad situation that complying with Apple's guidelines has resulted in the measurable degradation of the GUI toolkit whose port Apple sponsored. It may be a necessary compromise, but it's hard to be happy about it.

[/general] permanent link

Mon, 15 Sep 2014

Tkmacicon 1.2

I've released version 1.2 of my tkmacicon package for Tcl/Tk on Mac OS X.

The tkmacicon package provides the data necessary to platform-native icons on OS X as Tk images. The ::tkmacicon::geticonfrompath command takes three arguments: a file path, width, and height. The ::tkmacicon::geticonfromtype command takes three arguments: a file extension, width, and height. These commands provide the raw image data necessary to render a Mac icon as a native Tk image.

This release represents a significant API change from earlier versions, which wrote the image data to a PNG file and then read the file into a Tk image. Obtaining the data without the overhead of file writes and reads makes the package significantly faster.

For more information, see http://opensource.codebykevin.com/native.html#tkmacicon.

[/software] permanent link

Progressdock

I've released version 1.0 of my progressdock package for Tk on Mac OS X.

The progressdock package draws a nice progress bar over a Tk/Mac application's Dock icon. It is especially useful for showing download/uploads, and similar activities. The package is based on the UKDockProgressIndicator class by Uli Kusterer.

For more information on the progressdock package, see http://opensource.codebykevin.com/native.html#progressdock.

[/software] permanent link

Sat, 06 Sep 2014

Deployment of Perl and Ruby Tk apps on the Mac: Coming along

Over the past few years I've experimented with both Ruby and Perl as development languages for desktop apps on the Mac. Both are robust programming languages, have superb library support, and have excellent bindings for my GUI toolkit of choice. However, neither language is used much on the Mac for development of desktop apps. Their achilles' heel is deployment; in contrast to Python or Tcl/Tk, neither language has a standard, widely-used mechanism for bundling up code into a single standalone Mac application package that can be run without issue.

I've spent a lot of time over the past year or so investigating the deployment issue, periodically pulling out a couple of simple Ruby or Perl projects that I play with as part of the process of learning these languages as desktop app tools. And I'm happy to report that I've made good progress and have prototype apps running as standalone apps on the Mac.

In keeping with Perl's "there's more than one way to do it" spirit, I've actually found several ways to deploy an app. The one I've been working with the most is the Perl Archiving Toolkit and its pp module, which bundles up Perl code into a standalone executable than can then be run on another system. This works quite well, actually, and one could do the extra work of deploying the executable inside a Mac application bundle structure (a Mac app bundle, though it looks like a single file on the Mac, is actually a special directory structure). However, this mode has limitations, particularly with linking to the Tcl/Tk frameworks. The pp module has no knowledge of Mac application structure and no easy way to use standard Mac mechanisms for finding a Tcl/Tk installation bundled with the application. This proved to be a serious, and surprisingly difficult-to-solve, issue.

As a result, I decided to look elsewhere, and eventually found a much better solution in Perl's library ecosystem: Mac::QuckBundle. The QuickBundle library incorporates yet another library, mac-perl-wrapper, and does two things: it creates a proper Mac application structure for the deployment of Perl code, and automates the process so that an app can be built using a single configuration file. With this structure in place, it is much for me easier to do some additional configuration so that Perl can find the correct Tcl/Tk libraries in the application package.

Mac::QuickBundle is a near-perfect tool for my needs here, quite analogous to py2app or cx_Freeze in the Python world. I'm rather surprised I hadn't heard of it before, but I think this is reflective of the fact that Perl does not have a large constituency in the Mac desktop development community. Still, I'm glad the code was out there.

Ruby has proven to be a much more complex language to work with. Despite all my research, I've never found a general-purpose tool for deploying Ruby apps on the Mac. What tools exist are either Windows-only or limited to deep Ruby/Cocoa integration, which doesn't meet my needs. Still, after more work with Ruby and studying how the Perl and Python apps work, I've put together the basic steps to deploy a Ruby/Tk app on the Mac:

  1. Create a small stub launcher in C that embeds Ruby. That seems to be the secret sauce to nearly all the tools I've studied; they start as small applications written in C that then launch an embedded interpreter. This isn't surprising, actually, and simplifies the basic structure of the application package.
  2. Use MacPorts to build Ruby and build Ruby with an "--enable-load-relative" flag. There are dozens of ways to build Ruby on the Mac; it is actually not easy to do. Of all the ways I've tried, MacPorts seems to have the simplest, most complete support for building Ruby. I submitted a patch to MacPorts to allow Ruby to build under a "load-relative" flag, which means that its search path for its core libraries is not hard-coded (the default mode) but rather is encoded to be relative to the main Ruby executable. Building "load-relative" means that the Ruby installation can be relocated to another location, such as an application bundle, and will still work as long as the overall installation structure is preserved. (Otherwise the deployed Ruby will look in a fixed location, such as /opt/local, for all of its supporting code, which won't work if nothing is installed there.)
  3. Use MacPorts's search mechanisms to find the locations of the appropriate libraries and copy them into the application bundle. Ruby does not seem to have any mechanisms for searching and identifying code that an application depends on (Perl has its "scandeps" library and Python has its "modulefinder" library). However, MacPorts itself provides tools for searching out its installations, such as pkg-config, and these tools make it possible to build and link the application launcher, and find and copy over the Ruby installation, seamlessly.

Right now my work with Ruby has been distilled into a shell script that creates my builds; I'm not sure if it can be generalized into a general-purpose application builder for Ruby. Ruby still has too many quirks to be as easily deployable as Python or Tcl/Tk, for instance. But I will likely post the script and some documentation at some point in the future, when it's been further tested and I have a releasable app to use with it.

I'm pleased that I've been able to make progress with deployment of Perl and Ruby apps on the Mac; this work will allow me to access the considerable power of Perl and Ruby for new desktop applications.

[/software] permanent link

Sparkle back in business

A few months ago I replaced the popular Sparkle update mechanism for the Mac in my apps with a hand-rolled mechanism that used some of Sparkle's ideas and data format, but which was simpler to configure in my own code. This was motivated by the fact that Sparkle seemed to have fallen into obsolescence, having been unmaintained for a few years.

I'm pleased to report that development on Sparkle has resumed, with a new project page. Sparkle now has a community of developers working on it, and it seems to have received several much-needed updates. While I don't plan to go back to using Sparkle in my own programs, I'm very pleased to see that this vital tool show signs of life on the Mac.

[/software] permanent link

Wed, 11 Jun 2014

Sweeping back into town

I've brought TextSweep, my search-and-replace app for the Mac, out of storage and back into active development. A couple of months ago I had announced its discontinuation because it seemed horribly broken: freezing up whenever the user selected a file to display. I blamed insoluble bugs in its underlying GUI framework, Tk, for the problem.

As it turns out, the bug was in my own code, so embarrassingly simple to fix that I still can't believe it. Essentially, if no search term in the application were defined (i.e. if the search field was blank) then the app would lock up; if a search term was defined, it would run as normal. Fixing this is literally one line of code. Unfortunately, finding the source of the bug turned out to be fiendishly hard (I ran across the actual error essentially by accident), and kept the app from being usable for months.

Ah, well. Now that I've found the source of the error, I have released version 2.1 of TextSweep and submitted the same version to the Mac App Store for review and inclusion. I'm still not 100% satisfied with the app and want to refine its features further, but now, at least, it runs. That has to come before anything else.

[/software] permanent link

Sun, 04 May 2014

PortAuthority 6.2

I've released version 6.2 of PortAuthority, my GUI for MacPorts. This release fixes a bug in updating ports and includes a couple of minor internal code improvements.

[/software] permanent link

Wed, 16 Apr 2014

Swept away: TextSweep

I've decided to discontinue my TextSweep search-and-replace app. I've had a large number of problems getting it to run in a stable fashion on Mavericks, it hasn't sold anything this year (owing to its lack of functionality, I suspect), and it has never worked as well as I'd like even if it does launch correctly. Rather than invest time in rewriting it from scratch, I've decided to scrap it. I have released updates to several other apps this month, so I will continue to work on those.

[/software] permanent link

Sun, 13 Apr 2014

PortAuthority 6.1, PacketStream 6.1

I've released updates to both PortAuthority and PacketStream. These updates include the improved update mechanism based on Sparkle that I'm adding to each of my apps, and also a workaround for a serious system-level bug in Mavericks.

The bug in Mavericks--which affected these apps but which was not my doing--revolved around running AppleScripts with elevated user privileges, which is a requirement for some of the operations that PortAuthority and PacketStream perform (installing software, listening to network data). I had turned to AppleScript to provide a better, more native implementation of this process than older versions of my apps supported. Unfortunately, the AppleScript bug in the most recent update of Mavericks, 10.9.2, caused the AppleScript to hang; thus, my apps making use of this functionality never completed their operations and never displayed data. In short, they stopped working. Developers call this type of bug a "showstopper," and it's as serious as you can get.

Given that this was Apple's bug rather than something in my own code, my options for dealing with it were limited. I filed a bug report with Apple in hopes that a new update of OS X might resolve the issue, but no action has been taken to address the issue as of yet. I was reluctant to move away from using AppleScript because it provided a much better user experience than my older methods: native dialog, behavior more consistent with other apps, better performance. However, none of that matters if the app doesn't work.

As a result, I've reluctantly re-introduced my older approach to running privileged operations. Instead of running an AppleScript command with administrator privileges, I now presenting a password dialog to the user, then pipe that password to the command line "sudo" tool under the hood. "Sudo" does the same thing as my AppleScript method did, just without the fancy native dialog. Some developers suggest this method is a little less secure than the native method that AppleScript implements. However, even if that's true, it has the virtue of actually working. The new versions of these apps work as expected.

I took the opportunity to simplify the password code a bit, and I've retained the improved method of reading data into the apps that I implemented with the AppleScript approach; thus, the apps haven't lost anything in terms of their performance and responsiveness. (Another reason I went the AppleScript route was that my old password code was sluggish and could cause the apps to lock up at unpredictable times. I don't think that will happen here.)

In any event, I don't plan to go back to using AppleScript. It's highly unlikely that "sudo"--which is a long-stable, core Unix command--will ever run into the same kind of system-level bugs that AppleScript has.

If you've recently downloaded these apps, and wondered why they didn't work, please give the new versions a try.

[/software] permanent link