Code by Kevin

About
Code by Kevin, Programming, code, business, and other pursuits

Your Host
Kevin Walzer, software developer.



Home

Subscribe to RSS Feed
Get a syndicated feed of my weblog.


Archives
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006

Categories
Business
Software
General

December 2017
Sun Mon Tue Wed Thu Fri Sat
         
           

Privacy Policy

Site design: Skeleton

 

Thu, 07 Apr 2016

Deploying Python apps on Windows

In releasing the Python-based QuickWho 6.0 today, for the Windows version I followed the model I used for my Perl-based FileMorph app--I coded a stub executable in C that linked to the Python interpreter and made no effort to wrap everything up in a single "app" executable.

Python has more options for "freezing" or deploying standalone apps than Perl does, but the present state is a bit rough: none of the major libraries for Windows work to my satisfaction. Either they do not support Python 3.5, which has some significant changes over previous versions of Python on Windows, or they lack polish in certain respects in terms of customizing the icon or the version info bundled with the executable. This approach results in a larger distribution than the other approaches, but that's a trade-off I'm willing to make.

The other nice thing about using a C-based executable is that it can be ported to other languages with only minor changes, since all the languages that I use for desktop applications are themselves developed in C, and have a C API. The core of my exe stub does not change; it calls into Windows functions to set the working directory of the program, and requires only minor changes to hook into whatever language I am developing in at that time. This will make future projects on Windows simpler. (For an example, see http://fossil.codebykevin.com/fossil.cgi/quickwho/artifact/f3f87b14df46973c.)

The other thing I am doing with my Windows applications is using a simple installer and uninstaller setup that that includes only minor variations with each application. I use iexpress, a system tool that Microsoft includes with Windows to streamline the installation of software components and libraries. It essentially creates self-extracting archives, that only require a single click before each component is installed where I want. My installer package includes an installer script, an uninstaller script, a zip file of the application package, and a small unzip tool to decompress the archive. There are more complex software installer programs out there, both commercial and open-source, but iexpress is very simple and appeals to my Unix-based background. Developing the installer and uninstaller scripts was a great experience because I learned a great deal about Windows internals, its basic scripting language (batch), and so on.

[/general] permanent link

Fri, 06 Mar 2015

Playing to Tk's strengths: More Tk, less Cocoa

As I work on updating my apps to take advantage of the improvements to Tk-Cocoa that I've overseen, I realize I am moving in a different direction than I did a few years ago: more Tk, less Cocoa.

What I mean by this is that I am gradually adjusting the balance my apps strike between native elements and aspects that don't hook directly into native API's. This process has been going on for a while, especially with the user interface of my apps; I've moved away from trying to use as many native Cocoa elements as possible at the user level. This gives me more flexibility in making changes when necessary, and also simplifies the development process, because by default Tk is easier to work with than Cocoa. However, I've also started doing this at a lower level in my apps, and also within Tk's internals, as the maintainer of the toolkit on the Mac.

One of the big architectural changes of Tk that I oversaw was removing a lot of Tk's direct use of Cocoa-based widgets, such as NSButton. Tk's drawing model simply can't handle multiple NSView-based widgets in a single window; it functions best when it draws everything itself, within a single NSView. Fortunately there is a Mac API to draw elements with a native appearance, HITheme; this allowed me to hand drawing of these widgets (buttons, scrollbars) over to Tk and still retain the native appearance. The result is more stable and faster drawing. Nonetheless, the balance here is more Tk, less Cocoa.

Another example of this is how I choose to implement Mac-styled features in my apps. In recent years I've gone deep into Cocoa to integrate these elements into my apps. Sometimes that works well, and sometimes it does not; for instance, my use of the 10.7 "fullscreen" API is not very stable, and crashed constantly when I was updating PortAuthority. (It had required a huge amount of code at the Objective-C level to work with Tk, and the result was not very satisfactory.) As a result, I re-worked my "fullscreen" Tk package to hook into Cocoa at only a couple of points--to set the fullscreen button in the window--but to otherwise pass management of the fullscreen transitions to Tk. (In technical terms, I wrote a category on NSWindow that overrides the "toggleFullScreen" method call to emit a Tk event, to which I could bind Tk's standard call to "wm attributes $w -fullscreen.")

Going forward, the question I will ask in going with a Tk-based or Cocoa-based approach will depend on a few factors: how smoothly does the Cocoa feature integrate with my apps, and how robust is a Tk-based alternative? The answer will likely vary on a case-by-case basis. For instance, my "macsheet" package, which displays Tk windows as modal sheet windows, works reasonably well in its current implementation, which uses Objective-C at a low level to set some basic properties, and uses a lot of management at the Tk script level to control the display of the window. It would be possible to implement a pure-Tk approach to the sheet window, but it's not necessary. As an alternative, while I've developed a package that wraps a Cocoa WebView to display HTML content in an app, I'm likely going to keep using a pure-Tk approach: the Tk script code I have works fine, and integrates well into my apps. I use the same approach for my app updating mechanism, which implements a Sparkle-style UI in Tk (while, in fact, hooking into Sparkle's XML appcast format under the hood).

Bottom line: it's sometimes necessary to hook into Cocoa at a low level, and at those times my apps will do so. But Tk offers considerable strengths in terms of simplicity, and at those times I am going to favor Tk.

[/general] permanent link

Wed, 24 Sep 2014

Private API's and open-source projects

I've been doing a great deal of wrestling lately with a paradox: the impact of including private Apple Application Programming Interfaces (API's) in an open-source project.

For the past few years I've been the chief maintainer of the Tk GUI toolkit on the Mac. I've been a Tcl/Tk developer for a decade, started coding my own platform-native Tcl/Tk extensions in C and Objective-C about five years ago, and took over maintainer duties in 2011 after the author of the Cocoa port of Tk, Daniel Steffen, was hired by Apple. Daniel had been the lead maintainer of Tk on OS X for several years before I assumed the role.

One of the key parts of the Cocoa port of Tk was its use of private, undocumented Cocoa API's for rendering of application windows and widgets; Daniel added this into Tk to help make sure that Tk rendered graphics quickly and smoothly. He borrowed the technique, and likely a lot of code, from WebKit, the Apple-sponsored, open-source HTML framework that powers the Safari, Chrome and other web browsers. This code provides the WebKit developers, and Tk developers, a great deal of precision and low-level control over the drawing of GUI elements, more control than is typically afforded by the Cocoa frameworks. I can't speak for how WebKit works, but Tk's overall architecture requires a lot of low-level control over the layout of a GUI. On Windows, and on the older Mac Carbon frameworks, Tk's design lines up nicely with the design philosophy of those toolkits; it's a very different situation with Cocoa.

There's a pretty significant problem with including private API's: Apple strongly discourages it, with good reason. Private API's are generally considered by Apple to be internal to the operating system, and are subject to change and even removal. It's an inherently unstable situation, and Apple goes beyond simply discouraging the use of such code; any application calling into private API's or frameworks is rejected from the Mac App Store (and also the iOS store for mobile apps). This means that Apple will reject apps that bundle code from WebKit; the same is true for apps that bundle code from Tk-Cocoa. The only workaround is to link to the (often older) frameworks that are shipped with OS X; private API's are OK there because they are installed by Apple.

I've been increasingly uncomfortable with Tk's use of private API's, but it wasn't until this year that I felt knowledgable enough to try to remove them. Eventually, that's what I did. I was tired of not being able to ship up-to-date Tk code in my own applications in the Mac App Store, and more importantly, I felt that it was a bad idea to have an open-source library such as Tk resting on such a fragile foundation.

And now, having stripped out the private API calls, I can see why they were included.

I must be candid: removing the private API's did result in a degradation of Tk's drawing performance under Cocoa. Rendering that was snappy and accurate became glitchy, slow, and laggy, with weird artifacts like buttons scrolling outside their container window and scrollbars appearing in two different places when a window was remapped. Sometimes widgets would not remap after a window resized, and in a few instances too much resizing would cause a crash.

A lot of the work I've done over the past couple of months, in consultation with a couple of contributors, has been to try and mitigate the worst effects of removing the private API calls. The button and scrollbar issues have been fixed, and some improvement in rendering with window resizing and resizing of child windows is now in place. For instance, I've added some code to skip over some drawing operations in window resizing, which has smoothed things out a bit. Basic user interfaces, such as the ones in the Tk demo, render very well with little discernible loss of performance. Stability seems fine.

Unfortunately, there remains some laggy drawing in more complex interfaces. I've spent a lot of hours over the past several weeks becoming acquainted with Tk-Cocoa's drawing code, and I regret that I can't wring out much more improvement here. Those private API's are undocumented and I don't fully understand how they worked or what they actually did, but they did add a lot of low-level magic to drawing Tk widgets. If getting the best performance for graphic rendering is the goal, including those bits is absolutely the right call, and I fully understand why Daniel Steffen included them. If an Apple-sponsored open-source project (WebKit) includes this code, then it's entirely reasonable for another Apple-sponsored open-source project (Tk-Cocoa) to follow suit.

From a policy standpoint, however, such design is untenable, because Apple is so strictly enforcing against the deployment of code that makes use of such design.

It's hard for me to get past the absurdity of Apple's position here. It's simply baffling why one of the largest open-source projects they sponsor--WebKit--violates platform protocols by using private API's, and apps directly bundling such code can't be deployed on the platform's major distribution channel, the Mac App Store. Wouldn't it be better for Apple to open up these private API's, make them public, and allow third-party developers to use them if necessary? WebKit's use of these API's dates back to the earliest days of the project; I found commit messages from 2002 that report their inclusion. Tk would certainly benefit if use of those API's could be made legal. If a platform vendor's private API is used in a vendor-sponsored open-source project, how truly private is the API? If Apple is going to be consistent here, shouldn't WebKit remove these private API calls, and find another way to render browser windows in a smooth, crisp fashion?

Past a certain point, griping is pointless; this is where we are. And I don't want to be too critical of Apple; Apple has greatly benefited the Tcl/Tk community by sponsoring Daniel Steffen's work on the Cocoa port of Tk--I doubt it would have been written otherwise. No other major open-source GUI toolkit was the beneficiary of such largesse. Still, it's a sad situation that complying with Apple's guidelines has resulted in the measurable degradation of the GUI toolkit whose port Apple sponsored. It may be a necessary compromise, but it's hard to be happy about it.

[/general] permanent link

Thu, 31 Oct 2013

Mavericks Server

I've spent the last few days updating and tweaking my server installation, which happens to be OS X Server running on an iMac in my office. I've posted some observations about Mavericks Server here. Brief summary: Mavericks Server is a powerful, inexpensive, easy-to-use server platform that, in its current incarnation, offers the smoothest upgrade experience of any Mac server I've ever installed. Check out the link for more details.

[/general] permanent link

Thu, 29 Aug 2013

Mission creep

Maintaining and updating several applications at once isn't a simple process, especially if you don't have as much time as you'd prefer. After putting out point updates of a couple of my apps that implemented some of the work I've been doing this year, and planning to roll out similar incremental releases of my other apps, I've decided to hold off on any further incremental releases and just update everything in one big bang. This is definitely a case of mission creep, as my plans are now evolving in the direction of a complete user interface (UI) refresh of my apps, many revisions in their under-the-hood features, and revisions of their branding, but I think it's the best course to pursue.

In general terms, here's what the changes portend for my apps:

Why all these changes? Declining sales are one reason; I'm not satisfied with the way my apps have sold in recent years and I think it's time to try some major changes. But, apart from that, I also want to put the apps on a platform for continued evolution. That's a necessary task. I think they've gone as far as they can go in their current configuration and more than minor tweaking is required to get them ready for their next five years of life. That's the larger purpose of this project, even if sales don't revive to a dramatically higher level.

I realize that until I release them, these changes are just vaporware. My goal is to have them out after the new year, but time will tell.

[/general] permanent link

Fri, 12 Apr 2013

Perl isn't a pearl

Most of my work with scripting languages has been with Tcl and Python. Both are powerful languages, work well on OS X, have nice support for the Tk GUI toolkit, and well-defined libraries for deploying desktop apps in an easy-to-install fashion for end users.

Partly out of a desire for a new challenge, and partly to access a different range of capabilities than Python and Tcl offer, I've tried two different languages: Ruby and Perl. Ruby, while a powerful language and with excellent Tk bindings, proved to be a disappointment for reasons outlined here: poor desktop deployment tools for OS X. In that same blog entry, I added that I would be giving Perl a try next.

My time with Perl has been less of a disappointment than Ruby--I enjoy the language and have made it a part of my software business, particularly with enhancing my websites and providing the server side of one of my iPhone apps. I don't regret learning the language. But I've found Perl to be less effective as a desktop development language.

Perl's support for the Tk framework, which is one reason I decided to try it, is quirky and less robust than I expected. Perl's native support for Tk is primarily maintained by ActiveState, an excellent software company that provides commercial support for many scripting languages and which earns a good deal of income from its developer tools. While ActiveState's Perl Tkx module is open-source, it seems to work best with ActiveState's own proprietary tools; it has less of a constituency among developers who use Perl's open-source deployment tools, such as pp. Trying to wrap simple Perl scripts, I've had unexpected results (such as the script linking to the first version of Tk it finds on my system, even though it wasn't built against that version; or crashing unexpectedly when it finds a different version of Tk). Unexpected results aren't necessarily a surprise, but significantly, I've also been able to find little online discussion of the issues I've found; my queries to mailing lists on some of these issues have gone unanswered. That's a bad sign.

As a result, I have decided to stick with my current development languages, Tcl and Python; both have better community support for desktop Mac deployment, and troubleshooting, than either Perl or Ruby.

[/general] permanent link

Fri, 15 Mar 2013

The update treadmill

After spending a few months working on mobile apps, I am now back working on Mac desktop apps in earnest. The challenge when working on multiple apps, as I do, is how to prioritize updates in such a way that I can actually ship releases at semi-regular intervals.

It's easy to develop a long laundry list of new things to add to an app--it's hard to exhaust that list. Even if an app is fairly mature, it's easy to identify small improvements to make or bugs to fix. However, limiting the scope of such changes is necessary to keep the update treadmill in check; some changes can be deferred until a later release.

Historically, in my apps I've tended to alternate between app-specific changes and improvements--new features for a single app--and more general framework improvements, which can be implemented across all my apps. A rewrite of an app's underlying engine for improved speed is an app-specific improvement. Implementing a Cocoa-based toolbar integrated from Tk is a framework improvement that can be rolled into every app.

At present I'm working more in the framework mode. I have a large list of app-specific improvements to add, but I would like to add some of the framework features first--these will improve all of my apps in meaningful ways. Shifting afterward to more app-specific features and improvements will allow for more than a single release per year. Looking back last year, I see that I tended to combine both library and app-specific features into larger, monolithic releases, and that reduced the frequency of my app updates. Making more regular releases seems to be a better way to keep your apps in the public eye. So, that's what I plan to do this year.

[/general] permanent link

Sun, 05 Aug 2012

Back to the high-res drawing board

Well, both new apps that I submitted to the Mac App Store last week were rejected, both for minor issues. However, as Apple's guidelines are a continually moving target, I am now not able to re-submit them because they lack a high-resolution application icon optimized for the Mac's high-resolution Retina Display. So that means an as-yet-undetermined amount of work to get the apps looking crisp and clean on a high-resolution display. I'll keep you posted on how it's going...

[/general] permanent link

Fri, 11 May 2012

Scripting languages for desktop apps on OS X: Why not?

I recently ran across a blog entry by Chris Adamson, in which he discounted the idea of a Ruby-based development environment for iPhone, and it got a bit of steam coming out of my ears. I have no special interest in using Ruby to develop iPhone apps, but the basis of his reasoning grates in my craw: a major effort by Apple to make scripting languages a "first-class environment for Mac development" was a bust, and therefore, scripting languages aren't suitable for application development. Here's a quote:

The reason I can say that is that Ruby was already tried and rejected as a first-class language for Mac development. In Leopard, Apple made Ruby and Python first-class languages for Cocoa development, creating Cocoa bindings for those languages and fully supporting them with Cocoa-Ruby and Cocoa-Python project templates in Xcode. Tutorials were posted... code was open-sourced... developer sessions were presented... And yet, there's no indication that there was any significant developer up-take. The Ruby and Python project templates disappeared in 10.6s version of Xcode, a seemingly quick admission of defeat, or at least irrelevance.

And this:

I'm not going to say that the C-based languages are ideal for desktop and mobile development, but the fact that they haven't been seriously challenged by the scripting languages, despite all the interest in and shared knowledge about scripting languages, makes me think that that they're the best choice we have.

To which I reply: the best choice for whom?

This circular reasoning--scripting languages with Cocoa bridges haven't become popular, therefore they are lousy choices for desktop development on OS X--grates in my craw. His corresponding assertion--that because scripting languages are unpopular, therefore C-based languages are the best option--is even more dubious. Speaking as the developer of a half-dozen commercial desktop apps, all in scripting languages, I find the suggestion that scripting languages are unsuitable for desktop development to be not only absurd, but patently false.

First, Mr. Adamson, popularity is not a measure of any intrinsic value. It simply measures the quantity of individuals who have a favorable opinion of something. No one should make development decisions using the logic of the Facebook "like" button.

Second, Mr. Adamson, your summary of Apple's push to make scripting languages first-class development environments is superficial. I'm not an expert on Ruby, but I can tell you that PyObjC (the Python-Cocoa bindings that Apple briefly supported in Xcode) existed long before the release of Leopard, and still exists even following the removal of the Cocoa-Python Xcode templates in Snow Leopard. PyObjC has been continuously maintained since the days of NeXT. Its developer community is small, but those developers are releasing successful commercial and open-source applications based on it.

Finally, Mr. Adamson, you appear fundamentally unsympathetic to the viewpoint that scripting languages can be used to develop desktop applications. Outside of arguing that scripting languages are unsuitable because they are unpopular, the only technical point you make is by referring to the "inescapable overhead" of a scripting language--i.e, that they are not as fast as a compiled language. But mostly you speak of popularity contests:

It is not plausible to think that lots of developers haven't already tried to use scripting languages for desktop apps. Surely they must have, and the results haven't yet been compelling enough to dislodge the compiled C-based languages.

I would challenge the statement that lots of developers have tried scripting languages for desktop apps. A sympathetic argument suggests that Apple has touted Objective-C/Cocoa for a long time, most developers have become competent with Cocoa, and are disinclined to try anything else. A cynical argument would point to snobbery on the part of some Cocoa developers, that they believe scripting languages to be developer toys and unsuited for serious development. (Mr. Adamson, I think you are guilty of this to a degree.) Regardless, a two-year effort to promote Ruby and Python as alternatives to Objective-C for Cocoa development isn't going to have a huge effect on the installed base of Cocoa developers.

I'd also like to make an alternative case for scripting languages, outside of the context of scripting language bridges to Cocoa. My preferred GUI toolkit is Tk, which is based on the Tcl scripting language, and which has bindings to a number of other scripting languages, including Python and Ruby. Tk runs on OS X, Windows, and various flavors of Unix, and it is the only major GUI toolkit that is optimized for scripting languages instead of being based on a C/C++/Objective-C framework.

I first gravitated to Tk as a beginning developer, about a decade ago. At the time, Cocoa seemed intimidating, but learning Tk was dead simple. The canonical beginner's exercise in Tk is this simple line of code:

pack [button .b -text "Hello" -command {tk_messageBox -message "Hello world!"}]

This code, saved in a script file or executed in an interactive session with the Tcl/Tk interpreter, draws a button that, when pressed, pops up a dialog saying "Hello world!"

This friendly, low-friction way of programming--quickly specifying a UI in code and testing it--led me to develop more complex programs fairly quickly. I began programming with Tcl/Tk in 2003, and a year later, I released the first alpha version of PortAuthority, my GUI for the MacPorts software installation project. Even with my relative inexperience, PortAuthority gained a user base because MacPorts at the time lacked any kind of GUI tool, and it has remained the leading GUI tool for MacPorts (and my best-selling program) ever since. Along the way, I also developed some skills with Python, and began releasing Python-Tk programs as well. Over the years I have developed a file search application; a GUI for another software installation system, Fink; a browser for man page documentation; a domain search tool; and a GUI for a command-line network monitoring tool. And I have other applications under development as well.

Tk's appeal reflects the appeal of scripting languages in general: the rapid development cycle, the ability to achieve great power in fewer lines of code, and the lower learning curve. Further information about Tk can be found at a superb documentation site by Mark Roseman, TkDocs, and the main Tcl/Tk website.

This brings me to the biggest objection against Tk: because it is cross-platform, it is not optimized for the Mac, and provides a poor user experience as a result. It looks somewhat out of place, and does not hook into a great deal of system-native functionality that is provided by Cocoa applications. That is a fair criticism, and the answer to it is to find ways to integrate Tk more fully into the Mac. I have attempted to do this by learning a good deal of Objective-C and writing various libraries to integrate Tk and the Mac, to the point where I am now one of the core maintainers of Tk on the Mac.

Even though I prefer developing with scripting languages, being a maintainer of Tk on the Mac has forced me to become competent in a number of Cocoa APIs and design patterns, as well as C and Objective-C. Apart from having a basic knowledge of NSView, NSWindow and other APIs for maintaining Tk's Cocoa port, I've written Tcl/Tk extension packages that hook into the NSServices API; drag and drop; printing; icon display and manipulation; and more. (Browse the Subversion archive at http://tk-components.sourceforge.net for examples of my Objective-C code.) Nonetheless, although I can find my way around these APIs, I have no interest in using them for application development. My view is that Objective-C is a low-level systems programming language, necessary for accessing native APIs and necessary at times for raw speed; in these respects it is very powerful. But embracing the entire Cocoa application development gestalt, writing applications in a complicated low-level compiled language and trusting my entire development process to Xcode and Interface Builder, is uncomfortable for me. Xcode and C/Objective-C are heavy, complex, and cumbersome, when my development practices stress lightweight, simple, and agile practices: write my code in a text editor, run my code in the Terminal.

I suspect that my long experience in writing Tk GUI's in code has made it harder for me to embrace the Xcode approach. Most of the main code packages of my applications don't exceed 2,000 lines of code, and the no-compliation nature of coding in a scripting language means that I don't have long delays in trying out different ideas, then waiting for the application to compile again. If your experience with GUI development is coding UI's in C++ or C, then perhaps the Xcode/Interface Builder approach is a big win. (One of the reasons Tk become so popular on Unix in the 1990s was just this--the prevailing method of coding UI's, using the Motif C API, was extremely painful, or so I'm told.)

To bring this back to the main point of my rant, and the suggestion that lack of popularity signifies that scripting languages are unsuitable for desktop application development, let me conclude with a challenge: Do you want proof that scripting languages can be used to develop desktop apps on the Mac? Just download one of mine, and see.

[/general] permanent link

Sun, 29 Apr 2012

QuickWho updates in the works

I've submitted new versions of QuickWho for Mac OS X and QuickWho for iPhone to Apple for updates in their respective app stores. I'm hopeful these will be approved and released in the near future.

It's been an interesting experience working on these releases, and I wanted to share some observations. First, I've moved to a unified development and release cycle for the application, so that the iPhone and desktop versions will remain in sync. Improvements in one version will also make their way into the other version. That was the case with this release: the code that searches domain information, which is written in Python, underwent a significant improvement, and I was able to incorporate this into the desktop code (for the OS X version) and the server-side code (for the iOS version). While the Mac and the iPhone are different beasts, I want an application developed for one to offer substantially the same functionality as the other; that, to me, is the real value of a mobile app.

I continue to learn as I go in the realm of mobile development, and as I gain more experience, I'm developing a set of best practices that will guide me going forward. Part of the process of updating QuickWho for the iPhone was updating its "shell," the PhoneGap (now called Cordova) application wrapper. I don't automatically update libraries because integrating an update can be a lot of work, but this update fixed some critical bugs in the library, so the update was necessary. Working with PhoneGap also isn't fun because it is tightly tied to Apple's Xcode IDE; I prefer to do most of my development with a plain text editor, the Terminal for running commands, and Safari for testing my web code. In desktop development, I find that this approach is more lightweight and flexible. I'm finding the same to be true for mobile development, although I also understand that this approach for mobile development means that I am deploying a less-native application than the PhoneGap shell, which gives access to many native API's on the phone, allows. For the apps I'm developing, this is sufficient.

Going forward, I'm looking forward to continuing a round of updates on my desktop apps, and developing mobile versions of them when it makes sense to do so. I think this will offer more value to my customers.

[/general] permanent link