Monday, June 19, 2006

Blog Relocation

When I created this blog I was totally new to blogging. Blogger.com was a great way to get started. I could have set up something like WordPress or Movable Type on my own server, but I was concerned about the security of the server and the amount of effort required for a proper setup. Google for "wordpress exploit" and you'll get over eight thousand hits. It's certainly possible to do it right, but it's not a job to take lightly. Given that this was just an experiment with no long term plans, I wanted to focus on content and not get lost in the tools.

Though I haven't been able to write nearly as much as I want to, I do like having the blog available on the rare occasion when I do get to write up my thoughts. So given that I am going to keep my blog around long term, it's time to re-evaluate where I keep it hosted. I still think Blogger.com is a great tool, but I would really like something where I have a little more control. And though it's extremely unlikely to happen in the short term, who knows where Blogger will be ten years from now. If my writings are in my database on my server in an open format, I know I'll always have complete and unrestricted access to them.

I still have concerns about traditional tools like WordPress, but the advent of new web application frameworks like Ruby on Rails and Turbogears is leading to lighter (the heavy lifting is done by the framework) and more robust web applications. One that caught my eye is Typo. Typo is a Rails app that implements a simple yet capable blog. I've found that it's easy to install and even easier to use. I had it up and running on my own server in a couple of hours. Typo looks good and offers Markdown support (similar to reStructuredText) which I find very appealing.

As a result, this will be my last post here on Blogger.com. From now on all new posts will be on my own site:

Hope to see you there!

Friday, October 14, 2005

Zaurus pdaXrom IR Keyboard Drivers (part 7)

(continued from part 6)

Lua rocks! My little experiment to embed Lua in Zkbdd has been a huge success. I had a bit of a learning curve to climb on the API to access the Lua objects from C, but once I got the hang of it the code fell into place quickly. It's now possible for me to fully define a keyboard's behavior in a Lua script and have Zkbdd load that script at run time and execute that definition.

What that means to users is that the mess we had with several different builds of IRK for Qtopia with each supporting different keyboards and different ROMs can be completely avoided on pdaXrom. What's more is that the keyboard drivers themselves will work on any ROM that has a port of Zkbdd without any porting or rebuilding. Last but not least, motivated users can write new keyboard drivers without any development tools at all (beyond the text editor of their choice).

The "price" for all this flexibility? Pretty low. Zkbdd did get bigger, but it's still under 90KB which seems reasonable to me. Performance, which I was quite concerned about, is outstanding. Lua has an option to forgo floating point support and build an integer only version. My empirical observations are unable to see any difference between this and the straight C version.

I still have to clean up the new code a bit, write Lua definitions for the Targus and Palm keyboards (Belkin and Pocketop are already working), and write some documentation for folks wanting to create or edit drivers for other keyboards. Lots of work left to do but I think the results will be worth it.

Zaurus pdaXrom IR Keyboard Drivers (part 6)

(continued from part 5)

I've released an "alpha" version of Zkbdd for devices using the 2.4.18 pdaXrom kernel (C7X0, C860, 6000). The release notice on OESF has the details. Initial feedback is positive, so I need to look into getting modules built for the 2.4.20 kernel devices (C1000, C3000, C3100).

However, before I do that I'm tempted to implement a major change to how the keyboard specific portions are implemented. Currently, each supported keyboard has a source file that defines scancode mapping tables and a single function that implements the all of the keyboard handling logic. All these are compiled and linked into zkbdd at build time. However, only a tiny part of the keyboard handling logic is actually keyboard specific. That's the part that turns a sequence of bytes received on the IR port into a scancode. I'd like to have a way to define that logic that can be loaded at run-time and doesn't have to be compiled into a binary executable or library. Sounds like a job for a scripting language!

I'm a big fan of scripting languages. I've used Perl, Python, Lisp, etc. over the years with great success. Would it be possible to embed such a language in Zkbdd so that the keyboard specific portions of the driver could be scripted and supplied at run-time as a text file? Hmm, writing device drivers in a scripting language -- if that doesn't get me branded a "heretic" nothing will!

Now the "big" languages (Perl, Python, and Lisp) are probably total overkill for this application. We're talking about scripts that are maybe a dozen lines long. It's tough to justify growing the Zkbdd binary several hundred KB to support that. After all, the target here is a PDA, not a PC. I figured Forth is probably compact/capable enough to work here. I also thought that Ruby might be a candidate but I knew next to nothing about it so I set off to do some research. In the process I stumbled upon Lua. Lua fits the requirements almost perfectly! It's amazingly small (about 64KB), very easy to embed in a C program, and has a simple conventional syntax that can be learned in minutes (at least well enough to do a tiny keboard driver script). The only thing missing was bitwise operators and it turns out there's a Lua source patch to add those in.

Of course what looks good on paper, doesn't always work as well in the real world. At this point I need to hack out a proof-of-concept and put it to the test. I'll do that while I await more feedback on the alpha release -- the results should be quite interesting!

(part 7)

Tuesday, October 11, 2005

Zaurus pdaXrom IR Keyboard Drivers (part 5)

(continued from part 4)

It's been an insanely busy week and it's only Tuesday! But I am still on track for a "Zkbdd" alpha release by the end of the week. I have deviated so much from the original "Kbdd" code base that I'm going to rename this project to avoid confusion. Because of the Zaurus specific re-architecting, there's no way my changes could ever be rolled back into Kbdd. Better to just declare this an independent fork of Kbdd. I do wish I had a catchier name ... the name Zkbdd looks like output from a broken keyboard driver :)

I've split the code up into one source file per keyboard type plus one source file for the main daemon and support functions. Adding a driver for a new keyboard type will be pretty trivial for even a green C programmer. Simply create a new driver source file that mimics the design of an existing driver, add two lines of code to the "main" file, and rebuild.

I coded up "alpha" drivers for Targus and Palm IR keyboards, but with neither of tho
se keyboards to test with it will be interesting to see how close they are to being correct. I also added a "test9600" driver that simply opens the IR port at 9600 bps (the rate most IR keyboards use) and then displays every scancode it receives. It doesn't try to actually generate any keystrokes but it will be helpful for determining key maps and protocols for unsupported (or incorrectly supported) keyboards.

(part 6)

Sunday, October 09, 2005

Zaurus pdaXrom IR Keyboard Drivers (part 4)

(continued from part 3)

Just a brief status update in the ongoing IR keyboard driver saga ...

Pocketop IR keyboard support is now coded up and working. The extra "mode" keys (punct, num, func) needed a bit of extra logic, but nothing terribly difficult. I have keymaps and protocol descriptions for the Targus and Palm IR keyboards. My plan is to code up support for the Targus and Palm and then do a limited alpha release so folks who actually have those keyboards can tell me how bad my support for them is broken :)

It looks like the initial version is going to be "monolithic" where the only way to add new drivers is to write them in C and compile them into the kbdd application. Long term it seems like it should be possible to provide some kind of run-time description of the keyboard's protocol and mapping and have a driver automatically generated. However there are plenty of folks who could benefit from this software as it stands right now, so I don't want to let feature creep stand in the way of an initial release.

With a little luck I will have IPK files ready for alpha testing by the end of the week.

(part 5)

Saturday, October 08, 2005

Zaurus pdaXrom IR Keyboard Drivers (part 3)

(continued from part 2)

So if I want to avoid juggling for different key maps, its back to having kbdd convert the IR keyboard scancodes into the system scancodes. That does have the additional advantage that I'll be able to use the IR keyboard and built-in keyboard at the same time because I haven't changed the keymaps that the built-in keyboard is relying on. But the existing kbdd scancode tables are no good because they convert to a custom iPaq keymap that doesn't even come close to what the Zaurus built-in keyboard uses.

Modifying the conversion tables seems simple enough at first. For the most part each key on the IR keyboard has a matching key on the Zaurus. When you start to look at combinational keys (eg. shift + something) however, things get more complicated. If I convert shift + 2 on the IR keyboard to shift + 2 on the Zarus keyboard, I'll get a double quote. But what I want is an @ because that's what's marked on the 2 key on my IR keyboard. Why can't anything ever be simple!

Digging through the Zaurus keymaps burried in Sharp's Linux kernel source reveals that nearly all of the "symbols" on the Zaurus that you get using shift plus another key or FN plus another key actually have their own scancodes. So the solution is to hide the IR keyboard's shift key from the Zaurus. I modified kbdd to keep track of the IR keyboard's shift key internally and use that information in the scancode conversion. If the IR keyboard shift plus key combination results in a symbol that can be sent to the Zaurus with a single scancode, I send just that code and the Zaurus never sees the shift. In the cases where I really do want the Zaurus to see the shift(eg. shifted letter keys), I send a shift down, key down, key up,
shift up sequence.

Almost there! There are a couple of special cases where shift plus key needs to turn into FN plus key (such as {}) but those are easy enough to special case in the driver. My initial testing in a console login revealed that I had a few lookups wrong, but with those corrected I can now get every key on the keyboard to generate the right character on the Zaurus. For the real moment of truth, I do a startx, open Abiword and start typing ...

Success! All the character keys on the IR keyboard are doing the right thing in both the console and X with no juggling of kernel or X keymaps. Caps Lock is a little broken and will need some logic tweaks in the driver, but clearly this solution is a winner. Performance is excellent (CPU load sits at zero when I'm not typing) and auto repeat works perfectly (it's handled by the kernel so I don't have to deal with it in kbdd).

Now what remains to be done is create the kbdd maps for other keyboards (not everyone has a Belkin F8U1500). The original kbdd source lumps all the keymaps and IR protocols into a couple files, but I'm going to break them out so there's one file per keyboard. That should make it much easier for those that want to add support for
additional keyboards. I'll add the driver for the Micro Innovations Pocketop keyboard (since I have one to test with) and I may take a shot at the Targus IR keyboard since the maps and protocols are captured in the original Zaurus IRK source code.

That will get this to a beta release. From there I'll add more keyboards (if I can get assistance from keyboard owners to figure out the maps/protocols). I also want a GUI applet to launch and kill kbdd. I may also need to put some thought into supporting non-english letters (probably using "compose" sequence support already in the kernel). Last but not least I have some ideas for adding a "mouse mode" where the arrow keys on the keyboard could be used to move the mouse cursor. Suggestions are welcome.

(part 4)

Friday, October 07, 2005

Zaurus pdaXrom IR Keyboard Drivers (part 2)

(continued from part 1)

I thought that the additional kernel drivers would be the difficult part of porting kbdd to pdaXrom, but the kernel work proved to be the easy part. Getting the kbdd daemon to build and run on the Zaurus is easy -- getting it to do the right thing is a little more difficult!

The first problem is that kbdd is written for the iPaq and the iPaqs for which Familiar Linux has been developed, don't have built in keyboards. So kbdd is designed to work with its own kernel keymap which would normally be loaded via the kernel's "loadkeys" command before kbdd is started. The kbdd daemon then translates the keycodes it gets from the IR keyboard into scancodes that when looked up in the loaded key map by the kernel's keyboard driver will ultimately put the right characters into the input queue.

Not a bad solution, and I was able to get some limited tests with this architecture to work from a console login, but the dual lookup table approach (keyboard to kbdd map and then kbdd map to kernel) seemed a little redundant. Why not just install a map matching the IR keyboard in the kernel and let kbdd just directly pass the keyboard's scan codes to the kernel without an intermediate lookup?

So I coded things up that way and sure enough it worked in a console login -- almost. A few keys wouldn't make it through. Checked the map, double checked the map, tripple checked the map, still no go. I could see the correct scan codes going to the device drivers, so it had to be a driver problem. A little digging revealed that the keybdev driver assumes that the kernel is using a normal x86 keyboard map and attempts to translate incoming keycodes to x86 scancodes. However I've replaced the kernel's key map with one that matches the codes from my IR keyboard, so keybdev's little conversion is getting in the way.

Modifying keybdev so that when built for the Zaurus it passes all keycodes through with no translation corrected the problem. Now I can see all the keys correctly in a console login.

Next step -- try it in XFree86. Not being an XFree86 or pdaXrom expert, I foolishly expected that since my kbdd worked in the console it would work in X. No such luck. Key bindings in X were totally wrong. A little poking around revealed that when pdaXrom starts X11, it loads a Zaurus specific keyboard mapping for X.

So sticking with my "no mapping in kbdd" approach is going to mean I need a keyboard specific kernel mapping and a keyboard specific X mapping. And I'd need a way to automatically load them when kbdd is started and restore the previous mappings when kbdd is shut down (or the Zaurus built-in keyboard won't work). On top of that, this approach makes it impossible to use both the Zaurus and IR keyboards at the same time.

Time to re-think things ... (part 3)

Zaurus pdaXrom IR Keyboard Drivers (part 1)

My XVkbd IR keyboard hack worked reasonably well as a stopgap solution, but polishing it up to the point I can live with it long-term is going to be nearly impossible. The fact that it's only functional in the X11 GUI is a fundamental limitation. In addition to that, I'd have to learn a lot of low-level X11 programming to get performance up to a level I'd be satisfied with. Time to design a better solution!

It turns out there's a project at Handhelds.org called kbdd that aims to provide almost exactly what I'm looking for in Familiar Linux on Compaq iPaq PDAs. It has the advantage of being a daemon that injects the keystrokes at the Linux kernel level making them available to both the console and X11. And since it's not an X11 application, it can be modified without slogging through a big pile of low-level GUI code. Looks like a perfect candidate for a general purpose pdaXrom IR keyboard driver.

However, bringing it to the Zaurus presents a couple of issues. First of which is that it uses three Linux kernel drivers (uinput, keybdev, and input) to take data from the "user space" daemon and pipe it into the kernel's keyboard driver. Those driver modules don't exist in pdaXrom. In fact, the uinput driver didn't even exist until the 2.4.23 version of the Linux kernel and pdaXrom is based on Sharp's "official" 2.4.18 kernel version.

Luckily I have been able to backport the uinput driver to the 2.4.18 kernel, build all three drivers, and get them to run properly on my C760. The most difficult part was setting up a kernel cross-development environment and then reconstructing the source tree for the 2.4.18 Zaurus kernel from Sharp's collection of kernel source patches. Sharp's kernel source modifications can get pretty ugly, so I was pleased to find that it took nothing more than tweaking the .config file to get the needed additional drivers to build.

Next step, adapting the kbdd daemon ... (part 2)

Zaurus pdaXrom 1.1.0rc11

pdaXrom is a Zaurus ROM (kernel and root filesystem) that doesn't ditch the Zaurus ROM structure entirely, but does replace Qtopia with an optimized XFree86 and a host of X11 replacements for the standard Qtopia utilities. The goal is to turn the Zaurus into more of a mini-laptop than PDA. There is a GPE variant of OpenZaurus that endeavors to do something similar, but the pdaXrom developers tend to be a bit more pragmatic than the OZ folks leading to a ROM that has always worked better for me than the more technically correct (but often less finished) OZ.

I like the idea behind pdaXrom and so I've been following its development and periodically flashing new releases into my SL-C760 for evaluation. Unfortunately, each time I have found myself flashing back to the Cacko (Sharp compatible) ROM within a day because pdaXrom was missing something things that I just wasn't willing to do without. So when RC11 (the 11th release candidate for version 1.1.0) came out last week, I had to give it a try.

The results so far have been very favorable. I won't try to review it here, but I can say that for the first time, pdaXrom has stayed on my Zaurus for more than 24 hours. In fact, it's now been in place for an entire week and I still haven't identified any problems that are likely to send me back to Cacko.

The only issue that came close is the lack of support for my Belkin F8U1500 external IR keyboard. I tend to use my Zaurus for taking notes in meetings and the built in keyboard, while great for capturing info on the go, just doesn't cut it for any significant amount of data entry.

With a business trip planned just days after installing RC11 and a strong desire not to abandon the new ROM, it was clear that I was going to either have to hack up a solution fast. Out of the box, pdaXrom includes XVkbd -- an on-screen soft keyboard for X11. Since XVkbd takes mouse clicks and turns them into keypress events for the active X11 application, it looked like a likely candidate for a "hack attack".

It proved to be fairly trivial to modify XVkbd to open the IR port and listen for IR keyboard events in addition to mouse clicks. Since I had previously worked out the Belkin's protocol for addition to the IRK driver in for the standard Zaurus ROMs, I just had to drop that code and keymap code into XVkbd. In almost no time at all, I had a version of XVkbd that took keystrokes from the IR keyboard and dropped them into whichever app had the focus. It's not a perfect solution (it won't work unless X is running, it pegs the CPU at 100%, and it would be a pain to add support for different types of keyboards) but it looks like it will be good enough to let me keep pdaXrom until I can come up with something a little more elegant.

Tuesday, September 20, 2005

Hardware Vendors Supporting Linux

I saw a good article today from a Windows to Linux switcher that provides a pretty concise look at the state of Linux on the desktop. As I expected, one of the first comments it received was that hardware vendors don't provide Linux support because of Linux's small market share. Time to dispell that myth.

The problem with hardware vendors targeting Linux isn't market share, it's that there's no single "Linux" to target.

Let's say you're an open minded hardware vendor. You see that the overall Linux market share isn't huge, but it's big enough that targeting it could give you an advantage on your competition. Your software development budget is limited so you have to pick a specific "Linux" to target. How much does the Linux market share drop if you say "I'm going to target just the latest kernel"? I don't know the exact number but 75% or more wouldn't surprise me. Of course device drivers are not guaranteed to be binary compatible across even minor kernel releases, so you just picked up an ongoing maintenance task of releasing a new driver binary every time there's a new kernel release. Why not just release the source code and get it integrated into the kernel baseline? Well, right or wrong, most hardware vendors don't want their competitors to have any insight into the design of their hardware and in many cases driver code can be very revealing.

So you come to terms with the device driver issues and you're convinced that there's still enough market share to make it worthwhile, now what about the user space utilities? Do you target Gnome or KDE (or something else)? Now you not only take another big market share hit, but you so infuriate the other "camp" that you'll be subjected to a bunch of negative Linux press over your decision (and of course you've angered the CLI die-hards either way). But it doesn't end there. Which version of Gnome or KDE? They're both moving targets following the tried and true open source doctrine of release early, release often. There's that whole maintenance thing again. And here releasing source becomes even stickier because if you go that route there's literally nothing standing between your products and the Asian hardware cloners.

Now, depending on how complex your utilities are, you may also need to decide which additional shared libraries are safe to use? Which versions? Every decision divides the market share your development will be able to reach. But you're not done yet. Which package format do you distribute in -- RPM, DEB, custom installer? Another decision, another division.

Trying to target Linux with a binary softare release is like trying to nail Jello to a tree and that problem will persist despite any forseable market share gains. Until one single distro (with a commercial style release schedule) gains enough market share by itself, the situation won't improve. RedHat came close, but the inherently "forkable" nature of any Linux distro works against any of them gaining enough traction to become the "one true Linux" that vendors can easily target. Linux users who continue to chant the mantra of "choice is good" need to realize that "choice" is a double edged sword that will continue restrict "Desktop Linux" to the domain of the highly motivated.