Kuarepoti-Dju

kuarepoti-dju - the josef multilingual blogosphere

23.3.2008

Completing a bunch of bindings: GGZ-Java

Categories: — josef @ 23:07

Once again I spent a part of the day on library bindings and reimplementation in other languages for the GGZ Gaming Zone. Our goal would be to release the next version of GGZ with complete libraries for five programming languages, and we’re now very close already with the recent additions to the Java side.

The jggzdmod library implements the GGZ-server-to-game-server control protocol. It is written in 98% pure Java with a bit of JNI magic because Java lacks quite a few of the advanced socket functions which we need for efficient game launching. There’s now a comparison document for the five languages above regarding socket features and workarounds.

18.3.2008

GSoC 2008 offer: Adaptive printer dialogue

Categories: — josef @ 20:25

What does the future of printing look like? While nobody knows for sure, there are certain trends such like the ubiquitous availability of a printing service nearby, available from the user’s computer. When devices are discovered dynamically, their capabilities need to be conveyed to the user. Right now, only the most basic printer options show up as native controls in the printer dialogue, and often the more interesting ones are hidden. Furthermore, printing habits such as always wanting to save on toner translate very badly into the UI at the moment. The Linux Foundations Open Printing group has initiated a move away from today’s inflexible printer dialogues towards something better from a usability and a technical angle.

The Google Summer of Code task of working on the Common Printing Dialogue is about translating usability and interface engineering concepts, together with some more formal research on dynamic GUI generation, into a top-notch GUI to conclude and validate this effort. Two mentors, Jonathan Riddell and me, will be available, and hence up to two students could work on the topic. Contact either of us if you’re a potential student and want to know more about it beforehand or want to propose your own idea related to that one (see contact data below the idea description). If you’re in for more than just the coding, make sure to sign up to KDE’s kde-research and LF’s printing-summit mailing list as well.

9.3.2008

Multiplayer scenario

Categories: — josef @ 13:54

Somewhen after midnight last night GGZ 0.99.2 snapshot was uploaded. Just a few hours after, the next cool patch was created. Development runs at high speed at the moment, even though I’m still busy drawing presentation slides like a dull ODP monkey for my 2nd job (lecturer).

After my two FOSDEM presentations and some chatting after each I think that while the software required to achieve high-quality free multiplayer gaming isn’t there yet, the individual puzzle pieces already let us forebode a scenario of some sort, as a guide of what shall (or will) be possible. We have GSoC 2008 proposals coming up for those who want to help out.

The scenario goes like this: Bob wants to play Chess with Eve. (He used to play with Alice but they broke up, just to introduce some dramatic content here.) He might be logged into his GGZ core client already, possibly lurking at the Chess or Freeciv room, or he might hang around on Jabber. As soon as he finds Eve, he’ll launch a game of Chess and selects his favourite client, e.g. GGZBoard. Eve then joins this game using Tagua since she’s on KDE 4. Hurray for protocol compatibility!

Alice joins as a spectator and is totally jealous of the two players. She decides to found a player club called Toxic Pawns and invites tons of people. She’s a good club leader and also a helpful, above-average player and thus gets granted host status, which gives her privileges such as kicking players and sending private table messages, and of course hosting tournaments. In order to accomodate those players who came up with a new set of rules called Rookymotion Chess, who are believed to prefer playing audio files over playing games but like Chess nevertheless, she requests a new room from the community administrator who easily sets one up based on existing room templates through the community web frontend.

Meanwhile, Bob and Eve continue playing individual games, although Eve’s DSL connection often breaks for no reason. Of course, she can regain her abandoned game seat after a reconnection. One day, the server goes down, yet after a reboot all games are being restored. No data is lost, and no data is being leaked due to encrypted connections. The player’s personal page also features a privacy section which Eve prefers a lot to the ad-driven, privacy-invading other social networking sites she knows. Trust is important in online gaming, since technical efforts to prevent cheating can and will always be defeated at some point. But with karma points, veteran status and other community relationship indicators, Eve can rest assured that Bob’s friends are trustworthy people.

So much for the community-oriented view on all the technology which is currently on the work bench. Except for a few shameless plugs, the presentation should have shown that the software is making great progress and it’s about time to think about community and player-oriented concepts. Gaming is still one of the areas where free software isn’t #1, and changing that requires some vision beside the daily coding. Send comments to any game list I’m subscribed to.

2.3.2008

KDE on tour: CLT2008, CeBIT

Categories: — josef @ 21:37

The Chemnitzer Linux-Tage just finished - an enjoyable event like every year, with a certain community spirit not often met at events of that size. The KDE booth was staffed by four people (smajewsky, eckhart, tokoe and me). People were mostly interested in KDE 4.1 alpha to put it bluntly :) but we demonstrated 4.0.2 on most of the screens. Ruphy was kind enough to provide us with new oxygen-ised poster templates before the event so we printed two A0-sized posters. Unfortunately the boothbox didn’t make it so we didn’t get most of the booth decoration.

While working on some plasma data provider at the end of the second day we learned that there’s a secret tool called AutoCMake to help with the creation of cmake files. Now what’s totally missing is AutoCConf so we can come up with some decent AutoCTools! Just kidding. Some more serious issues were about my xrandr-powered Xinerama-like setup which Plasma didn’t handle too well. Also dynamically adding and removing batteries of the laptop isn’t well-supported as of yet in the battery applet. Nevertheless, most visitors were satisfied with 4.0.2 quality over their previous 4.0 impressions which they mostly got from CDs shipped with computer magazines.

The next important event for KDE will be CeBIT. According to tackat’s message on kde-events, volunteers are still needed for CeBIT next week. Don’t miss the opportunity to represent the latest trends in free desktops to an audience which still needs that sort of information.

16.1.2008

Lufthansa und die Ticketpreise

Categories: — josef @ 20:00

Warum allerdings bei der Untersuchung Internet-Tickets teilweise erheblich teurer gewesen seien als im Reisebüro, konnte sich Lufthansa zunächst nicht erklären. Dies werde man prüfen.

Das war die Reaktion der Lufthansa auf eine Untersuchung der enormen Preisunterschiede im September 2007, welche auch durch die Tagespresse veröffentlicht wurde. Dennoch hat sich nichts getan. Auf der Suche nach vernünftigen Personentransportangeboten von Berlin nach Brüssel bekam ich heute eine Spanne von 150 € bis 1400 € angezeigt. Letzteres bringt mich auch nach Südamerika, egal ob mit oder ohne Lufthansa. Fazit: Null Punkte für Mayrhuber, null Euro von mir seine Angebote.

9.1.2008

The wicked world of wireless woes

Categories: — josef @ 11:24

I want to operate my new notebook in master mode. With my old notebook, the MadWifi driver supported this for the Atheros-based internal adapter on its own. With my new notebook, I have several choices, none of which seem to be particularly supported:

  • The internal Intel-based adapter runs with iwl4965 which doesn’t support master mode. In such cases, one can use the HostAP daemon which among other chipsets supports the generic mac80211 layer (named devicescape) on top of which the iwl4965 is built. Unfortunately, the Debian package ships hostapd with devicescape disabled, and trying to build it on my own also failed due to incompatibilities with a new kernel (2.6.22).
  • The external Prism2-based adapter works with Linux-WLAN-ng, which doesn’t support WPA but otherwise works well. Unfortunately, the linux-wlan-ng driver isn’t supported by hostapd. There’s also the hostap driver (without the -d) which supports Prism2 chips, is naturally supported by hostapd and even became part of the kernel sources recently, EXCEPT that it doesn’t support USB-connected Prism2 devices, and mine is one of those.
  • The external Ralink-based adapter is not supported with its native driver (rt61). The new driver version (rx200) is based on mac80211 and will likely lead to the same issues that the iwl4965 has.

Oh well. Of course WLAN adapters are cheap nowadays, but I still would like to see the existing hardware supported :-)

10.11.2007

Rebooting made in Germany

Categories: — josef @ 22:50

For whatever reason, German transportation companies refuse to use reliable systems for their passenger information screens. Resilience against bumpy roads is probably hard to achieve, but based on discussions I had with some of them and especially live experience, one can conclude that most of the outages and reboots are due to crappy software.

The images below are just moments where such systems came across my way in summer and autumn 2007 in Berlin, Frankfurt and Saxon Switzerland. Please folks, upgrade now!

22.10.2007

Design mistake in udev - and more horrible Kubuntu upgrade stories

Categories: — josef @ 14:10

Today I upgraded from 7.04 to 7.10, which was reported to be a smooth upgrade by many people. I cannot complain about the upgrade process itself, as there were no evil package dependency issues like they happened when going 6.06 -> 7.04.

However, there are a couple of issues. First of all, the udev rules include a file 85-evms.rules. Took me a while to find that beast - it’s been responsible for constantly querying my CD-ROM drive, causing a loud clickery every 2 seconds, and certainly reducing its MTBF. My first idea was to run /etc/init.d/udev stop. Ha - udev manages pseudo-devices (PTY) as well, and so I was left in a limbo, unable to open any konsole/xterm kind of application. Great for debugging especially since SSH wasn’t running yet, so this upgrade forced me into a reboot. Welcome to the 80s! Removing this file and restarting udev helped. Those who use EVMS are getting punished still I assume.

Second, but that’s really more a KDM issue, it wouldn’t let me log in because the PAM plugin missed a symbol. The new KDM was already installed, but the old one turned up when logging out of the X11 session. This could be done better, conceptually.

Another issue is that amixer is shutdown with the wrong parameters on halt.

A debian bug report recently was closed without giving a solution to the problem of not being able to access the scanner directly after installing the sane family of packages. Just to mention that the rantfest could go on with other distros as well. Having a bit of a headache, I won’t therefore file bug reports against Kubuntu right now.

9.10.2007

My new job @ Turkish secret service

Categories: — josef @ 21:32

To many people I seem to have disappeared recently. However, I had just changed my location to İstanbul to assume my new role of being a special agent for the Turkish secret service. This exciting new job was kept secret even from myself, until I learned about it on the Mısır Çarşısı, the Egyptian Bazaar, while trying to buy some Lokum-like sweets, as the following dialogue tries to reconstruct (with the vendor phrases translated into English to keep it simple):

  • Me: Merhaba.
  • Vendor: Hey Sir, what can I do for you.
  • Me: Yüz gram of… pointing to the unknown sweets
  • Vendor: Ah, Türkçe… where are you from?
  • Me: Doğu Almanyalıyım.
  • Vendor: So you learnt Turkish there?
  • Me: Hayır, öğrendim burada İstanbul’da.
  • Vendor: Hm, how long have you been here?
  • Me: Bir hafta.

At this point, the vendor stops packaging up the sweets, sits down and looks at me.

  • Vendor: There’s something fishy here. You’re from the police… or from the secret service!

Later I figured that I must be a very lousy special agent because I was uncovered. Nevertheless, the sweets were really good ones.

3.8.2007

SMC AR4505

Categories: — josef @ 07:19

Ein wenig wundert es mich schon, dass man zu obiger Modellbezeichnung eines WLAN-Routers quasi keine Einträge in einschlägigen Suchmaschinen findet. Dem sei mit dieser Meldung nun Abhilfe getan. Falls jemand solch ein Teil mit {Open/Free/DD-}WRT betreibt, würde ich gern etwas darüber hören. Momentan sind sämtliche SMC-APs in der Hardware-Datenbank als WiP deklariert.

Das Baby wurde allerdings bereits auf den Namen ‘’parada'’ getauft.

26.1.2007

Sorting out the Java mess

Categories: — josef @ 18:15

Java can be considered the lingua franca in the area I’m working in. I’m always positively surprised when a student or two come up with ideas in other languages, but for the most part, there doesn’t seem to be a way around Java, since this is the only halfway-quality language most students can handle.

With Java becoming GPL, ideological problems should vanish. Or so one thinks. Of course, the idea is nice, and I whole-heartedly support it, but a lot of work will be needed to turn it into a reality. During this year’s upcoming FOSDEM event, OpenJDK/Sun people and Classpath people want to meet. This isn’t the first such meeting, but AFAIK the first big one since the GPL announcement which coincided with the last Ubuntu meeting which already had some Sun and Classpath (or Java-Debian, for that matter) people on board.

The issues I see are manyfold. Java has gained a lot of capabilities in terms of implementations of existing JSR specifications. Before JDK 1.6, one has needed a few additional libraries of what is now available for Java proper. Especially for Apache libraries, which are ASL 2.0-licenced for most parts, this means a GPLd Java runtime will be preferred by GPL application authors. However, the build system must still support the Apache libraries just in case the JDK is not at 1.6 yet. Furthermore, the GPLv3 seems to clear up the (already highly theoretic) incompatibility between ASL 2.0 and GPLv2.

Such additions to the Java core are not new. Already in the past, a lot of XML capabilities were added and therefore fallbacks must be available in case the JDK or at least the JRE at the deployment site is too old.

For one of my upcoming releases next Monday, I finally had to look a bit more into this and see which Sun Java and GNU Classpath implementations would fit my bill.
Interestingly, this flashed my mind on many fronts, since quite a few weird runtime errors I’ve seen can be attributed to version incompatibilities.
The outcome is that JDK 1.5 wins, while JDK 1.6 is not ripe yet and JDK 1.4 is way too old and requires too many additional libraries. Classpath 0.93 is nearly there, but I added some bug reports to their tracker.

We’re incrementally getting there. Of course, the flashy parts such as mobile phone emulators are still proprietary software, and time will tell if they will ever get opened up. They’re not available for Mac users, but they’re available for Linux, so the majority of complaints of them being non-free should interestingly enough come from the Mac crowd :)

As soon as everything is free, distros might offer Java installed by default and therefore turn into a rocking development platform for this language. However, since Java has been kind of isolated for so long, it comes with its own build infrastructure (ant, maven) and its own repository, which doesn’t really handle versioning well. Proper packaging will help in identifying compatibility issues.

It remains to be seen how third-party Java developers, especially IBM, will react to these developments. Some certainly nice IBM Java tools are still proprietary software, which makes them unsuitable for students to explore and to improve. Anyway, after nearly 2 years of working with Java software extensively, the overall opinion is that it has improved to the better, and cheers go out to those who made it happen.

30.11.2006

Telekom und FLOSS

Categories: — josef @ 20:04

Die Deutsche Telekom hat sich bisher leider nicht sehr unterstützend im Bereich Open Source hervorgetan. Die Verstärkung der strategischen Partnerschaft mit einer dieser Redmonder Firmen dürfte dem auch nicht zuträglich sein. Dabei profitiert die Firma doch recht stark von dem Einsatz …
(aufgedeckt durch scharfe Beobachtung am Dresdner Studentenwerk durch M.E.)

3.11.2006

Machines

Categories: — josef @ 13:47

This is a permanent entry which will be updated from time to time, similar to the activity/publications entry.
I thought it is about time to document what kind of machines I have and had. I had such an overview page some years ago but it got lost. Here we go…

The only machines in operation are Vista, Nada (still), Madrugada, Caramba and Samba.
The backup computer Noosphere2 seems to be broken but final death is not confirmed yet.

Samba is my latest work notebook, the successor of Nada. It was inaugurated in late 2007.
Specs: FSC Lifebook E8410, Intel Core 2 Duo T7500 (2*2,20 GHz), 2 GB RAM, 120 GB HDD running Debian/sid

Caramba replaces Noosphere2 as private backup computer and general-purpose PC. It was bought in pieces and assembled just like its predecessors, and is very light since no external drives are present, although for safety reasons a RAID-1 has been created. A 19′’ TFT accompanies the machine.
Specs: AMD Athlon64 X2 3800+, 1 GB RAM, 2*250 GB HDD running Debian/etch

Madrugada is a cute subnotebook, model JVC MP-XP 7210. It was bought as a used machine and although limited in its capabilities is very useful sometimes, especially on travel.
Specs: Intel Pentium III 800 MHz, 256 MB RAM, 30 GB HDD running Debian/etch

Nada is a work-sponsored hacking notebook, and as such the successor to Armada and Bizarre. I got it already as a used notebook in late 2006 and so I hope it will survive long enough.
Specs: Intel Pentium M 1700 MHz, 768 MB RAM, 60 GB HDD running Debian/sid

Vista is my main computer at work, a FSC Esprimo machine. The name was influenced by the great panorama view on the skyline of my city. I also go by josef|vista on IRC when logged on there, which is almost always as I never switch the thing off.
Vista is equipped with a nice 20′’ TFT.
Specs: AMD Athlon 64 X2 Dual-Core 3800+, 1 GB RAM, 85 GB HDD running Kubuntu 6.06 (Dapper Drake)

Armada was a notebook I’d temporarily been donated by a friend of my brother. As the name suggests, it’s a Compaq Armada M700, certainly not the fastest notebook on earth but I can do some hacking when travelling. Unfortunately I only could keep it for some weeks in autumn 2006.
Specs: Intel Pentium III 450 MHz, 256 MB RAM, 20 GB HDD running Kubuntu 6.06 (Dapper Drake).

Bizarre was my previous notebook, a new-world Apple iBook of the second series. Bought in early 2003 it lived until mid-2006.
Specs: IBM G3 PowerPC 750FX (700 MHz), 256 MB RAM, 20 GB HDD running Debian sid (mostly what later became sarge and etch) and OS X 10.2 (not really used).
Replacements: power adapter in mid-2005

Itajuba was the computer in the lab of UNIFEI where I studied in 2004.
Specs: Intel Pentium II 700 MHz(?), 128 MB RAM, 20 GB HDD (?) running my very own TUDix upgraded to Debian/unstable
Note: memories fading, will need some backed-up notes on that one

Noosphere2 started out as main private machine in mid-2002 under the name Noosphere, influenced by ESR’s hacker tales. After a drastic surgery, it changed its purpose to be my backup computer and so I decided to modify the name.
Specs: AMD Athlon XP 2000+, 512 MB RAM, 80 GB HDD running Kubuntu 5.04 (Hoary Hedgehog) and FreeBSD 5.x/6.x.
Replacements: hard disk in early 2005, CD burner addition in late 2004

Résistance has been the main machine before Noosphere. It was bought in 1999 and assembled, including the 17′’ monitor which later served Noosphere as well. Its first name was Athlon500, following the old naming scheme. After Noosphere was bought, it served as server and gateway and thus got its name.
Specs: AMD Athlon 500 (first series with Slot-A), 128 MB RAM, 13 GB HDD running Debian sid (mostly what later became woody) as well as some other OSes (OpenBSD, Hurd).
Replacements: CD burner addition somewhen in 2003

Pentium133 was my first own machine, bought in one piece with a 15′’ CRT monitor in 1996.
It was factory-equipped with MS-DOS and Windows but was switched over to Linux in 1999.
A lot of DOS-based development was perfomed on this machine.
Specs: Intel Pentium 133 MHz, 16 MB RAM, 1 GB HDD running MS-DOS 5.22/Windows 3.11, SuSE Linux 6.1 and Mandrake 7.x.
Replacements: CD-ROM drive addition and exchange in 1998

Schorsch had the honour to be the first computer I’ve ever extensively worked with, an Olivetti machine bought in late 1992. It was actually only named so in retrofit around late 1997.
Specs: Intel i286, 1 MB RAM, 120 MB HDD running MS-DOS 5.0/GeoWorks.
Replacements: The 14′’ colour monitor failed around 1998 and a monochrome one had to be put into place

Other devices not really usable as a generic computer:

Zaurus is a Sharp Zaurus SL-5500 donated in 2002 by Sharp. It was used to do some embedded GUI hacking.
Specs: (Intel) StrongARM 200 MHz, 96 MB RAM/ROM, 32 MB of which are used as RAM, running some embedded Linux.

10.4.2006

April 10 is Cool Projects Day

Categories: — josef @ 17:51

While space travel fans will look forward to April 12 (День космонавтики, or Cosmonaut Day), or April 11, the National Hero’s Day in Costa Rica, today also is a commemorable day. Not just because 36 years ago, Paul McCartney announced he’d leave the Beatles, but also because in 2006, many cool projects were released in a new version again.

Cool Projects Day

17.3.2006

Apache content negotiation vs. Browser

Categories: — josef @ 16:26

Es gibt Leute, die haben richtig viel Ahnung von der Erstellung eines Webbrowsers.


XX.YY.ZZ.QQ - - [05/Mar/2006:19:36:49 +0000] “GET /games/ggzcards/index.php.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en.en HTTP/1.1″ 302 5 “http://www.google.de/search?xxxxx” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)”

Leider betrifft das auch ein paar (ältere?) Firefox-Versionen und Epiphany, laut Logstatistik. Dabei ist HTTP 302 doch eigentlich klar geregelt :(

1.3.2006

Avionics: Jubiläum

Categories: — josef @ 14:02

Nachdem ich gestern mit yinjieh über Flüge gesprochen habe, ist mir erst in der Nacht aufgefallen, dass ich in derselben Nacht vor genau zwei Jahren zum ersten Mal im Flugzeug saß. Das Jahr 2004 war ein Schaltjahr, und während ich am 29. Februar von Dresden nach Frankfurt und dann weiter flog, kam ich am 1. März in São Paulo an. Seitdem sind nicht nur 24 Monate, sondern auch 25 Flüge vergangen, dazu kommen noch einige Flughafenbesichtigungen während der Zwischenlandungen.

Eine kleine Statistik darf natürlich nicht fehlen:

  • Flughäfen, die ich von innen gesehen habe: 14
  • Länder, die ich so neu bereist und kennengelernt habe: 5 [Brasilien, Argentinien, Paraguay, Spanien, Portugal]
  • Anzahl der Fluggesellschaften: 6 [Lufthansa, Varig, TAM, VASP, Air Berlin, Germanwings]
  • Anzahl Zwischenlandungen: 5 [in Belo Horizonte (2x), Belém, Fortaleza, und in João Pessoa]
  • Anzahl Fluggesellschaften, die einem nichts zu essen geben: 1 [Germanwings]
  • Anzahl in der Luft verbrachter Geburtstage: 1
  • Anzahl lustiger Piloten, die kurz vor der Landung noch eine Runde drehen und dazu Karibik-Musik über die Bordlautsprecher durchgeben: 1 [Anflug auf Salvador]

Wem all das noch nicht genug Hack ist, dem sei noch folgende Geschichte erzählt, die ich mal leicht anonymisiert weitergebe: Es gab da einen Kontributor eines bekannten OSS-Projekts, der immer nach dem Flug seine Änderungen ins CVS eingecheckt hat. Nun ist Hacken im Flugzeug nichts Ungewöhnliches - in dem Falle aber schon eher: Er ist der Pilot.

27.2.2006

struct sockaddr is evil

Categories: — josef @ 16:50

When accept()ing a connection, one gets connection information about the peer host. Usually, this isn’t stored in userspace, but the kernel keeps track of it all the time. The normal way of getting this information is, and always has been, calling getpeername() on an open file descriptor. Afterwards, inet_ntop() or better getnameinfo() will be able to convert this structure to a numeric host (IP address) or human-readable host name.

What your parents and man pages never told you, even though RFC2553 is a couple of years old: Instead of casting a struct sockaddr to struct sockaddr_in, a struct sockaddr_storage must be casted. Funny enough, for most other operations using a struct sockaddr is working even with IPv6.

getpeername(fd, (struct sockaddr*)&addr, &addrsize);
blah((void*)&(((struct sockaddr_in*)&addr)->sin_addr))…

Thankfully, there is now ggz_getpeername(fd, resolve) to avoid using brain-dead casting-is-cool APIs.

7.2.2006

Advanced Python Hacking III

Categories: — josef @ 11:38

Just when one thinks enough is enough!, it’s actually just starting… More functional features for Python please!
Here they come:

  • partial function application
  • lazy evaluation

The meaning of partial function application is that when a function is called with less arguments than it would need according to its specification, the return value is actually another function which can then be called with the remaining arguments in order to get the result. This makes it possible to pass around the half-baked function and can optimise repetitive coding a lot.

Go Python:


#!/usr/bin/env python
import partial
# bar returns a tuple of its two arguments
def bar(a, b):
print “<bar> a =", a, “b =", b
return (a, b)
# Let’s make it partial now!
partial.verbose(1)
partial.partial(bar)
# First call: get intermediate function object
partialfunc = bar(3)
print partialfunc
# Second call: finish the computation
result = partialfunc(4)
print result

Just for the reference, the whole debug output is given now so the functionality can be traced alongside the calls…


[partial] called bar with (3,) and {}
[partial] - first slot 0
[partial] - assign 3 to slot 0 as argname a
[partial] = partial result:
[partial] = 1 out of 2 arguments
<partial .__partialclass instance at 0x30079bc0>
[partial] called bar with (4,) and {}
[partial] - first slot 1
[partial] - assign 4 to slot 1 as argname b
[partial] = full result:
[partial] = (3, 4)
(3, 4)

Now, a somewhat more difficult topic is lazy evaluation. Not possible with Python? Well, not easily, but still possible to some extent. It means that if a function is called, the actual computation is not done unless the final value is required. Calling foo(bar(), 2) does not call bar() if e.g. foo only uses the second argument, and thus saves precious computation time. Code ahead:


# Some addition function
def bar(a, b):
return a + b
# Make it lazy!
lazy.verbose(1)
lazy.lazy(bar)
# See how the type is not int, but type-instance instead
x = bar(3, 4)
print “Type", type(x)
# Now it really gets calculated internally to 7
print “double (= 14?)", x * 2
print “Type", type(x)
# From now on, the 7 is cached internally
print x + 11
# … and so on

As has been shown, functional programming with Python is much more than what is written in the documentation. Free your mind and use the powerful features!

6.2.2006

Advanced Python Hacking II

Categories: — josef @ 12:34

Python is not just an object-oriented imperative programming language, but also contains several features known from functional programming, such as higher-order functions, list comprehension, map/reduce/filter functions and lambda terms. However, more advanced concepts are not present in the language. Can they be emulated with existing Python facilities?

It turns out that this is indeed the case for some features.

  • Infinite data structures
  • embedding Haskell code
  • others (upcoming…)

Infinite data structures are for example lists which are generated and can be traversed for many thousands of years without reaching the end. In Haskell, lazy evaluation makes this possible. Things are a bit less native in Python, but using generator-style sequence objects, infinite lists can be brought to the programmer:


# Generator ‘Y’ around Fibonacci function
seq = Y(fib)
# Let’s print some slice of this list
seqslice = seq[5:20]
print seqslice
# Eternal iteration…
for y in seqslice:
print y

Y is in this case the generator sequence object. Note that lazy evaluation is one of the features which probably cannot be done in Python without modifying the interpreter, so the zip() function will return a Memory Error.
But, we’re cool, eh?


# Our own zip which handles infinite structures
def xzip(l1, l2):
return lambda x: (l1(x), l2(x))
# Make a companion function to fib()
def id(n):
return n
# Use it like this
for t in Y(xzip(id, fib)):
print t

Now let’s make things even more cool: How about freely mixing Python and Haskell code?
Getting the best out of both worlds is the goal of the haskellembed module. A quick functionality overview:


#!/usr/bin/env python
# Import the haskellembed module
import haskellembed
# Define some code (could also be in external *.hs file)
code = “"”
– konstanter Wert
fs :: Int
fs = 4
– Addition um konstanten Wert
fs2 :: Int -> Int
fs2 n = n + 1
– Rekursive Berechnung der Fibonacci-Funktion
fs3 :: Int -> Int
fs3 1 = 1
fs3 2 = 1
fs3 n = fs3 (n - 2) + fs3 (n - 1)
– Wildcards
fs4 :: Int -> Int -> Int
fs4 n _ = n * 3
– Guards
fs5 :: Int -> Int
fs5 n | n > 0 = 42
| otherwise = -10
– Lambda-Notation
fs6 = x y -> x + y
“"”
# Now, convert Haskell to Python on the fly
# and inject the public functions into our namespace
# Alternatively, without giving globals(), a sandbox is created in h which lets you access internals as well and inject as needed
h = haskellembed.convert(code, globals())
# Finally, just use the Haskell functions!
print fs6(10, 20)

The cool thing is that using Python functions from within the Haskell code now becomes natural. Just use time.time() in an expression and voilà, as long as Hugs or other interpreters accept the code, Python will execute it just fine.

Of course, the conversion support is limited to the above examples for the time being. But interesting times are ahead, adding more features can be done within minutes.

23.1.2006

Advanced Python Hacking

Categories: — josef @ 16:06

While preparing for an exam and thinking about which kind of course to offer for students at one of the upcoming events at University, a hopelessly non-performing recursive Fibonacci calculation made me think about caching function values in a generic way for constant functions and, as an extension, eventually constant functions. It resulted in a cache module for Python.

Consider the following code snipped:


#!/usr/bin/env python
import cache
def fib(n):
if n < 3:
tmp = 1
else:
tmp = fib(n - 1) + fib(n - 2)
return tmp

Now, calling fib(35) takes many minutes to complete. Already fib(20) takes about half a minute and more than 150049 function calls!
(The absolute time is worsened a lot by the benchmark redirections - but still the number of calls should indicate that the recursive calculation is not a good idea.)

Now, let’s enable the cache:


cache.cache(fib, globals())

That’s all! fib(35) finishes in less than a second, needing only 47 invocations, of which 22 yield cache hits and 25 yield cache misses.
Using a backing file means that cache values can be reused from previous runs - which is a nice feature for long running programs so crashes won’t harm.


cache.backingfile("__cache.file")

Now a single iteration is sufficient for calculating fib(35)!

Another advanced usage is that of an eventually constant function. One of this kind is a download function, which lets you use the cache module as a WWW proxy:


#!/usr/bin/env python
import cache
import random
# Simulate an error-prone file download
def download(file):
x = random.randint(0, 1)
return (None, “xyz")[x]
cache.cacheretry(download, globals(), None)
cache.verbose(1)
print “– Download:”
print download("x")
print download("x")
print download("x")
cache.stats()

With the cacheretry statement, the eventual constness is made known to the cache module.
Of course, several different Python methods can be cached at once.

Download the file cache.py to use it right away. It comes with quite a bit of documentation (at least compared to my usual habit of not documenting source code at all).

16.1.2006

Frischfleisch

Categories: — josef @ 16:38

Wer ein aufmerksamer freshmeat.net-Leser ist (geht das heutzutage überhaupt noch?), der hat sicherlich die neuesten Tarbälle gesehen, die ich in den letzten Tagen unter’s Volk gestreut habe:

Wichtige Hacks kurz vor der Prüfungsperiode. Enjoy…

5.1.2006

RR&S: DNS, /etc/hosts and you

Categories: — josef @ 16:07

Resolving host names, that is, finding a connection-level address from their human-readable representation, is a standard task in client applications which need to connect to a server. This is quite a complex task: Information about such mappings might be spread, and new additions like IPv6 or internationalized domain names (IDN) are appearing once in a while, although admittedly not often.

These properties make it desirable to use a ready-made library to perform hostname lookups. But of course, there’s plenty of choice, so which possibilities exist out there? Let’s look at the most common ones: System library functions (libc, resolv, anl), adns, firedns and Qt.

resolv:
A fairly simple interface to DNS servers is provided by the resolver function res_query(), which has its root in the BIND sources and shipped with BSD 4.3 initially. The usage takes a name, constants for the class (always ns_c_in for Internet resources) and type (always ns_t_a for hostname lookups), as well as a string buffer provided by the application. For IPv6 however, ns_t_aaaa would have to be used additionally.
Before using the call, res_init() should be called, although this is not a requirement. The call to res_query() might block the application, which is a big drawback. Another issue for the real-world usage is that this library really only supports DNS queries, therefore not taking the contents of /etc/hosts into account, which severely limits its usability for network-less development machines, for instance.
Note that the glibc version is modified a bit so that no global state is kept within the library, thus enabling resolv to be used with threads. Note also that the returned IP address is really a string, which might or might not have to be converted to a network address structure using inet_pton(), which again has the drawback of not being IPv6-transparent, albeit it can support this address family.

libc:
Probably the most well-known “classic” function is gethostbyname(). It appeared in BSD 4.3 and has also been a POSIX standard function. The call is synchronous, and supports looking up IPv6 addresses. It queries the /etc/hosts file, DNS and other name servers like NIS, depending on the global configuration in /etc/host.conf. This function is reasonably easy to use (except for a few OS incompatibilities which might occur), but is not acceptable for use with interactive programs since it will block the execution thread. To overcome this limitation, a second thread could be created manually, but gethostbyname() is not threadsafe since it uses a library-internal static buffer, so gethostbyname_r had to be introduced to let the application programmer specify a buffer.

Note that IPv6 support was not present initially, for which the substitute function getipnodebyname() has been introduced, according to RFC 2553. This function was obsoleted soon, both by adding address family support to a variant called gethostbyname2(), which is glibc-only, and other functions.

Finally, gethostbyname() has been superseeded by getaddrinfo(), or GAI for short. This function does a little more than resolving hostnames, for instance it can resolve service names as well. Otherwise, it is similar to the gethostbyname2_r() call with IPv6 support and application-provided threadsafe buffers, but also including the undesirable blocking behaviour. This function has also been named in RFC 2553.

anl:
This is a not-yet-standardised extension to GAI, and provides a function called getaddrinfo_a() and several other support functions. As the name suggests, it supports asynchronous hostname lookups, and is present so far in recent versions of the glibc library, where it relies on the AIO layer. The features are similar to getaddrinfo(). Of note is that the notification might happen in two flavours, either by thread creation or by being signalled. Both are not really optimal for all kinds of applications, although it’s not terribly hard to handle the notifications. These are the drawback costs one pays for letting the system library do the polling for the results.

adns:
One of the oldest resolver libraries which natively support asynchronous operation, in addition to synchronous mode. The adns library is initialized with a global state variable, while for each query or status check a structure is initialized by allocating memory. The main functions to use are adns_init(), adns_submit(), adns_processany() and adns_check(). A few things are noteworthy. First, it embeds well with applications since the internally used file descriptor is made available, which means notification can happen within the regular execution thread, for instance using select(). Second, a lot of information is provided, in many cases in form of a struct or union so that the relevant parts can be accessed easily, like the resolved hostname as a string and as a network address structure, depending on the further processing needs.
One drawback is shared with the resolv library, however: The file /etc/hosts is not used, an accessible DNS server is required for the library to work. Another one is that IPv6 is not supported yet.

firedns:
Looking at the big picture, firedns is similar to the adns library, although there are a few differences. For example, IPv6 is supported, although no transparent support is given - similar to the resolv library, both IPv4 and IPv6 queries have to be done separately. Integration happens similarly using a file descriptor which must be polled. A drawback shared with adns is that only real DNS queries are performed.

Qt:
A special contender in this comparison is the Qt library, which is a library for creating C++ applications, providing both GUI and non-GUI classes and a fairly complete platform-independent programming platform. In Qt3, the QDns class was available, which took the hostname and optionally the wanted resource type, and then processed the query asynchronously until a result was available. Then, a Qt signal was sent, which basically means calling back a C++ method (or Slot in qt speak), in which a QValueList contained entries of type QHostAddress, which in turn could be passed further along to Qt networking functions or reveal the IP address in string format. IPv6 addresses and the /etc/hosts file are supported by Qt. In Qt4, QDns was renamed to QHostInfo, which provides a simplyfied interface, abstracting away from DNS terminology such as resource types. Also, support for IDN (via Punycode) was introduced.

Conclusion:
It is difficult to really find the “best of breed” when all methods of resolving have their drawbacks. For Qt-based applications, using the Qt resolver is the most compelling and most sensible way of getting the work done. For all others, especially graphical clients to network services, it gets really difficult.

Let’s do a simple score table:

libc/resolv: hosts -1, ipv6 +/-0, idn -1, nonblocking -1, score = -3
libc/gethostbyname: hosts +1, ipv6 +1, idn -1, nonblocking -1, score = 0
libc/getaddrinfo: hosts +1, ipv6 +1, idn +1, nonblocking -1, score = 2
libc/anl: hosts +1, ipv6 +1, idn +1, nonblocking +1, score = 4
adns: hosts -1, ipv6 -1, idn -1, nonblocking +1, score = -2
firedns: hosts -1, ipv6 +/-0, idn -1, nonblocking +1, score = -1
Qt: hosts +1, ipv6 +1, idn +1, nonblocking +1, score = 4

There is a clear distinction between the favorites and the others, but depending on the usage, this might not tell much - for instance, C programmers will not be able to use Qt!

For this reason, the GGZ library for networking and other programming aids, libggz, has taken a twofold approach. It provides a function named ggz_resolvename() and a function to set a result callback. If libggz has been configured with support for the anl library (providing gethostyname_a()), this callback is called whenever the results are available, or otherwise directly after a blocking call, transparently to the programmer.
Since libggz has a convenience method for creating sockets, called ggz_make_socket(), the developer is even relieved from resolving the socket first by setting up a callback for ggz_make_socket(), which if activated already holds the file descriptor, ready to be used on either IPv4 or IPv6 connections. The only remaining drawback would be missing IDN support, although even this is covered for all getaddrinfo() calls with some IDN flags when using recent glibc versions (>= 2.3.4) which are compiled with IDN support.

Otherwise this is really the recommended way to go for application developers: No blocking calls (when configured correctly), inclusion of /etc/hosts, IPv6 support and convenient API.

2.1.2006

RR&S: Temporary files and transactional file handling

Categories: — josef @ 19:44

Temporary file handling on POSIX-ish systems is a mine field - there are a lot
of standard (yet harmful) functions available, and close to none standard (and
harmless) ones, with mkstemp() and tmpfile() probably being the exceptions.

But does harmlessness also bring usefulness? Let’s look at what a current
modus operandi of creating temporary files is. A file shall be created,
no matter what its name is, but the resulting name should be known for
continued processing - maybe, because it is registered somewhere.

The tmpfile() function only exists for anonymous temporary files, thus it
is out of question here.
The current method for solving the would thus be:
- using mkstemp(), providing a template
- calling fdopen() on the file descriptor
- registering the modified template as file name
Alternatively, using mktemp() and open() with O_EXCL would work, too.
Note that the recommendation against mktemp() due to security issues is not
valid when using O_EXCL (unless on NFS), but it might still involve an
inconvenient race.

To refine the only remaining alternative, the following library function
is proposed.


FILE *f = tmpatomic(const char *directory, char *template, int templatesize, int backingfile);

The parameters are as follows:

- directory: Directory where the final file shall be created.
- template: Modifyable string buffer which will contain the filename. If templatesize
is set to zero, template must contain a certain C format string.
- templatesize: Template buffer size in bytes, or zero for format-string.
One byte needs to be reserved for the string terminator in the former case.
- backingfile: If set to 1, the file is really created as a temporary file first,
and only renamed to its final filename after calling fclose().

This sounds a bit over-engineered, but it will actually help with the most common
use cases.
Design considerations are:
- With the help of the backingfile switch, atomicity for file creation is achieved,
without requiring the programmer to handle the rename() call.
- The flexible template handling guarantees that tempfile creation will not fail
until the maximum possible file name length has been reached, and in the case
of providing a format string (like “tmp-%s-%i") more control can be executed
over which name and random characters will be used.
Additionally, one can exclude the randomness entirely in case only transactional
file creation is wanted.
- The implementation can hide the O_EXCL problem and NFS special-casing as follows:
Both the backing file and the temp file are “safely” opened, that is, using O_EXCL
in the normal case and using symbolic link refcounting on NFS.
Since ownership of both is then guaranteed exclusively, the renaming can be
performed without interception. Of special convenience would be that renaming
could take place across partition boundaries, as to hide file system-specific
details about mount points.
Note that without any advanced file system support, the final file can only
be opened upon the fclose() call, which might then fail with EEXIST.
In this case, a facility to make the name of the backing file available would
be needed, or to tell the truth, the whole backing file concept would not
work and two distinct functions for temp file creation and renaming would
be needed again. This does not invalidate the other advantages though.
Alternatively, if the application can live with a zero-byte final file
in the case of a crash, this issue could be considered solved, too.

In summary, here are two useful applications:

tmpatomic("/tmp", “XXX", 3, 0);

- A classic temporary file

tmpatomic("/var/lib/myapp", “game-%i", 0, 1);

- A file which is created temporarily and then moved atomically to its destination

tmpatomic("/home/user", “some.conf", 0, 1);

- Deterministic transactional file creation

It is not easy to define what the root of the evil is which makes such complex
designs necessary. The inability of some file systems to “reserve” names until
the files are either created or a cleanup run is done (after a crash) is certainly
part of the issue.
Likewise, one should think out of the box to let the file naming be part of flose()
instead of fopen(): No matter where the file is created first, it should be stored
safely afterwards. This resembles the backing file principle, and from current
practice it is known, that in the case of name conflicts a routine to select
an alternative name would have to be called anyway - which could simply be another
random name in this case, but requires to give a (modifyable) buffer to a function
like tmpatomicclose().

Rants, Ramblings & Solutions

Categories: — josef @ 19:42

This is a new article series devoted to the brain-dead-ness of POSIX interfaces, Unix legacy constructs and the like. I simply felt like writing about those issues would help the cause more (i.e. cause people to think about it) than hurt it (i.e. discredit me). That being said, everything is mostly written during the night, when I’m really sleepy at times and without internet access for further validation, and does not inhibit paper publication quality.

This is a permanent entry; below the individual articles will be linked in the future. Note that I’m going to abbreviate the series title with RR&S.

7.11.2005

aLANtejo 2005

Categories: — josef @ 18:19

Este é só para dar um sinal de vida, faz tempo que eu tinha publicado algo neste blog.
Estava na aLANtejo 2005, um evento da computação no Portugal, que era dividido em uma parte LAN party (com muita gente rodando computadores com jogos) e uma parte Congresso, para o qual eu tinha um convite. Tudo isto acontecava na velha cidade de Évora, patrimônio mundial da UNESCO, conhecida pela arquitetura romana.
Dei uma pequeno workshop sobre as Técnologias KDE, e no dia seguinte eu participei no painel junto com o Juan do projeto GNOME. O meu português ficou meio ruim, mas ainda assim a gente gostou de trocar idéias, dicas e até código fonte. O evento era muito bem organizado, com hospedagem em um velho convento e comida da região. Encontrei gente das ilhas Açores e do Cabo Verde, era muito legal mesmo.
Depois, eu fui para Lisboa junto com o Alex do projeto MPlayer e a namorada dele. Viemos o centro da cidade, e eu ficei mais dois dias para conhecer outras partes, como Barreiros no outro lado do rio Tejo.

test

20.8.2005

Samstag = Debian-Tag

Categories: — josef @ 16:43

Zumindest diesen Samstag beschäftige ich mich von morgens bis abends mit dem Paketebauen. Damit ist es quasi mein persönlicher Dia-D. Es handelt sich dabei um sämtliche GGZ-Pakete, die momentan ohne Maintainer sind und damit haufenweise sich über die Monate angesammelte Fehler aufweisen. Es sind bereits 17 Pakete gebaut, aber es kommt noch eine Handvoll dazu, vor allem solche, die es bisher noch gar nicht in den Archiven gab. Auch hat sich qualitätsmäßig einiges getan, was es jetzt wieder aufzuholen gibt, beispielsweise Package Tags oder Übersetzungen der Paketbeschreibungen, ebenso das Menüsystem. Ziel ist es, die Annäherung an den Release-Prozess wiederzufinden, um die nächste Version komplett ohne Patches paketieren zu können.

Nebenher läuft nahezu selbständig der Remastering-Prozess für die TUDix 3.0-Live-CD, die - unabhängig ob mit Sponsor oder nicht - wieder im Herbst erscheinen wird. Ein paar Bugs in den Buildskripten waren schnell beseitigt, aber es sind etliche Pakete momentan nicht in sarge bzw. überhaupt nicht verfügbar. Wenn die KDE-Pakete schnell in unstable reinkommen (was sie momentan zu tun scheinen), wird TUDix also dieses Jahr auf unstable basieren.

22.7.2005

Wir sind Meta

Categories: — josef @ 23:43

Trotz gegenteiliger Prognosen scheinen sich diverse Webservices zu etablieren, auf die man frei zugreifen kann. Dabei ist frei noch näher zu definieren - es kann unbegrenzte Abfragbarkeit ebenso erlauben wie eingeschränkt auf 1000 Abfragen pro Tag oder, im Grenzfall, 100 Abfragen pro Nutzer, welcher sich mit einer Applikations-ID bzw. einem Schlüssel ausweisen muss.

Im Gegensatz zu traditionellen Diensten, die auf dem industrie-geprägten UDDI/SOAP/WSDL aufbauen und eigentlich nur von den großen IT-Firmen propagiert werden, kommen dabei in letzter Zeit immer mehr REST-basierte Dienste zum Einsatz. Diese funktionieren oft bereits durch eine HTTP-GET-Abfrage, deren URL die Parameter-Wertepaare enthält. Zurückgeliefert wird dann ein XML-Dokument oder eine andere Datenstruktur.

Sich dieser Prinzipien annehmend, existiert OpenSearch, ein von Amazon initiiertes Projekt, welches eigentlich nur ein REST-Format ist. Dabei wird analog einer WSDL die Eingabeschnittstelle anhand einer OpenSearchDescription-Datei (OSD) definiert, hingegen entfällt dies für die Ausgabe, da die Daten alle im RSS-2.0-Format ausgegeben werden.
Technisch eher unklug ist die Entscheidung, Tags wie AdultContent mit aufzunehmen. Auch die komplette Beschreibung als CDATA-Sektion inklusive HTML-Daten ist ziemlich schwer handhabbar, speziell in nicht-HTML-Ausgabeformaten. Insgesamt betrachtet ist OSD aber ein netter Ansatz, der zumindest einen Startpunkt für die automatisierte Nutzung eines Dienstes enthält.

Dies scheint auch für die Wikipedia-API ein mögliches Konzept zu sein. Da die Wikimania in Frankfurt in wenigen Tagen stattfinden wird, wird sich diese Frage schnell klären. Die Einbindung in KDE erfolgt dann nicht nur über einen speziellen Client namens Knowledge, sondern auch über den Webservice-Navigator Kung, der für das Ergebnis den RSS-Reader aKregator als KPart einbinden könnte.

Leider enthält auch die OSD in keinster Weise semantische oder auch nur beschreibende Merkmale. Diese Lücke füllt die Ergänzung dieser Dateien mit GUIDD-Definitionen, an denen ich gerade arbeite. Soweit also ein aktueller Einblick in meine Hiwi-Arbeiten.

18.7.2005

Noch mehr IPv6

Categories: — josef @ 16:36

Irgendwie laufen alle Dienste seit dem letzten Kernelupdate automatisch mit IPv6, sofern unterstützt. Das ist an und für sich ganz gut. Nur ist leider die Unterstützung dafür auf glibc-Ebene immer noch nicht voll durchdacht. Wie bekommt man den (bzw. irgendeinen) Hostnamen von dem Rechner, welcher am anderen Ende eines verbundenen Sockets sitzt?
Hier ist die Änderung im Quelltext, dafür wird es noch eine Wrapperfunktion char* ggz_peername(int s) geben.
Sehr ärgerlich ist, dass die Funktion ntohl die Unterscheidung nicht erkennt und brav als Rückgabewert 0 zurückliefert - mit der IP-Adresse 0.0.0.0, was ein connect() quasi in ein bind() umwandelt.
Ebenfalls betroffen sind alle Programme, die SDL_net 1.2.x nutzen. Es gibt schon einen alten Patch dafür, den man sogar API-kompatibel machen könnte, um nicht erst SDL_net 2.0 abwarten zu müssen, um ihn einzpflegen. Dazu später mehr (Maintainer sind kontaktiert).

6.7.2005

FFII und Slashdotting

Categories: — josef @ 22:17

Die Welt feiert heute das 1:1 der Softwarepatentgegner, nachdem uns in der ersten Halbzeit in einem undurchsichtigen Foulspiel der Befürworter das 0:1 aufgezwungen hatte. Die FFII-Mannschaft, obgleich durch die Strapazen der letzten Wochen leicht angeschlagen und in der Unterzahl spielend, gewann dennoch mit Hilfe der unzähligen und lauten Unterstützer.

Ein ganz anderes Thema ist:

jos00: [HS] Traffic-Warnung

Diese Grenze wurde gestern somit bei einer Ausnutzung von 105,6%
überschritten.

Ja was ist denn dort passiert? Nach einer unschlüssigen Denkpause, ob ich in der Nacht zuvor nicht doch wieder schlafwandelnd das eine oder andere Projekt veröffentlicht hatte, warf ich einen Blick auf die Logdatei.

mindx.josefspillner.de xxx.xxx.xxx.xxx - - [05/Jul/2005:14:35:17 +0200] “GET /kde/plasma/ HTTP/1.1″ 200 1499 “http://www.google.co.uk/search?q=KDE+plasma&…”

Aha. Der Aaron ist schuld gewesen. Denn sein Plasma ist von Slashdottern geschmolzen worden, und so zog die Horde zu Google, um auf mein Plasma zu stoßen. Blöd nur, dass die Projekte beide nichts miteinander zu tun haben, auch wenn beide KDE-bezogen sind. So habe ich zur Un-Bildung von ein paar Slashbots beigetragen. Rache ist süß.

3.7.2005

Webservices auf der Kommandozeile

Categories: — josef @ 23:39

Für den Webservice Desktop Exchange Service habe ich in Perl einen kleinen Kommandozeilenclient geschrieben. Er ermöglicht es, publizierte Datensätze anzusehen bzw. herunterzuladen sowie selbst Daten zu publizieren. Mit wenigen Modifikationen sollte er für Flickr, Google und dergleichen nutzbar sein.

Einige Operationen, die stateless verlaufen, sind dabei direkt über die Kommandozeile abrufbar.

$ ./wsclient.pl –information
Provider: KStuff Development Server
Server: Desktop Exchange Service (DXS) (Version 0.2)
URL: http://kstuff.org/

Diese unscheinbare Zeile hat schon SOAP::Lite, Getopt::Long und XML::DOM gequält. Richtig interessant wird es aber dann, wenn man es ohne Argumente aufruft - dann landet man in einer kleinen Shell. Genauer gesagt ist es eine Term::Shell.


$ ./wsclient.pl
Webservice is set to: http://localhost/cgi-bin/kstuff-ws.pl
dxs> categories
Category: ccheckers/theme
Category: cchess/theme
Category: hastings1066/level
Category: image/unknown
dxs> list image/unknown
Entry: (10) Staunton (Asif Ali Rizwaan)
Entry: (6) Gtk+ Default (Jonas Ström)
Entry: (7) Dragon (Jonas Ström)
dxs> details 6
Name: Gtk+ Default

Preview: http://localhost:10000/hotstuff/directory/image/unknown/preview_defaultgtk.png

Nimmt man nun noch LWP::Simple hinzu, so kann man die angegebenen Dateien gleich betrachten. Wobei diese Implementierung nur vorläufig ist - später soll dies natürlich über den Webservice selbst ablaufen.
Interessant ist noch das Cache-Concept - alle Daten, die irgendwie gespeichert werden können, werden es auch - das fängt bei der Benutzer-Authentifizierung an (dank HTTP(S) braucht man sich darum nicht groß zu kümmern), und umfasst auch alle übertragenen Daten.
Jetzt fehlt nur noch eine schicke GUI. Aber die kommt schon bald…

1.7.2005

Organising my slides with Dispositionator

Categories: — josef @ 17:32

Yesterday in the evening I asked around on #debian.de whether anybody knew a tool which, although not easy to define, would be something in between an online picture gallery and a CMS. What I tried to accomplish was the catalogisation of my talks which I give occasionally, together with slides and papers, optional translations and various derivates of different formats, such as the HTML export. In addition, the tool should allow sorting and filtering the entries, and it should be easy to integrate it into existing webpages, which more or less makes templates obligatory.

Well, just a few hours into the idea (no, I didn’t sleep much last night) the release 0.1 of Dispositionator saw the light. It is only the 2nd Cool Projects newcomer this year (after TarDiff), and I’m glad that finally I had time to hack something on my own again.
Technically, a lot of details could be improved, but even the first version already does a pretty good job. Support for export and import of XML data with Pentabarf comes to mind and, equally interesting, a desktop frontend for integration with KPresenter and the like.

The origin of the name is influenced by Pyntor, a presentation tool written in Python. The only logical choice of name had to include a latino word (be it Spanish or Portuguese or even both) and it should express that I want to make my slides available to everyone in an easy way. Voilà, making them disponível is what the intent was about.

An awkward detail was that this time I forgot to set the licence in the freshmeat announcement and it was set (automatically?) to “Free to use but restricted". This, sir, is not quite the truth :-)

18.6.2005

Wunschliste

Categories: — josef @ 10:01

Meine Unix-Basissystem-Wunschliste hier mal kurz aufgeschrieben. Es sind (im Gegensatz zu meiner 10-Punkte-Linux-Rant-Liste) keine allzu abwegigen Ideen dabei, sondern durchaus praktische Änderungen, die ich dringend benötige.

Ich wünsche mir, …

  1. dass Subversion nicht nur Verzeichnisse über svn:externals in verschiedenen Modulen gleichzeitig verwenden kann, sondern auch einzelne Dateien.
  2. dass ich als normaler Nutzer Privilegien wie Netzwerkzugriff, Dateizugriff schreibend/lesend pro Verzeichnis, etc. aufgeben darf.
  3. dass Software, die mit Zeichenketten hantiert, deren Inhalte nicht mehr auf 7-Bit prüft, sondern maximal noch die unteren 32 ASCII-Zeichen ausschließt - auch hier ist Subversion schuldig, aber auch viele andere.

Nun brauchen wir nur noch eine gute Geek-Fee.

26.5.2005

Me, the sysadmin: Sarge upgrade

Categories: — josef @ 20:40

I’ve upgraded my server to sarge. It was pressing for some time, and needing subversion as I convert the KDE CVSfilter to Commitfilter I simply had to do this. (Note - one of the two links won’t work depending on your DNS. Good luck.)

So I started:

$ uname -a
Linux vs011 2.4.26-1um #4 Tue Apr 20 17:09:55 PDT 2004 i686 unknown
$ uptime
17:59:20 up 305 days, 8:09, 1 user, load average: 0.51, 0.43, 0.38

Not bad, eh? And then: vim /etc/apt/sources.list; apt-get update; apt-get dist-upgrade
The transition went smooth, although there are some oddities, and I’m going to list them here.

  1. when updating python 2.1 to 2.3, the pygame package is removed instead of updated
  2. there are conflicts where there shouldn’t be: proftpd vs. vsftpd, inetd vs. xinetd - it’s perfectly fine to have them installed side by side, no need to let them conflict!
  3. all the small cron jobs, which are cluttered around and asked for conffile change permission, should go away
  4. the helper scripts should dismiss all keyboard input before displaying their questions. This is a real programming bug
  5. somehow related to xinetd, it disabled some services in inetd.conf but not all of them. Huh?

Of course I disabled all services before the upgrade (except for SSH), and I had to reenable them afterwards. The conversion of exim3 to exim4 was painless. Also, Apache continues to work as expected. The biggest fish, mailman, is yet to be done, as are the databases. But since I wanted to work with subversion, here we go:

svn co svn://svn.kde.org:/home/kde/trunk/KDE/kdelibs

And it just hangs and does nothing. No timeout, no message, no network traffic - nothing. Damn. With anonsvn, it tells me: connection refused. Damn, again.

17.5.2005

Artikel und Projekte

Categories: — josef @ 15:28

Dieser Eintrag soll mal eine Übersicht über die Artikel geben, die ich irgendwo veröffentliche. Es lohnt sich dann, aller paar Wochen mal wieder vorbeizuschauen, denn meist werden noch ein paar Kommentare hinzugefügt.

Rechnerzoo

Aktuell:
Artikel-Serie: Rants, Ramblings & Solutions

SILC-Tutorial, Teil 1
SILC-Tutorial, Teil 2

Vom FISL zum LinuxTag (pro-Linux, 16.06.2005)
Eine Million ‘PC Conectado’ ohne Windows (pro-Linux, 14.05.2005)
FLISOL, die größte Linux-Installationsparty der Welt (pro-Linux, 07.04.2005)

Auch ein paar kleinere Projekte, die keine reinen Softwareprojekte sind, sollen hier mit aufgelistet werden.

FISL 6.0 (deutsche Übersetzung)

Deutschsprachiger Aufruf zur Unterstützung von Sérgio Amadeu (mit über 11000 Unterschriften im Original)

Eigentlich würden auch Vorträge hier hinein gehören. Aber mal sehen, bis wann ich das Chaos organisiert bekomme :/
Erstmal eine Liste:

GGZ Gaming Zone (LinuxTag Karlsruhe, 25.06.2005)
Open-Source-Projekterstellung am Beispiel KDE (Tag der Fakultät Informatik Dresden, 23.05.2005)
Verteilte Versionskontrolle (C3D2 Themenabend, 22.04.2005)
L4-Anwendungsentwicklung (Chemnitzer LinuxTage, 06.03.2005)
Free Software in South America (21C3 Berlin, 27.12.2004)
Internationalisierung (LUG-DD, 24.11.2004)

17.4.2005

So viel zu tun

Categories: — josef @ 18:14

Von meiner Seite her gab es eine Vortragsausarbeitung für den LinuxTag inklusive Paper, 10 Seiten lang, naja knapp darunter aber es ist ja noch nicht die finale Version. Momentan hat KDE noch gar keinen Stand reserviert, das ist jetzt sofort dran. Aaron Seigo hat einen neuen KNewStuff-Download-Dialog entworfen und Enrico Ros hat ihn erweitert und eingebunden. Der Tag der Fakultät fordert ebenfalls Tribut, das ist dann noch ein Vortrag, bzw. ein kleiner KDE-Workshop. Und für die aKademy 2005 sollte ich auch noch was einreichen. Senko hat jede Menge Vorstellungen eingebracht, die alle noch umgesetzt werden wollen. Pierre hat heimlich über Nacht mein CVS in ein SVN konvertiert, natürlich nur als Kopie, aber trotz des noch nicht erfolgten Sarge-Releases komme ich wohl nicht umhin die Umstellung jetzt abzunicken. Ricardo hat mir einen CVS-Zugang gewährt, dort werde ich meine bisher manuell zugesendeten Patches für das Buildsystem einpflegen. Mit Caio habe ich über dessen Theme IRL (In Real Life) diskutiert, und vorgeschlagen, das Format mit dem von ksirk identisch zu halten, das wird allerdings nicht trivial werden. Der C3D2-Themenabend ist ebenfalls demnächst, und dort gibt es nicht nur einen Vortrag sondern auch ein neues, in Pygame geschriebenes Vortragstool. Trotz all dieser Dinge hab ich heute noch eine Stunde Zeit gefunden, mein Polnisch etwas aufzubessern. Co to jest? To jest studentem. Na wenigstens etwas.

2.4.2005

SILC-Tutorial, Teil 2

Categories: — josef @ 10:01

Nachdem im ersten Teil das grundlegende Quelltextskelett erstellt wurde, geht es heute darum, bis zum Senden und Empfang der ersten Nachrichten zu kommen. Zur Vorbereitung legt man eine weitere Variable vom Typ SilcChannelEntry an, diese braucht man später, damit die letzte empfangene Nachricht einem Kanal zugeordnet werden kann, sowie eine SilcClientConnection, die für die gesamte Sitzung gilt.
Die weitere Entwicklung orientiert sich am Callbackmodell der libsilcclient. Nach der Eingabe der Passphrase, die auch leer sein darf (siehe Teil 1), wird zuerst der öffentliche Schlüssel des Servers überprüft. Es scheint, als müsse man im Callback silc_verify_public_key gar nichts weiter tun, denn nach ein paar Sekunden läuft der Prozess sowieso weiter, aber im Sinne des Erfinders ruft man die mitgelieferte Callback-Funktion auf:

(completion)(TRUE, context);

Als Warnung sei gesagt, dass sämtliche Überprüfungen in diesem 2. Teil des Tutorials wegfallen, denn die dahinterliegende Kryptographie ist noch einmal einen Teil für sich wert. Übrigens wertet auch Silky zwar den Schlüssel aus, liefert aber immer einen Erfolgscode zurück.
Als nächstes wird die Funktion silc_get_auth_method angesprungen, die anfragt, ob die Anmeldung des Nickname im Gastmodus, oder als registrierter Nutzer mit Passwort (wie im IRC) beziehungsweise unter Verwendung des öffentlichen Schlüssels stattfinden soll. Der Einfachheit halber sei hier mal die erste Methode verwendet:

(completion)(TRUE, SILC_AUTH_NONE, NULL, 0, context);

Und dann steht auch schon die Verbindung, allerdings sollte man in der entsprechenden Funktion darauf abprüfen, dass der Status SILC_CLIENT_CONN_SUCCESS ist, anderenfalls ist die gesamte Session über silc_client_close_connection abzubrechen . Der Server sendet dann jede Menge Benachrichtigungen, Statistiken und die Meldung des Tages (MOTD). Je nach SilcNotifyType kann man diese also ignorieren oder anwendungsspezifisch auswerten. Sobald das SILC_NOTIFY_TYPE_MOTD eingetroffen ist, kann man übrigens über

silc_client_command_call(client, connection, NULL, “JOIN", room, NULL);

den gewünschten Kanal betreten. Mit dem Meldungstyp SILC_NOTIFY_TYPE_JOIN ist auch dieser Vorgang abgeschlossen. Auf jedes abgesetzte Kommando bekommt man übrigens sowohl silc_command als auch silc_command_reply zurück, so dass man sowohl Fehler erkennen als auch zentral die Kommandos loggen kann, die ja von verschiedenen Punkten der Anwendung aus abgesetzt werden könnten.
Nachrichten kommen nun über die Callbacks silc_channel_message und silc_private_message hinein, während die Ausgabe erfolgt über:

silc_client_send_channel_message(client, connection, lastchannel, NULL, 0, token, strlen(token), TRUE);

Und das war es auch schon. Weitere Aktionen sind alle über die verschiedenen API-Varianten von silc_client_command_call durchführbar, die man ohne Slash (wie normalerweise im SILC-Client) angibt.

Vielen Dank auch an Astro für das Korrekturlesen :)

30.3.2005

Statistiker

Categories: — josef @ 16:05

“In der mathematischen Statistik macht man überhaupt häufig die Erfahrung, daß 4 schon eine große Zahl ist.”

(Prof. Dr. B. L. van der Waerden, Mathematische Statistik)

Es geht hierbei natürlich um Konfidenzintervalle, aber für sich allein gestellt ist das schon ein fortune-cookie wert. Interessant auch der folgende Satz aus demselben Buch:

“Gerade die Schwänze können unter Umständen für die Beurteilung einer Abweichung von der Normalität entscheidend sein.”

Gemeint sind hier die beidseitig extremal werdenden z-Werte einer angenäherten Normalverteilung.
Erstaunlich gutes Buch aus den 50er Jahren, 3. Auflage 1971, mit vielen Beispielen aus der Biologie (Lethalitätsabschätzungen inbegriffen).

26.3.2005

SILC-Tutorial, Teil 1

Categories: — josef @ 18:02

SILC ist im einfachen Sinne betrachtet ein Chatprotokoll, ähnlich wie IRC. Allerdings achtete man beim Design auf die Vertraulichkeit der Gesprächsinhalte, so dass die Kommunikation nun verschlüsselt abläuft, ähnlich wie bei GGZ.
Man kann über eine Bibliothek namens libsilcclient diese Funktionalität in eigene Anwendungen mit aufnehmen. So soll beispielsweise unser Chatbot die Möglichkeit bekommen, über SILC die Leute aus dem c3d2-Channel zu ärgern. Vor allem Mandy hat es ihm ja angetan.

Nun, wie geht man als C-Programmierer vor? Zuerst bindet man zwei Headerdateien ein, die jede Menge Systembibliotheken schon enthalten, also stdio, unistd, TRUE/FALSE und dergleichen. Das spart Schreibarbeit.
Desweiteren benötigt man ein globales SilcClient-Objekt. Dies ist eine Struktur, die noch initialisiert wird. Als dritte Sache im globalen Namensraum werden noch (möglichst statische) Callback-Funktionen benötigt. Dazu später mehr, man sollte sich jedoch schon mal die Datei client_ops_example.c aus den SILC-Quellen anschauen.

In einer Funktion, die die Verbindung herstellt, legt man nun zuerst ein paar weitere Objekte auf den Stacks, und zwar von den Strukturtypen SilcPKCS, SilcPrivateKey und SilcPublicKey. Dann allokiert man sein globales client-Objekt über:

client = silc_client_alloc(&ops, /*&params*/NULL, NULL, silc_version_string);

Die Operationen sind dabei vom Typ SilcClientOperations. Diese müssen nur dann vorinitialisiert sein, wenn nicht die oben genannte Beispieldatei herangezogen wird. In meinem Fall hingegen werden etliche Funktionspointer nicht benötigt. Da die API-Dokumentation leider verschweigt, dass diese nicht NULL sein dürfen, habe ich sie mit einem speziellen Präfix belegt (no_silc_…), so dass die ops-Struktur damit initialisiert werden muss. Der zweite Parameter sollte zu Beginn immer NULL sein, man kann hierüber später nette Einstellungen wie die Darstellung des Nicknames vornehmen. Die Variable silc_version_string ist global verfügbar.

Die Client-Variablen username, hostname und realname müssen anschließend gesetzt werden. Es ist darauf zu achten, dass all diese Felder auch nach dem Verlassen der Funktion verfügbar sein müssen, also empfiehlt sich ein strdup(), welches zum Schluss wieder mit free() gradegebogen wird.

Der nächste Punkt betrifft die Kryptographie. Hier ist Handarbeit angesagt, aber zum Glück gibt es Standardfunktionen:

ret = silc_pkcs_register_default();
ret = silc_hash_register_default();
ret = silc_cipher_register_default();
ret = silc_hmac_register_default();
ret = silc_create_key_pair(NULL, 0, NULL, NULL, NULL, NULL, &pkcs, &pubkey, &privkey, FALSE);

Die Rückgabewerte sind dabei als boolesche Variablen aufzufassen, sollten also nicht Null sein. Man kann natürlich alle Ergebnisse mit OR verknüpfen und dann auf Nullverschiedenheit prüfen.

Nun hat man auch pkcs, pubkey und privkey, welche man ebenfalls den gleichnamigen Feldern des globalen silcclient-Objektes zuweist. Und nun geht es los auf die Datenreise:

ret = silc_client_init(client);
silc_client_connect_to_server(client, NULL, 706, hostname, NULL);

Dies alles ist für eine Konsolenapplikation gedacht, d.h. die Passphrase muss man auch beim Start der Anwendung eingeben. Graphische Anwendungen wie silky nutzen die SILC-PKCS-Funktionen zum Auslesen dieser Werte aus einer Datei, und Eingabe über Dialogfelder über diverse Callbacks. Das soll hier aber noch nicht Gegenstand des Tutorials sein, denn für heute reicht das obige sicherlich erstmal zu.

Beim Compilieren muss darauf geachtet werden, dass man sowohl gegen die libsilcclient als auch die libsilc linkt. Besonders bei Plugins/Bibliotheken sieht man dies nicht auf den ersten Blick.

Weitere Informationen findet der geneigte Hackspecht in der SILC-API-Dokumentation.

17.3.2005

Kommandozeilenproblemanalyse

Categories: — josef @ 09:13

¿Warum nur ist es nach wie vor so schwierig, aufbauend auf ungepatchter Software komfortabel zu arbeiten? Der Kernel hat zwar ein wenig Unterstützung für UTF-8 in der Eingabe und Ausgabe auf virtuellen Terminals, aber ein Modell, das einfach funktioniert, hat er nicht. Stattdessen darf man folgendes zur Bootzeit im Userland erledigen:

unicode_stop
unicode_start
loadkeys de-latin1
consolechars -f lat9u-16.psf

Und auch das ist nicht perfekt, denn abgesehen von fehlenden Zeichen in dieser Schriftart gehen dabei die Akzente verloren. Jede Distro hat aber ihre eigenen Ideen und Patches, und Sonderwege sowieso.
Wer so richtig coole Sachen machen will, muss sich wohl so oder so Alternativen suchen.

15.3.2005

Networking without networking source code

Categories: — josef @ 15:38

An idea I had back in 2002 already, but only now is implemented so that it actually works:
Instead of sending bytes or strings in one application over the network and reading them in another application at the other side of the world, the protocol is given as an XML file and the corresponding source is generated from it, and included into the program. Using scripting languages, this inclusion could even be done at runtime.

Recently I played around with XSLT so the network protocol description file can be turned into a HTML file which documents the protocol. Cool stuff. Using external documentation references, this would even work for internationalized documentation, something which is not yet easily doable with doxygen, for instance.

Another advantage would be to add a debug mode. Even when using an efficient binary protocol, the generators could (on both sides) decide to spit out plain text so debugging can easily be done. In the end it could even send out itself as a protocol!

2.3.2005

Cross-compilation

Categories: — josef @ 15:47

Yay for cross compilation.

clt2005@bizarre:~$ i586-pc-linux-gnu-gcc -v
Using built-in specs.
Target: i586-pc-linux-gnu
Configured with: ../gcc/configure –target=i586-pc-linux-gnu –prefix=/home/clt2005/x86-usr –enable-languages=c,c++
Thread model: posix
gcc version 4.0.0 20050302 (prerelease)

This required about 15 changes in the source code, make files and configure scripts. I also tried installing GCC 4.1pre but this failed horribly. Now on to using the compiler…

25.2.2005

Launch of easycnm

Categories: — josef @ 17:00

The easycnm library and an accompaigning website has been launched today. Students who are in need of easy-to-understand numerical algorithms should check it out at http://www.inf.tu-dresden.de/~js177634/easycnm/.
I’ll certainly work on it some more during the next days.

20.2.2005

Running QEmu

Categories: — josef @ 00:14

I never really toyed around with QEmu. Today I gave it a try… expecting it to deliver a running L4 system (which is compiled for x86) on my PowerPC notebook. It boots up without problems, and using full-screen mode gives a nice system emulation. However, its graphical capabilities are not better than those of Bochs - neither the standard VGA nor the Cirrus Logic adapter emulation seem to work for me.


L4 running in QEmu

17.2.2005

Desaparecido na Bolivia

Categories: — josef @ 18:34

Que coisa… uns colegas meus tiveram problemas sérias quando viajando na Bolivia. Na página de Tobias (em alemão) pode ler o que aconteceu. Até a embaixada já se preocupava… ae rapaz cuidado na natureza, não está feito para geeks comuns :)

Valeu Manja, agora tá melhor… :)

16.2.2005

6. FISL

Categories: — josef @ 20:55

Anfang Juni wird wieder das Fórum Internacional de Software Livre (Internationales Forum Freier Software) in Porto Alegre, Brasilien, stattfinden. Um die Internationalität zu erhöhen, betreue ich ab sofort eine deutsche Übersetzung der
Webseiten des 6. FISL.
Der Call for Papers läuft übrigens noch bis 28. Februar.
Untere anderem war das Weltsozialforum in Porto Alegre komplett mit Freier Software betrieben worden.
Es werden sicherlich auch wieder jede Menge Politiker aus Südamerika und sonstwo her anreisen - aber mindestens doppelt soviele Hacker.

13.2.2005

Eröffnung

Categories: — josef @ 04:22

Nach ein paar Monaten Auszeit habe ich mich entschieden, doch wieder unter die Blogger zu gehen. Aufgrund der Prüfungszeit wird allerdings nicht allzuviel hier zu finden sein. Eher suboptimal finde ich die Kategorienaufteilung von WordPress, da die Sprachen orthogonal zu den eigentlichen Kategorien stehen. Es wird jedenfalls auf (portugiesisch, englisch, deutsch) gebloggt werden, und die Themenliste richtet sich nach meinen Projekten.
Auf der Domain kuarepoti-dju.net werden demnächst auch noch andere Dinge zu finden sein.

11.2.2005

There we go

Categories: — josef @ 02:11

After having had to stop my Brazil photoblog,
the idea of continuing some sort of written outlet did not want to disappear, and thus I’m now trying out wordpress to brag about the categories desktop, games, society, local news and miscellaneous.

Powered by WordPress