There is a reason that Debian is being singled out, which you ignore - Debian likes to add lots of distribution-specific patches to everything they package. Other OSes/distributions are not as problematic, because they are much more likely to ship software as the author wrote it (unless it's ridiculously broken and unmaintained and important.)
Also: why should software authors have to write bug reports to Debian?
With 1000's of packages, and 1000's of maintainers, saying that "Debian" likes to patch "everything" is a pretty big claim.
A lot of the patches that do go in make whatever package integrate better with the rest of the system.
> Also: why should software authors have to write bug reports to Debian?
Because there's a bug?
> "Even though the piece of software I asked aptitude to install requires these libraries to run, they refuse to install it."
At first glance, that sounds like a packaging bug to me, with Rubygems, not with one of Zed's packages. The best thing to do in those cases is report the bug, rather than attempt to "make Debian pay" (for the thousands and thousands of hours of work to give you a free operating system, presumably?)
I don't use rubygems myself so I'm just commenting as a spectator. But I think it's very reasonable to ask that if someone (anyone) discovers a bug, then the right thing to do is to report it. They don't have to fix it themselves; maybe that responsibility falls on the rubygems package maintainer. But the maintainer will never know there's a bug until someone reports it.
Yes, it does seem to be a critical and obvious-to-catch bug. But as davidw said, maybe the maintainer already had the prereqs installed, maybe they were having a bad day, and they just ran a quick test and saw that it worked and released it. Mistakes happen, and I'm sure the maintainer would be happy to fix those mistakes, but they can't be fixed if the maintainer doesn't even know about them.
On a side note, I agree with Zed's rant on Debian being unique with its configuration and packaging layout. I use Ubuntu on my work laptop and so I've gotten used to the different filesystem layout for config files, etc. When it comes time to set up a server, am I going to go with a different distro and remember which distro has which config file in which location, or am I just going to stick with Ubuntu server because I already know it? I don't like having to make that choice.
Debian does have general policies and a culture. (Yes, we could say it's just a bunch of files on a bunch of FTP servers, but we can agree that that's stupid, right?)
As to the bug report thing, let me clarify: why should upstream authors have to pay the support cost (in e.g. time and negative impressions of them) for bugs introduced by Debian's packagers?
> why should upstream authors have to pay the support cost (in e.g. time and negative impressions of them) for bugs introduced by Debian's packagers?
They are just as capable of saying "that's a Debian bug, go file a bug report there". That's a small price to pay for being open source, in any case. Debian certainly picks up bugs that are intended for upstream, and there are certainly instances when Debian maintainers have been quite helpful with fixing them. And others where they haven't been. With that many people there are bound to be some really good ones, and some that are below average...
"That's a Debian bug" is not useful - you can only determine that after you have analyzed the bug thoroughly, and gone to the length of getting out a Debian system instead of whatever you're usually developing on.
But yes, some maintainers are very good and some are quite bad; however, Debian's policies and patch-happiness means that bad maintainers (can/tend to) do much more damage than in other distributions/OSes.
Well if you really want to play tough about it you just say that all bug reports for your software running on Debian must go through the package maintainer. You could even ignore them if you get a lot of bogus ones.
Yes, but even that is a half-solution - Debian users are still likely to think you suck and/or use alternatives, which means less users/prospective developers/testing.
Why should the author report a bug when it's the incompetence of the Debian maintainer?
The 'gem' command not working after installing the 'rubygems' package does seem like a bit of a bug, to put it mildly, you'd think a cursory test by the maintainer would catch that.
> Why should the author report a bug when it's the incompetence of the Debian maintainer?
Because he cares about the bug being fixed, rather than just whining? The point is to focus on fixing the system, not on pointing fingers or telling eachother to fuck off.
The maintainer most likely already had the ruby-ssl package, and so the 'cursory test' worked for him.
Look, I don't think the rubygems package on Debian is a fantastic bit of work, but ultimately:
It's free software. If you don't like it, either help fix it or don't use it. Point it out if you like, but don't be such a dick about it.
The author put mongrel out there under a free license, as well as some unnamed gem, and various other things. The gem is not included in Debian. As to the "benefits" of Debian including anything, they certainly aren't pecuniary, but accrue mostly to the end users (or not, in this case). So, if you help Debian fix something, the beneficiaries are ultimately all the users of Debian, not really 'Debian' itself.
> Mac OS X - Fine. FreeBSD - Fine. RedHat/CentOS - Fine. SuSE - Fine.
Any sufficiently complex system, including every one of the above, has problems. They may differ from Debian's in type, nature and even quantity, but they exist. Including missing commands, disabled functionality, and misconfiguration.
Why should the author report a bug when it's the incompetence of the Debian maintainer?
1. Its the polite thing to do.
2. The bug can be tracked and commented on.
3. The bug is more likely to be fixed.
4. Thats how most open source projects work.
I've not used Homebrew on the Mac, but it does look like an promising alternative approach. If you don't like something, you can fork the distro, fix the bug and put in a pull request yourself.
This. The point of Gems and Maven and the CPAN client and all this other crap is to smuggle code into a production box while carelessly failing to get it into the platform's database of what's installed where and what depends on it. They replace one tool that knows everything that's going on, with many tools each of which can't.
Somehow we got stuck with a critical mass of small-scale developers who never faced the nightmare of somehow keeping the right tarballs on hundreds of machines in different roles without good tools, so now we all get to relive it.
My response remains the same: Debian (or what-have-you) is but one distro among many, and those of us that treat such things as important but fundamentally commodity deployment platforms have no wish to spend our lives babysitting whatever package manager happens to be on the operating system we're using that month/year/whatever. Further, plenty of people have plenty of good reasons for deploying on Windows and OS X, so using tools that work well there make sense.
The priorities of the various linux distros aren't everyone else's priorities – which somehow seems to come as a galloping shock to some.
That is why I use Slackware. Installing new software by hand from tarballs is a bit tedious compared to package managers, but it is transparent, and when I am done I can be pretty sure it is going to work. I have tried other distros, even Debian once (for some reason the Debian wouldn't let me sign in as root, I never did figure out why), but I keep coming back to Slackware - for its transparency.
And Slackware's packaging is light enough that it is fairly simple to write wrappers around gems/CPAN modules/Python eggs. The latter two have prewritten SlackBuilds that you can use, assuming that the setup.py is not doing anything too shady.
With CPAN (and local-lib and/or perlbrew) I can have per-user libraries, per-application libraries, and it doesn't interfere at all with the dependencies of the system. If a library that both my application and the system use, and they fix a bug that includes an incompatibility, I can upgrade my applications without fiddling with the system, I don't need to take care to break APT in the process (fun times). I'm not even talking about possibilities like copying someone else's local-lib to my own environment to help them debugging, while still having my own setup stay untouched.
So, I guess I disagree. These tools provide a lot more than the usual package management.
To me it seems backwards to try to manage libraries globally when their use-cases are often already localized.
This has been done, by several people. The problem is that the resulting packages don't fit with Debian policy, and in a few cases result in file collisions. If we want to mend the situation, we need both sides to agree that the solution is suitable.
The Right Way to do it, in my mind, is to take a snapshot of the available gems, and convert only those versions of the gems to debs such that rubygems and the Gem constant aren't involved at all. This means that "gem 'foo'" gets thrown away (or replaced with a check that the correct distro package is installed), using the Gem constant is a blocking bug, and "require 'rubygems'" is at least a no-op. Files from lib/ should be installed to /usr/lib/ruby (or at a pinch site-ruby), so that "require 'foo'" just works. Anything that assumes File.expand_path(__FILE__,"..") is under that library's control is again a blocking bug (I'm looking at you, haml).
I know there are people working on this, so it's not entirely theoretical, but besides, these are all good ideas anyway.
> The Right Way to do it, in my mind, is to take a snapshot of the available gems, and convert only those versions of the gems to debs such that rubygems and the Gem constant aren't involved at all.
If you think that's going to work, you don't know enough about the Ruby library ecosystem. You can't take a snapshot of the available versions and think that'll satisfy users. Users need to be able to install different versions of the same package in parallel, tons of things rely on this. Take a look at the rationale behind Bundler: http://gembundler.com/rationale.html
Sadly dpkg doesn't support multiple package versions.
Oh, I know about the Ruby ecosystem. Having to rely on multiple versions of the same library is a bug, not a feature. People do it because the tool lets them, not vice versa.
tl;dr dpkg models this just fine, thank you very much ;P
I don't feel it is accurate to say "dpkg doesn't support multiple package versions"; let's say it let you install multiple versions of the same package at once: now you are just going to have file-on-file conflicts. From dpkg's perspective these different versions of the same files, which live in different places on the filesystem and can coexist on the same system, are just different packages.
The real question I feel the Ruby community should be asking, and thankfully the very question that I have personally heard from Yehuda, is "why doesn't C feel the need to do this?". The answer is "well, they do, but".
For the "well, they do" part, we can easily call into evidence that on my system I have both libreadline 5.x and libreadline 6.x: there is a (hilarious) fundamental break between these two versions of the library, leading to them deciding to increase the major version number, which then causes different applications to need to continue linking to one or the other.
Therefore, Debian models this: these two major versions of readline are each a separate package, libreadline5 vs libreadline6. You can install them both at the same time: they do not conflict.
Now for the "but", where the question becomes "what if I want to install 6.0 and 6.1" and the answer is "you don't want to do that, no one wants to do that, that is not something you should want to do", and this, I feel, is where the conversation starts breaking down with the Ruby community.
The reason no one wants to install 6.0 and 6.1 is that there is no reason to: 6.1 is newer and better than 6.0. Meanwhile, software that currently works with 6.0 /will/ work with 6.1. If software that worked with 6.0 didn't work with 6.1 it would not be called 6.1, it would be called 7.0.
Now, exactly whether the major version or the second-most major version (libxml2 uses this) or whatever other scheme someone comes up with is used to determine this isn't important. The way it works for C is that the version you are using gets fixed at compile time, which I think people should actually start thinking as analogous to creating their Gemfile.
Here's how that works (for Linux gcc with typical paths): the developer of a project passes -lreadline to the compiler. This looks for /usr/lib/libreadline.so, which in turn is a symlink to a file which is named with the "compatibility version" (in this case, 5 or 6), which in turn is a symlink to the specific version of the library in question (although there might be some more levels if someone is being silly).
Then, every binary has in it an "installed name" or "soname" which is the path to the library that will be encoded in any compiled binaries that have linked against it. This name is based on the compatibility version, and not the specific version, as there is an assumption that versions of the library that have the same compatibility version are, by definition, compatible.
So, this means that even though this symlink scheme theoretically supports having 6.0 and 6.1 installed at the same time, it would have no effect: all software wanting 6.x will be linked against libreadline.so.6, which will in turn only be able to point to either 6.0 or 6.1.
This assumption is a good thing: it means that we can upgrade libraries. People should not be wondering whether upgrading to 6.1 will break their program: if it does then the person who released 6.1 is not doing their job right, and there is a bug in that library that needs to be considered critical.
Luckily, system integrators such as Debian go to great lengths to test and verify that these libraries really are as compatible as they claim. This sometimes gets them into really hot water, however, as if upstream goes nuts and starts breaking their ABI, even slightly, Debian either needs to fix that bug or not take the update: it is not acceptable to Debian to allow an incompatible release of readline into the ecosystem, as it will have unpredictable consequences on the software that is using it.
That assumption is really important: there is no "we came really close, but some random things have changed that you now need to go fix". The idea in this world is that there is no good reason for that: there could be an arbitrarily large number of people who are relying on that feature, so it is irresponsible to unilaterally decide that you can just change it, even if it makes more sense.
This belief that "incompatibilities are serious bugs" leads to a number of common C idioms. To draw a few examples: * rather than change old APIs, add new ones; * rather than changing structures every now and then, use version numbers to tell different users apart; * use C for public external interfaces over C++ (which tends to have ABI fragility problems on most platforms) whenever possible.
Given this belief, if libreadline6 comes out, you also would never even consider using it in place of libreadline5 without going back to the source code to try to verify if it works (at least compiling it, but there may also be semantic issues at work in the API). With this belief in mind the idea of using ~> (which Yehuda has to spend a lot of time convincing people of) becomes "obvious": if a major version number (which unfortunately is not what ~> correctly models, but at least it is better than nothing to get the idea out there) changes, your package should not assume that will work.
You also find yourself being really happy that there is someone in the ecosystem--Debian in this case--who is doing all of that really hard, incredibly grueling, and (apparently) often thankless job of standing at the gates making certain software doesn't just willy-nilly enter the ecosystem until it has been reasonably regression tested. The developer's goal is, sadly, not always aligned with the grand unified vision, and that's what the users of these systems are buying in to (and I'll even go so far as to say: and that is where most of the value is, not the individual software projects).
Ok, </rant>. ;P (If anyone is curious: the reason libreadline5 and libreadline6 are incompatible with each other is actually that 5.x is GPLv2 and 6.x is GPLv3; afaik, and I'll admit I might be wrong, not only is the API between these two major versions identical, but so is their ABI: only in the world of politics and legalities was there an interface break. It still makes a simple example that a lot of people have run in to, though. ;P)
> This has been done, by several people. The problem is that the resulting packages don't fit with Debian policy, and in a few cases result in file collision
Then, by definition, it has not been done - it has been attempted and the problems proved hard.
I think the best way to go about it is leave the Debian Ruby subsystem alone and either go the virtualenv route or install it from the source tarball and place it in /usr/local or /opt
Why do people keep representing this as bizarre when C libraries parallel install just fine and you probably all have multiple versions of the same library installed right now?
Accept it's a legitimate situation please. What it needs is radical change in languages like ruby/python/etc to resolve dependencies explicitly at runtime. Currently they hide the problem with import path priorities.
Multiple versions of a library are certainly possible with C, but are generally avoided on Debian where possible. Where that isn't possible -- for instance, for libraries which completely changed their API like gtk1.4 -> gtk2, they simply create completely separate packages for the two versions.
With gems, this isn't quite so easy, as there's no way to distinguish between a minor bugfix and a huge API-breaking change without either guessing based on the version number or having a human read the changelog. Even then, there's no guarantee that dependent packages will depend on distinct sets of versions.
The naive way will just load the highest-numbered version you have installed:
require 'rack'
When you have rubygems loaded, this is equivalent to:
gem 'rack', '>=0'
You can specify exact and relative dependencies easily. There are a fuckton of frameworks to handle this for you, many of them oriented around copying the depended-upon gems into your app when it's deployed, as is common in the Java world.
Because the gem format probably doesn't have a proper way to say "you must install OpenSSL such-and-such"? At least CPAN and Python's eggs don't have one...
OpenSSL is an interesting case, because it's part of the standard Ruby library, and it should be available in every single copy of Ruby. But Debian faces export restrictions on OpenSSL, so their copy of Ruby is missing a number of important standard libraries by default.
Rubygems can't really clean this problem up, because Rubygems assumes it's running on an actual copy of Ruby, and not a half-broken Debian Ruby.
To get a working Rubygems installation on Debian, I usually 'apt-get install ruby-full', which installs a non-broken Ruby interpreter, and then install Rubygems itself from a tarball so that apt-get stays _far_ away from my Ruby setup.
I don't blame Zed for being angry about this. Debian's Ruby has often been pretty broken even by vendor standards.
Have they offered to fix the underlying causes? E.g. Patch OpenSSL out for gnutls? Change the structure/metadata of gems so they are trivial to package?
These are the likely prerequisites to a slick experience for their users on any distro
I believe, yes, Zed Shaw is trying to make a big claim. Reread the original post and you will see that he does in fact accuse them of liking to patch everything because they like control and lockin.
Will you do me a favor and file this bug report for me? I'm allergic to sanctimony which I'm sure is all you'll receive from the package maintainer:
"gem update --system is disabled on Debian. RubyGems can be updated using the official Debian repositories by aptitude or apt-get."
That's a pretty absurd claim. Is there anything to back it up?
I mean, releasing an open source operating system doesn't seem to be the epitome of "control and lock in" to me. Ubuntu probably wouldn't exist if Debian were any good at "control and lock in", for one.
Also, your "bug report" is quite different than the one Zed points out. You're suggesting a fundamental change to the way rubygems works on Debian. That's something that merits further discussion, rather than a simple "bug report", obviously.
Excuse me, but I tried to direct you back to the original premise. Now it's an absurd claim? Did you read the article? It's not even my claim. And yes, if you read http://twitter.com/zedshaw you will see that the bug I posted is one that Zed references.
By the way, your "post" is quite the "epitome" of total "air quotery".
Yes, that "It's simply a tactic to make sure that you are stuck on Debian." was, and is, an absurd claim without a shred of support.
He also says that "it is sadly pure business and has nothing to do with open source, quality, or culture." which is pretty odd, given that Debian is a non-profit, voluntary organization.
I don't think it's anything to do with control and lockin, it's to do with the fact that Debian feel they have a far superior package management system and allowing other, inferior (from their perspective) systems is a Bad Idea which they'll fight tooth and nail every step of the way.
It does seem silly from a rubyist's perspective but their approach does make a heck of a lot of complicated software work together in a way that makes administering complex server setups a doddle compared to how it could be.
There's just a yawning philosophical gap between how the Debian and Ruby communities look at the world and approach software management.
1. Perl isn't nearly as broken on Debian as Ruby is. Don't know about the others.
2. These packaging systems were all designed by and for developers. Debian was designed by and for system administrators. The fact that there are common problems is not surpising.
Fair enough re: #2. Given that, it's not at all surprising that the ruby and python folk have done rvm and pip (which, in a faux-spectrum, is a big shift towards what maven et al. provide in the java world).
Of course, it's the debian maintainers' prerogative to structure things such that others generally pick up their toys and go play in their own sandboxes. I just get irritated when I hear people griping about it when it happens, as if system-level package managers have some kind of inherent priority.
I'd say that it's more that if there's no (current) way to track what the other package management system is doing on the distro (any distro, not just Debian) then they can't tell what packages are installed and hence whether the distro-supplied packages can work with them...
> Also: why should software authors have to write bug reports to Debian?
What is the alternative? Should software authors be given carte blanche control of their own packages, regardless of their knowledge and commitment to the Debian platform and ideals? The goal is to make a working system, not to indulge every author that thinks their package is the exception to the guidelines laid out by the Debian members.
I have to say, I just installed Debian's postgresql-9.0 package and I was pleasantly surprised. They've packaged things in such a way that you can install 9.0 and 8.4 in parallel. Yes, it is different than the default postgres install. Instead of everything being in one directory they put config files in /etc, library files in /usr/lib and your db in /var/lib--just like every other package on the system. This seems like the right kind of packaging to me--it makes the system consistent.
Of course, for every good package like postgres there's a package with some epically stupid decision (netcat-traditional's "-q" patch).
If Debian made it a policy to change packages as little as possible (which is not "not at all"), these problems wouldn't show up.
Also, yes, your shiny PostgreSQL package now matches every other Debian package, but it is different from every non-Debian PostgreSQL install out there. Zed wasn't talking about lock-in for nothing.
> If Debian made it a policy to change packages as little as possible (which is not "not at all"), these problems wouldn't show up.
No, but others would show up. Come on, these things are engineering tradeoffs, with both positive and negative aspects. Even where we disagree about the benefits and disadvantages, we should agree to disagree in a civil way, rather than calling for "in-person confrontations". There are certainly technical complaints to be made about Debian (I could make a long list myself), but Zed's attitude is way out of line.
I don't think you can consider that locked-in in the slightest--it's not like they changed the DB format or removed the ability to dump a db. Do you really think anyone is going to load up postgres on Fedora, find that it's installed in /var/postgres and not /var/lib/postgres, throw their hands in the air, scream like a girl and then go reinstall Debian? Having stuff moved around can be coped with. I'll be able to transfer the data itself to another disto just fine, thank you very much.
This because upstream projects usually can't be bothered to do integration work, and it's up to the distributions to do it and that means patches (which are normally separated from the upstream tarball in the source packages).
Debian adds patches, but so does Red Hat/Fedora. Since these are the two main distributions (maybe not by themselves, but because they form the base for most others), then you criticism can be applied to 99% of the distributions out there.
And bug reports about integration work should go to the distribution, and not upstream developers.
Look, I understand that combining gcc/binutils/glibc is a complete clusterfuck that needs some patches (each of these projects is "individually" at least somewhat sane, but they don't like working together at all), and I understand that there is a lot of shitty code in the average desktop environment. Really, some patches probably are necessary. (Although there's a whole debate about feeding them upstream that you could have at this point.)
But why does Debian split up Postfix (well-written, actively maintained) into a zillion different packages stitched together by dynamic loading, instead of just configuring it properly? Why does Debian fuck up OpenSSL?
No, it's not hard at all - I don't think it took me more than one minute to figure out. It is, however, an example of a gratuitous change.
With respect to the OpenSSL stuff, I'm not talking about the GPL license - I'm talking about http://www.debian.org/security/2008/dsa-1571, one of the most painful security issues in recent memory and caused by a meddling maintainer. (Certainly, the upstream team could have more clearly warned him, but without the maintainers changes everything would have worked just fine.)
I understand you spend a lot of your free time working for free, and I appreciate your good intentions and the good that Debian and Debian's developers have done (e.g. writing lots of man pages for programs that have none) - but I do think Debian has some really questionable sides as well.
Also: why should software authors have to write bug reports to Debian?