www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Glibc hell

reply dsimcha <dsimcha yahoo.com> writes:
Apparently, DMD for Linux requires a non-ancient version of Glibc to work, and
worse yet, all the stuff compiled by it requires a similarly non-ancient
version.  The problem is that I'm trying to run some jobs on a cluster that
has an ancient version of Linux that my sysadmin doesn't want to upgrade.
Given that i don't have any root/admin privileges, and will never get them,
does anyone know of a workaround to make DMD, or at least stuff compiled by
it, work with ancient versions of glibc?

P.S.  Nothing illegal/unethical, such as cracking the root acct. please.
Jan 21 2009
next sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
dsimcha wrote:
 Apparently, DMD for Linux requires a non-ancient version of Glibc to work, and
 worse yet, all the stuff compiled by it requires a similarly non-ancient
 version.  The problem is that I'm trying to run some jobs on a cluster that
 has an ancient version of Linux that my sysadmin doesn't want to upgrade.
 Given that i don't have any root/admin privileges, and will never get them,
 does anyone know of a workaround to make DMD, or at least stuff compiled by
 it, work with ancient versions of glibc?
 
 P.S.  Nothing illegal/unethical, such as cracking the root acct. please.

Can't you just statically link to a non-ancient glibc? Alternatively you could probably link with it dynamically, possibly with some renaming and/or playing with LD_LIBRARY_PATH.
Jan 21 2009
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:
 Apparently, DMD for Linux requires a non-ancient version of Glibc to work, and
 worse yet, all the stuff compiled by it requires a similarly non-ancient
 version.  The problem is that I'm trying to run some jobs on a cluster that
 has an ancient version of Linux that my sysadmin doesn't want to upgrade.
 Given that i don't have any root/admin privileges, and will never get them,
 does anyone know of a workaround to make DMD, or at least stuff compiled by
 it, work with ancient versions of glibc?
 
 P.S.  Nothing illegal/unethical, such as cracking the root acct. please.

I'm facing the exact same problem (and on a cluster as well!). What I currently do is compile on my laptop, copy the binary with scp, then launch via ssh. It would be great if anyone knows of a better solution. Andrei
Jan 21 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 dsimcha wrote:
 Apparently, DMD for Linux requires a non-ancient version of Glibc to work, and
 worse yet, all the stuff compiled by it requires a similarly non-ancient
 version.  The problem is that I'm trying to run some jobs on a cluster that
 has an ancient version of Linux that my sysadmin doesn't want to upgrade.
 Given that i don't have any root/admin privileges, and will never get them,
 does anyone know of a workaround to make DMD, or at least stuff compiled by
 it, work with ancient versions of glibc?

 P.S.  Nothing illegal/unethical, such as cracking the root acct. please.

currently do is compile on my laptop, copy the binary with scp, then launch via ssh. It would be great if anyone knows of a better solution. Andrei

Yeah, I'm doing something like this, too. The problem is that, even after I get the code to _compile and link_, it still won't _execute_ with ancient glibc. I'll keep working at it, since it's not like I have anything better to do while waiting for my code to execute with the minimal resources that I have w/o this cluster.
Jan 21 2009
prev sibling next sibling parent Spacen Jasset <spacenjasset yahoo.co.uk> writes:
dsimcha wrote:
 Apparently, DMD for Linux requires a non-ancient version of Glibc to work, and
 worse yet, all the stuff compiled by it requires a similarly non-ancient
 version.  The problem is that I'm trying to run some jobs on a cluster that
 has an ancient version of Linux that my sysadmin doesn't want to upgrade.
 Given that i don't have any root/admin privileges, and will never get them,
 does anyone know of a workaround to make DMD, or at least stuff compiled by
 it, work with ancient versions of glibc?
 
 P.S.  Nothing illegal/unethical, such as cracking the root acct. please.

Get Walter to recompile DMD on centos 2, or install the gcc-compat libraries and compiler and again recompile DMD with that so it's forward compatible. glibc shouldn't be statically linked, it leads to crashes. libc++ however can be.
Jan 21 2009
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
"dsimcha" wrote
 Apparently, DMD for Linux requires a non-ancient version of Glibc to work, 
 and
 worse yet, all the stuff compiled by it requires a similarly non-ancient
 version.  The problem is that I'm trying to run some jobs on a cluster 
 that
 has an ancient version of Linux that my sysadmin doesn't want to upgrade.
 Given that i don't have any root/admin privileges, and will never get 
 them,
 does anyone know of a workaround to make DMD, or at least stuff compiled 
 by
 it, work with ancient versions of glibc?

 P.S.  Nothing illegal/unethical, such as cracking the root acct. please.

You may have to statically link (which, of course, is not officially supported by glibc, for very stupid reasons). I'm not sure how much of the new glibc is required by dmd, part of glibc is in the dynamic linker, which I think is hard to use a local copy of. However, the linker does support a way to override which libraries to use. I'd suggest the following. I assume you have a local account with some space. mkdir ~/libs cp my_glibc.so ~/libs cp my_supporting_lib.so ~/libs ... # bash syntax, not sure of csh syntax export LD_LIBRARY_PATH=~/libs What this does is make the dynamic linker use ~/libs to look for dynamic libs to link in before the default ones. You can check which libraries the dynamic linker is using to resolve dependencies by using 'ldd executable'. Hope this helps -Steve
Jan 21 2009
parent reply Spacen Jasset <spacenjasset yahoo.co.uk> writes:
Steven Schveighoffer wrote:
 "dsimcha" wrote
 Apparently, DMD for Linux requires a non-ancient version of Glibc to work, 
 and
 worse yet, all the stuff compiled by it requires a similarly non-ancient
 version.  The problem is that I'm trying to run some jobs on a cluster 
 that
 has an ancient version of Linux that my sysadmin doesn't want to upgrade.
 Given that i don't have any root/admin privileges, and will never get 
 them,
 does anyone know of a workaround to make DMD, or at least stuff compiled 
 by
 it, work with ancient versions of glibc?

 P.S.  Nothing illegal/unethical, such as cracking the root acct. please.

You may have to statically link (which, of course, is not officially supported by glibc, for very stupid reasons). I'm not sure how much of the new glibc is required by dmd, part of glibc is in the dynamic linker, which I think is hard to use a local copy of. However, the linker does support a way to override which libraries to use. I'd suggest the following. I assume you have a local account with some space. mkdir ~/libs cp my_glibc.so ~/libs cp my_supporting_lib.so ~/libs ... # bash syntax, not sure of csh syntax export LD_LIBRARY_PATH=~/libs What this does is make the dynamic linker use ~/libs to look for dynamic libs to link in before the default ones. You can check which libraries the dynamic linker is using to resolve dependencies by using 'ldd executable'. Hope this helps -Steve

I am not sure that the reasons are "stupid". It is similar, for example to kernel32.dll on windows, which you cannot link to statically *at all* libc is a comparable interface in that it calls into the kernel. Linking to an older libc should ensure a good level of compatibility going forward, but I am not an expert on this. We do this for some of our software at the moment, which works without trouble.
Jan 21 2009
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
"Spacen Jasset" wrote
 Steven Schveighoffer wrote:
 You may have to statically link (which, of course, is not officially 
 supported by glibc, for very stupid reasons).

I am not sure that the reasons are "stupid". It is similar, for example to kernel32.dll on windows, which you cannot link to statically *at all* libc is a comparable interface in that it calls into the kernel.

The reasons are not so concrete. It's more a matter of opinion than requirement. For example, see this web page: http://people.redhat.com/drepper/no_static_linking.html I ran into this when trying to get busybox working. It took me a while to figure out, but I ended up using a dynamic glibc. Apparently, there are bugs in glibc with static linking that the developers refuse to fix because "static linking glibc isn't a valid requirement." Yet some very important programs are statically linked. For example, ldconfig and nash (Red Hat's system loader shell). It just seems stupid to me when they preach that you should *always* dynamically link, yet there are cases where they found it more suitable to statically link. Just my opinion. -Steve
Jan 21 2009
parent reply Spacen Jasset <spacenjasset yahoo.co.uk> writes:
Steven Schveighoffer wrote:
 "Spacen Jasset" wrote
 Steven Schveighoffer wrote:
 You may have to statically link (which, of course, is not officially 
 supported by glibc, for very stupid reasons).

kernel32.dll on windows, which you cannot link to statically *at all* libc is a comparable interface in that it calls into the kernel.

The reasons are not so concrete. It's more a matter of opinion than requirement. For example, see this web page: http://people.redhat.com/drepper/no_static_linking.html I ran into this when trying to get busybox working. It took me a while to figure out, but I ended up using a dynamic glibc. Apparently, there are bugs in glibc with static linking that the developers refuse to fix because "static linking glibc isn't a valid requirement." Yet some very important programs are statically linked. For example, ldconfig and nash (Red Hat's system loader shell). It just seems stupid to me when they preach that you should *always* dynamically link, yet there are cases where they found it more suitable to statically link. Just my opinion. -Steve

unreasonable to require dynamic linking for things such as glibc. (again, just like kernel32.dll and user32.dll on windows) At our company we compile against older versions of glib c and the same code runs without problem on the latest versions. This is not the case with static linking it often breaks unless you run on the same system you linked on. As for bugs in glibc preventing static linking I can't comment except to say that the the kernel apis do sometimes change and I have read that static linking will case crashes, which it does. It appears they don't want to maintain static compatibility in this way. However, if you do build glibc from source there is a configurable option to provide compatability wrappers, I noticed this when building a cross compiler. I know that static linking of glibc is sometimes used so that the libraries are not required for base operating system components, in particular some maintenance binaries are linked in this way so that a system can be recovered without the need for a working usable /usr/lib directory. Unfortunately it does seem easier to compile against an older glibc using an older distribution, but there may be a better way. Installing an older gcc may suffice. At our company we took the "easy way" and used a virtual machine. For us Linux DMD users a bug should be raised against dmd so that Walter will hopefully compile against an older glibc on future releases.
Jan 23 2009
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
"Spacen Jasset" wrote
 Steven Schveighoffer wrote:
 "Spacen Jasset" wrote
 Steven Schveighoffer wrote:
 You may have to statically link (which, of course, is not officially 
 supported by glibc, for very stupid reasons).

to kernel32.dll on windows, which you cannot link to statically *at all* libc is a comparable interface in that it calls into the kernel.

The reasons are not so concrete. It's more a matter of opinion than requirement. For example, see this web page: http://people.redhat.com/drepper/no_static_linking.html I ran into this when trying to get busybox working. It took me a while to figure out, but I ended up using a dynamic glibc. Apparently, there are bugs in glibc with static linking that the developers refuse to fix because "static linking glibc isn't a valid requirement." Yet some very important programs are statically linked. For example, ldconfig and nash (Red Hat's system loader shell). It just seems stupid to me when they preach that you should *always* dynamically link, yet there are cases where they found it more suitable to statically link. Just my opinion. -Steve

unreasonable to require dynamic linking for things such as glibc. (again, just like kernel32.dll and user32.dll on windows)

Sure, I agree with you there. No matter how much whoever complains about this, it's not going to be addressed, so complaining generally isn't very productive ;) I don't think it's unreasonable to require glibc to be dynamic, but again, you have issues that come up which might be solvable if glibc supported static linking. It's all fine in theory until you start encountering reality.
 As for bugs in glibc preventing static linking I can't comment except to 
 say that the the kernel apis do sometimes change and I have read that 
 static linking will case crashes, which it does. It appears they don't 
 want to maintain static compatibility in this way. However, if you do 
 build glibc from source there is a configurable option to provide 
 compatability wrappers, I noticed this when building a cross compiler.

I don't think this is the reason for the bugs. I think they are just design flaws which the developers consider moot because nobody should be statically linking in their opinion.
 For us Linux DMD users a bug should be raised against dmd so that Walter 
 will hopefully compile against an older glibc on future releases.

As long as it doesn't cause weird problems. I have had weird stuff happen when you are running an older app against a newer lib (specifically, the program exited silently). Plus, I think the OP essentially was looking to run programs that DMD compiled, not necessarily running DMD itself. -Steve
Jan 23 2009
parent Spacen Jasset <spacenjasset yahoo.co.uk> writes:
Steven Schveighoffer wrote:
 "Spacen Jasset" wrote
 Steven Schveighoffer wrote:


 For us Linux DMD users a bug should be raised against dmd so that Walter 
 will hopefully compile against an older glibc on future releases.

As long as it doesn't cause weird problems. I have had weird stuff happen when you are running an older app against a newer lib (specifically, the program exited silently).

problems. Perhaps some sort of trial with DMD would be helpful.
 Plus, I think the OP essentially was looking to run programs that DMD 
 compiled, not necessarily running DMD itself.
 
 -Steve 

Ideally, we (as in someone) will want to do both of these things.
Jan 26 2009
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Spacen Jasset wrote:
 For us Linux DMD users a bug should be raised against dmd so that Walter 
 will hopefully compile against an older glibc on future releases.

Yet when I do that the other half of the linux users have a problem.
Jan 23 2009
parent Spacen Jasset <spacenjasset yahoo.co.uk> writes:
Walter Bright wrote:
 Spacen Jasset wrote:
 For us Linux DMD users a bug should be raised against dmd so that 
 Walter will hopefully compile against an older glibc on future releases.

Yet when I do that the other half of the linux users have a problem.

Do you know what problems they had Walter, I think this problem should be able to be ironed out somehow. I believe that is is ok to statically link with the c++ libraries* and we do this at our workplace, otherwise the target users also have to have the correct c++ libraries installed. The link flags I use for our builds are like this: LINK_FLAGS=-Wl,-Bstatic,-lstdc++,-Bdynamic -static-libgcc and are passed to gcc for the linking phase. Would it be possible for you to generate a version of DMD built in this way for testing as well as how you do it currently? If you have already done this and found no way to make it work then perhaps it's not worthwhile, but this really should work on Linux. * - you don't get the "benefit" of bug fixes via the dynamic library updates.
Jan 26 2009