in fcntl, use unsigned long instead of long for variadic argument type

neither is correct; different commands take different argument types,
and some take no arguments at all. I have a much larger overhaul of
fcntl prepared to address this, but it's not appropriate to commit
during freeze.

the immediate problem being addressed affects forward-compatibility on
x32: if new commands are added and they take pointers, but the
libc-level fcntl function is not aware of them, using long would
sign-extend the pointer to 64 bits and give the kernel an invalid
pointer. on the kernel side, the argument to fcntl is always treated
as unsigned long, so no harm is done by treating possibly-signed
integer arguments as unsigned. for every command that takes an integer
argument except for F_SETOWN, large integer arguments and negative
arguments are handled identically anyway. in the case of F_SETOWN, the
kernel is responsible for converting the argument which it received as
unsigned long to int, so the sign of negative arguments is recovered.

the other problem that will be addressed later is that the type passed
to va_arg does not match the type in the caller of fcntl. an advanced
compiler doing cross-translation-unit analysis could potentially see
this mismatch and issue warnings or otherwise make trouble.

on i386, this patch was confirmed not to alter the code generated by
gcc 4.7.3. in principle the generated code should not be affected on
any arch except x32.

1 files changed, 2 insertions, 2 deletions

diff --git a/src/fcntl/fcntl.c b/src/fcntl/fcntl.c
index 4a713fd4..2c4f5351 100644
--- a/src/fcntl/fcntl.c
+++ b/src/fcntl/fcntl.c
@@ -7,10 +7,10 @@
 
 int fcntl(int fd, int cmd, ...)
 {
-	long arg;
+	unsigned long arg;
 	va_list ap;
 	va_start(ap, cmd);
-	arg = va_arg(ap, long);
+	arg = va_arg(ap, unsigned long);
 	va_end(ap);
 	if (cmd == F_SETFL) arg |= O_LARGEFILE;
 	if (cmd == F_SETLKW) return syscall_cp(SYS_fcntl, fd, cmd, (void *)arg);