fix for broken kernel side RLIM_INFINITY on mips

On 32 bit mips the kernel uses -1UL/2 to mark RLIM_INFINITY (and
this is the definition in the userspace api), but since it is in
the middle of the valid range of limits and limits are often
compared with relational operators, various kernel side logic is
broken if larger than -1UL/2 limits are used. So we truncate the
limits to -1UL/2 in get/setrlimit and prlimit.

Even if the kernel side logic consistently treated -1UL/2 as greater
than any other limit value, there wouldn't be any clean workaround
that allowed using large limits:
* using -1UL/2 as RLIM_INFINITY in userspace would mean different
infinity value for get/setrlimt and prlimit (where infinity is always
-1ULL) and userspace logic could break easily (just like the kernel
is broken now) and more special case code would be needed for mips.
* translating -1UL/2 kernel side value to -1ULL in userspace would
mean that -1UL/2 limit cannot be set (eg. -1UL/2+1 had to be passed
to the kernel instead).

1 files changed, 8 insertions, 0 deletions

diff --git a/src/misc/getrlimit.c b/src/misc/getrlimit.c
index b7bbd062..b073677f 100644
--- a/src/misc/getrlimit.c
+++ b/src/misc/getrlimit.c
@@ -3,16 +3,24 @@
 #include "syscall.h"
 #include "libc.h"
 
+#define FIX(x) do{ if ((x)>=SYSCALL_RLIM_INFINITY) (x)=RLIM_INFINITY; }while(0)
+
 int getrlimit(int resource, struct rlimit *rlim)
 {
 	unsigned long k_rlim[2];
 	int ret = syscall(SYS_prlimit64, 0, resource, 0, rlim);
+	if (!ret) {
+		FIX(rlim->rlim_cur);
+		FIX(rlim->rlim_max);
+	}
 	if (!ret || errno != ENOSYS)
 		return ret;
 	if (syscall(SYS_getrlimit, resource, k_rlim) < 0)
 		return -1;
 	rlim->rlim_cur = k_rlim[0] == -1UL ? RLIM_INFINITY : k_rlim[0];
 	rlim->rlim_max = k_rlim[1] == -1UL ? RLIM_INFINITY : k_rlim[1];
+	FIX(rlim->rlim_cur);
+	FIX(rlim->rlim_max);
 	return 0;
 }