It would do compress=TRUE if the input is not too big, otherwise it would fallback to compress=FALSE. For example:
RleList(Rle(8, 2e9), Rle(5, 200), compress=NA)
would do RleList(Rle(8, 2e9), Rle(5, 200), compress=TRUE) i.e. return a CompressedRleList object.
But:
RleList(Rle(8, 2e9), Rle(5, 2e9), compress=NA)
would do RleList(Rle(8, 2e9), Rle(5, 2e9), compress=FALSE) i.e. return a SimpleRleList object.
Then compress=NA should probably be made the new default (right now it's compress=TRUE, which is problematic because it fails when the input is too big).
See Bioconductor/VariantAnnotation#71 (comment) for some background.