[regression] ENOTTY returned for tty fds

[stsp <stsp2@xxxxxxxxx>]

Hi!

I've noticed that one of my progs failed
to work with serial ports. To distinguish
between real serial ports and PTS nodes,
it tried TIOCMBIC, checking errno for EINVAL.

Obviously that behaviour was changed
and now ENOTTY is returned. Besides
being backward-incompatible, I think this
is also wrong because isatty() returns 1
on such FDs.

Here's the test-case:
socat PTY,link=/tmp/ttyS20,raw,echo=0,b9600 
PTY,link=/tmp/ttyS21,raw,echo=0,b9600

Then run this program:

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <unistd.h>

int main() {
  int fd, err;
  int data = TIOCM_DTR | TIOCM_RTS;
  fd = open("/tmp/ttyS20", O_NONBLOCK);
  if (fd == -1) {
    perror("open()");
    return 1;
  }
  err = ioctl(fd, TIOCMBIC, &data);
  if (err && errno == ENOTTY && isatty(fd))
    printf("Test FAILED: ENOTTY for tty fd\n");
  return 0;
}