Re: Possible to run the server with ANSI/ISO string escapeing

[Ken Johanson <pg-user@xxxxxxxxxxxxx>]

Martijn van Oosterhout wrote:

On Sun, Feb 27, 2005 at 06:25:18PM -0700, Ken Johanson wrote:
  

I feel somewhat confident (very actually) that a config option that 
disabled the backslash behavior globally(*) would be acceptable, BUT 
leave the current backslash behavior turned on by default so that 
current users are not impacted at all. Only a conscientious decision by 
the db admin to turn it on could cause problems, but _only_ if he/she 
didn't warn all his/her users beforehand of the impending change and its 
consequences (rtm).
    

It's not just a question of warning the users, all interfaces to the
database will instantly break. For example: JDBC, Perl DBI, PHP PEAR
etc. They will continue to send queries with the backslashes embedded.
These interfaces would need to be modified to handle both situations
and detect which situation they're dealing with.
  

All interfaces will NOT break IF the legacy db behavior stays its default. This means NONE of the current users would be hurt until they start experimenting with the new option. Yes, the built in prepared stmt components of those interfaces will still add the backslash by default and break queries for legacy drivers, but this is not an issue for the straight-through query/update exec(s) calls, and prepared stmt users can hack the Prepared stmts behavior until the same option is officially supported in the driver also (probably by auto-detecting what the DB expects its backslashes to look like).

Like I said, users should be warned beforehand, that they need to get a hacked or official driver update, if the dbadmin decides to turn on the 'new' mode. Seems prudent to me.

The thing is all these interfaces handle the quoting transparently for
you, so the code is portable already. What you're complaining about is
that you have your own query marshalling and it is not portable.
  

As you say, the portability you describe REQUIRES the use of prepared stmts type queries - but one CANNOT issue a portable query, say, jdbc:stmt.execQuery() or execute update. Those "lower-level" calls need to be portable where string escaping is concerned, and they are not. They arguably break both the JDBC spec and SQL spec since some additional, nonstandard string preprocessing is REQUIRED for them to work. And not to repeat what you already know, but Prepared stmts are not suitable or available for certain query types (performance sometimes better w/o PS, dynamic query building, batch queries, etc) and drivers (small foot print ones).

Incidently, if you disable the backslash quoting, how does one enter
raw binary data including NUL (\0) characters? 
  

I'm not sure if you're talking about API interfaces or shell, or both. If shell, a piped sql-compliant escape processor would clearly be needed.

As for APIs, I suspect the current design of each driver handles nulls and the like, stems from how the underlying db protocol was originally built --so they use either run length encoding (chunking), boundary-delimiting, or are null terminated (whichever PG uses natively, I don't know). In any case the very lowest level of the db interface can translate as necessary (doubling the backslashes if necessary), but queries should be able to issue any byte, including nulls, with the only requirement that apostrophes are the string-escape character, for themselves. In other words nulls should need no special treatment from the query interface layer.

The only viable solution I can think of is that it is set at
*connection* time (maybe extra parameters), and unchangable for the
rest of the session. This means that unmodified client interfaces won't
see a difference.

  

Yes, sessions (connections) could work, and also perhaps per-user or group, wherein the db the escape handing is handled the old or new way on a per-user basis.

I can say, that I for one would enable the no-backslash config option 
out of the box -globally -so that we can start using pg now without any 
more upper managerial concerns/excuses about language/interface 
compliance..I can also say that (what we already know) the longer we 
wait to provide the 'right' option, the *more* legacy apps (and 
interfaces) will be built around it and consequently suffer when the 
need for change eventually comes (almost wholly caused by interop 
concerns). And market gain is being hurt now by this incompatibility 
with commercial offerings; that's an unfortunate fact.
    

Even if PostgreSQL implements this now, you will have to wait for new
versions of any client libraries before it's usable. See the autocommit
disaster for an example why people are not rushing into this...

  

I fully agree. I can see waiting at LEAST 1-3 months before the db itself has changes committed for alpha testing, but that SURE BEATS procrastination --which means years worth of more apps and interfaces being built around the 'backslash' (again, not everyone uses prepared statement - its not required and not suitable for all situations). Conversely, the very day the server has an alpha build supporting the no-backslash mode is the very _first_ day that the jdbc/perl driver developers can start testing against the changes. Until then all parties are just sitting still.

Incidentally, I believe PG is now one of the last widely used DBs to not offer the standard escape behavior; the other popular OS db now has this option (albeit buggy), at the request of a very large software company that wanted to adopt their db no less...

Let me just finish by saying that I'm uncomfortable and sorry to be making waves like I am, but I feel it's in everyones best interest long term to start making this change now and teach the users sooner than later - this topic is has been a real handicap for my own adoption of the database (peer-wise), and the problem will only fester if nothing is done. Pain now or pain later.

ken