Re: java client cannot visit rgw behind nginx

Mark Kirkwood <mark.kirkwood@xxxxxxxxxxxxxxx> · Thu, 3 Sep 2020 16:52:58 +1200

I think you might need to set some headers. Here is what we use 
(connecting to Swift, but should be generally applicable). We are 
running nginx and swift (swift proxy server) on the same host. but again 
maybe some useful ideas for you to try (below).

Note that we explicitly stop nginx writing a temporary copy of any 
objects being uploaded (that is the last 3 lines)

--- config ---

server {
  listen       *:8443 ssl;
  server_name  swift-proxy;

  ssl on;

  ssl_certificate           /var/*refacted*;
  ssl_certificate_key       /var/*redacted*;
  ssl_session_cache         shared:SSL:10m;
  ssl_session_timeout       5m;
  ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
  ssl_prefer_server_ciphers on;

  client_max_body_size 5368709124;
  index  index.html index.htm index.php;

  access_log /var/log/nginx/swift-proxy-access.log combined;
  error_log /var/log/nginx/swift-proxy-error.log;

  location / {
    proxy_pass            http://127.0.0.1:8080;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_redirect        off;
    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      Proxy "";
    proxy_http_version 1.1;
    proxy_max_temp_file_size 0;
    proxy_request_buffering off;
  }
}

On 3/09/20 2:19 pm, Zhenshi Zhou wrote:
Hi Tom

Thanks for the reply. Here is my nginx configuration.
Did I miss something or is there some special option to set?
What's more, our Flink can work well by connecting to the frontend.

image.png

Tom Black <tom@pobox.store> 于2020年9月3日周四 上午8:13写道：

    It seems like your nginx has the wrong configuration for reverse
    proxy
    of S3.

    Thanks.

    Zhenshi Zhou wrote:
    > this is ES error log：
    > {
    >    "error": {
    >      "root_cause": [
    >        {
    >          "type": "repository_verification_exception",
    >          "reason": "[test] path  is not accessible on master node"
    >        }
    >      ],
    >      "type": "repository_verification_exception",
    >      "reason": "[test] path  is not accessible on master node",
    >      "caused_by": {
    >        "type": "i_o_exception",
    >        "reason": "Unable to upload object
    > [tests-CX3jGTbyRgOeOZJYci8MnQ/master.dat] using a single upload",
    >        "caused_by": {
    >          "type": "sdk_client_exception",
    >          "reason": "sdk_client_exception: Unable to execute HTTP
    > request: oldelk-snapshot.rgw.abc.cn
    <http://oldelk-snapshot.rgw.abc.cn>
    <http://oldelk-snapshot.rgw.abc.cn>",
    >          "caused_by": {
    >            "type": "i_o_exception",
    >            "reason": "oldelk-snapshot.rgw.abc.cn
    <http://oldelk-snapshot.rgw.abc.cn>
    > <http://oldelk-snapshot.rgw.abc.cn>"
    >          }
    >        }
    >      }
    >    },
    >    "status": 500
    > }
    >
    > Tom Black <tom@pobox.store> 于2020年9月2日周三 下午4:55写道：
    >
    >     Zhenshi Zhou wrote:
    >      > My fellows wanna use ceph rgw to store ES backup and
    Nexus blobs.
    >      > But the services cannot connect to the rgw with s3
    protocol when I
    >      > provided them with the frontend nginx address(virtual
    ip). Only when
    >      > they use the backend rgw's address(real ip) the ES and
    Nexus works
    >      > well with rgw.
    >
    >     you should provide both the client and server's error logs.
    >
    >     Thanks.
    >     _______________________________________________
    >     ceph-users mailing list -- ceph-users@xxxxxxx
    <mailto:ceph-users@xxxxxxx>
    >     <mailto:ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>>
    >     To unsubscribe send an email to ceph-users-leave@xxxxxxx
    <mailto:ceph-users-leave@xxxxxxx>
    >     <mailto:ceph-users-leave@xxxxxxx
    <mailto:ceph-users-leave@xxxxxxx>>
    >

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx