Hi Yee Ting, The RGW replied with 400 "Invalid Input". can you please send the RGW logs? They should hold more information about the nature of the problem. Regarding the certificate file, it is not loaded when configured, but only when the connection to the broker is created and the first notification is sent, so, this is probably not the cause of the problem. However, note that the file is expected to be on the machine where the RGW is running, and not where the S3 client is. Yuval On Mon, Oct 17, 2022 at 11:27 PM Li, Yee Ting <ytl@xxxxxxxxxxxxxxxxx> wrote: > Hi, i'm on quincy trying to register a kafka notification; i can get > non-authenticated kafka working fine, however, when i attempt to register a > topic with credentials and a kafka cluster certificate i get an 'unknown > error'. > > one question i have is where should the `ca-location` cert be, it doesn't > look like the command line uploads it to the gateway? (but even without the > attribute, it returns the same error). > > cheers, > > Yee. > > ❯ singularity pull docker://amazon/aws-cli > ❯ alias s3="singularity exec aws-cli_latest.sif aws --endpoint-url > https://s3dfrgw.example.com s3 --profile=prompt-processing" > (register creds with profile prompt-processing) > > ❯ sns --region=default create-topic --name=processing-test > --attributes='{"push-endpoint": "kafka://my-user:<redacted>@ > 172.24.5.223:9094", "use-ssl": "true", "ca-location": "cluster.crt" }' > --debug > > 2022-10-17 08:37:56,043 - MainThread - awscli.clidriver - DEBUG - CLI > version: aws-cli/2.7.34 Python/3.9.11 Linux/4.18.0-372.16.1.el8_6.x86_64 > docker/x86_64.amzn.2 > 2022-10-17 08:37:56,043 - MainThread - awscli.clidriver - DEBUG - > Arguments entered to CLI: ['--endpoint-url', 'https://s3dfrgw.example.com', > 'sns', '--profile=processing-test', '--region=default', 'create-topic', > '--name=processing-test-test<https://s3dfrgw.example.com',%20'sns',%20'--profile=processing-test',%20'--region=default',%20'create-topic',%20'--name=processing-test-test>', > '--attributes={"push-endpoint": "kafka://my-user:<redacted>@ > 172.24.5.223:9094", "use-ssl": "true", "ca-location": "cluster.crt" }', > '--debug'] > 2022-10-17 08:37:56,064 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function add_s3 at > 0x7fb5de42c4c0> > 2022-10-17 08:37:56,064 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function add_ddb at > 0x7fb5de606040> > 2022-10-17 08:37:56,064 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <bound method > BasicCommand.add_command of <class > 'awscli.customizations.configure.configure.ConfigureCommand'>> > 2022-10-17 08:37:56,064 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function change_name at > 0x7fb5de6a9310> > 2022-10-17 08:37:56,065 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function change_name at > 0x7fb5de62f160> > 2022-10-17 08:37:56,065 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function alias_opsworks_cm at > 0x7fb5de43de50> > 2022-10-17 08:37:56,065 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function add_history_commands > at 0x7fb5de5cbc10> > 2022-10-17 08:37:56,065 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <bound method > BasicCommand.add_command of <class > 'awscli.customizations.devcommands.CLIDevCommand'>> > 2022-10-17 08:37:56,065 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.main: calling handler <function add_waiters at > 0x7fb5de43d0d0> > 2022-10-17 08:37:56,065 - MainThread - botocore.loaders - DEBUG - Loading > JSON file: /usr/local/aws-cli/v2/2.7.34/dist/awscli/data/cli.json > 2022-10-17 08:37:56,067 - MainThread - botocore.hooks - DEBUG - Event > top-level-args-parsed: calling handler <function resolve_types at > 0x7fb5de4fadc0> > 2022-10-17 08:37:56,067 - MainThread - botocore.hooks - DEBUG - Event > top-level-args-parsed: calling handler <function no_sign_request at > 0x7fb5de501940> > 2022-10-17 08:37:56,067 - MainThread - botocore.hooks - DEBUG - Event > top-level-args-parsed: calling handler <function resolve_verify_ssl at > 0x7fb5de5018b0> > 2022-10-17 08:37:56,067 - MainThread - botocore.hooks - DEBUG - Event > top-level-args-parsed: calling handler <function resolve_cli_read_timeout > at 0x7fb5de501a60> > 2022-10-17 08:37:56,067 - MainThread - botocore.hooks - DEBUG - Event > top-level-args-parsed: calling handler <function > resolve_cli_connect_timeout at 0x7fb5de5019d0> > 2022-10-17 08:37:56,068 - MainThread - botocore.hooks - DEBUG - Event > top-level-args-parsed: calling handler <built-in method update of dict > object at 0x7fb5de3d8f00> > 2022-10-17 08:37:56,068 - MainThread - botocore.session - DEBUG - Setting > config variable for profile to 'processing-test' > 2022-10-17 08:37:56,068 - MainThread - botocore.session - DEBUG - Setting > config variable for region to 'default' > 2022-10-17 08:37:56,068 - MainThread - awscli.clidriver - DEBUG - CLI > version: aws-cli/2.7.34 Python/3.9.11 Linux/4.18.0-372.16.1.el8_6.x86_64 > docker/x86_64.amzn.2 prompt/off > 2022-10-17 08:37:56,068 - MainThread - awscli.clidriver - DEBUG - > Arguments entered to CLI: ['--endpoint-url', 'https://s3dfrgw.example.com', > 'sns', '--profile=processing-test', '--region=default', 'create-topic', > '--name=processing-test-test<https://s3dfrgw.example.com',%20'sns',%20'--profile=processing-test',%20'--region=default',%20'create-topic',%20'--name=processing-test-test>', > '--attributes={"push-endpoint": "kafka://my-user:<redacted>@ > 172.24.5.223:9094", "use-ssl": "true", "ca-location": "cluster.crt" }', > '--debug'] > 2022-10-17 08:37:56,068 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function add_timestamp_parser at > 0x7fb5de42caf0> > 2022-10-17 08:37:56,068 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function register_uri_param_handler > at 0x7fb5deeeaee0> > 2022-10-17 08:37:56,068 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function add_binary_formatter at > 0x7fb5de4203a0> > 2022-10-17 08:37:56,068 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function no_pager_handler at > 0x7fb5deeeb310> > 2022-10-17 08:37:56,068 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function > inject_assume_role_provider_cache at 0x7fb5dee51d30> > 2022-10-17 08:37:56,069 - MainThread - botocore.utils - DEBUG - IMDS > ENDPOINT: http://169.254.169.254/ > 2022-10-17 <http://169.254.169.254/2022-10-17> 08:37:56,071 - MainThread > - botocore.credentials - DEBUG - Skipping environment variable credential > check because profile name was explicitly set. > 2022-10-17 08:37:56,071 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function attach_history_handler at > 0x7fb5de5cbaf0> > 2022-10-17 08:37:56,071 - MainThread - botocore.hooks - DEBUG - Event > session-initialized: calling handler <function inject_json_file_cache at > 0x7fb5de5ffe50> > 2022-10-17 08:37:56,082 - MainThread - botocore.loaders - DEBUG - Loading > JSON file: > /usr/local/aws-cli/v2/2.7.34/dist/awscli/botocore/data/sns/2010-03-31/service-2.json > 2022-10-17 08:37:56,089 - MainThread - botocore.hooks - DEBUG - Event > building-command-table.sns: calling handler <function add_waiters at > 0x7fb5de43d0d0> > 2022-10-17 08:37:56,101 - MainThread - awscli.clidriver - DEBUG - > OrderedDict([('name', <awscli.arguments.CLIArgument object at > 0x7fb5dda98e80>), ('attributes', <awscli.arguments.CLIArgument object at > 0x7fb5dda98eb0>), ('tags', <awscli.arguments.ListArgument object at > 0x7fb5dda98df0>), ('data-protection-policy', <awscli.arguments.CLIArgument > object at 0x7fb5dda98dc0>)]) > 2022-10-17 08:37:56,102 - MainThread - botocore.hooks - DEBUG - Event > building-argument-table.sns.create-topic: calling handler <function > add_streaming_output_arg at 0x7fb5de42cdc0> > 2022-10-17 08:37:56,102 - MainThread - botocore.hooks - DEBUG - Event > building-argument-table.sns.create-topic: calling handler <function > add_cli_input_json at 0x7fb5dee56550> > 2022-10-17 08:37:56,102 - MainThread - botocore.hooks - DEBUG - Event > building-argument-table.sns.create-topic: calling handler <function > add_cli_input_yaml at 0x7fb5dee56790> > 2022-10-17 08:37:56,102 - MainThread - botocore.hooks - DEBUG - Event > building-argument-table.sns.create-topic: calling handler <function > unify_paging_params at 0x7fb5de6065e0> > 2022-10-17 08:37:56,115 - MainThread - botocore.loaders - DEBUG - Loading > JSON file: > /usr/local/aws-cli/v2/2.7.34/dist/awscli/botocore/data/sns/2010-03-31/paginators-1.json > 2022-10-17 08:37:56,116 - MainThread - botocore.hooks - DEBUG - Event > building-argument-table.sns.create-topic: calling handler <function > add_generate_skeleton at 0x7fb5de4fa3a0> > 2022-10-17 08:37:56,116 - MainThread - botocore.hooks - DEBUG - Event > before-building-argument-table-parser.sns.create-topic: calling handler > <bound method OverrideRequiredArgsArgument.override_required_args of > <awscli.customizations.cliinput.CliInputJSONArgument object at > 0x7fb5dda98f40>> > 2022-10-17 08:37:56,116 - MainThread - botocore.hooks - DEBUG - Event > before-building-argument-table-parser.sns.create-topic: calling handler > <bound method OverrideRequiredArgsArgument.override_required_args of > <awscli.customizations.cliinput.CliInputYAMLArgument object at > 0x7fb5dda98fd0>> > 2022-10-17 08:37:56,116 - MainThread - botocore.hooks - DEBUG - Event > before-building-argument-table-parser.sns.create-topic: calling handler > <bound method GenerateCliSkeletonArgument.override_required_args of > <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument > object at 0x7fb5dda98f70>> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.name: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > process-cli-arg.sns.create-topic: calling handler > <awscli.argprocess.ParamShorthandParser object at 0x7fb5dee915b0> > 2022-10-17 08:37:56,117 - MainThread - awscli.arguments - DEBUG - Unpacked > value of 'processing-test-test' for parameter "name": 'processing-test-test' > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.attributes: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > process-cli-arg.sns.create-topic: calling handler > <awscli.argprocess.ParamShorthandParser object at 0x7fb5dee915b0> > 2022-10-17 08:37:56,117 - MainThread - awscli.argprocess - DEBUG - Param > attributes looks like JSON, not considered for param shorthand. > 2022-10-17 08:37:56,117 - MainThread - awscli.arguments - DEBUG - Unpacked > value of '{"push-endpoint": "kafka://my-user:<redacted>@172.24.5.223:9094", > "use-ssl": "true", "ca-location": "cluster.crt" }' for parameter > "attributes": OrderedDict([('push-endpoint', > 'kafka://my-user:<redacted>@172.24.5.223:9094'), ('use-ssl', 'true'), > ('ca-location', 'cluster.crt')]) > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.tags: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.data-protection-policy: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.cli-input-json: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.cli-input-yaml: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > load-cli-arg.sns.create-topic.generate-cli-skeleton: calling handler > <awscli.paramfile.URIArgumentHandler object at 0x7fb5ddba77f0> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > calling-command.sns.create-topic: calling handler <bound method > CliInputArgument.add_to_call_parameters of > <awscli.customizations.cliinput.CliInputJSONArgument object at > 0x7fb5dda98f40>> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > calling-command.sns.create-topic: calling handler <bound method > CliInputArgument.add_to_call_parameters of > <awscli.customizations.cliinput.CliInputYAMLArgument object at > 0x7fb5dda98fd0>> > 2022-10-17 08:37:56,117 - MainThread - botocore.hooks - DEBUG - Event > calling-command.sns.create-topic: calling handler <bound method > GenerateCliSkeletonArgument.generate_skeleton of > <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument > object at 0x7fb5dda98f70>> > 2022-10-17 08:37:56,118 - MainThread - botocore.credentials - DEBUG - > Looking for credentials via: assume-role > 2022-10-17 08:37:56,118 - MainThread - botocore.credentials - DEBUG - > Looking for credentials via: assume-role-with-web-identity > 2022-10-17 08:37:56,118 - MainThread - botocore.credentials - DEBUG - > Looking for credentials via: sso > 2022-10-17 08:37:56,118 - MainThread - botocore.credentials - DEBUG - > Looking for credentials via: shared-credentials-file > 2022-10-17 08:37:56,118 - MainThread - botocore.credentials - INFO - Found > credentials in shared credentials file: ~/.aws/credentials > 2022-10-17 08:37:56,120 - MainThread - botocore.loaders - DEBUG - Loading > JSON file: > /usr/local/aws-cli/v2/2.7.34/dist/awscli/botocore/data/endpoints.json > 2022-10-17 08:37:56,133 - MainThread - botocore.hooks - DEBUG - Event > choose-service-name: calling handler <function handle_service_name_alias at > 0x7fb5e0a6a5e0> > 2022-10-17 08:37:56,135 - MainThread - botocore.hooks - DEBUG - Event > creating-client-class.sns: calling handler <function > add_generate_presigned_url at 0x7fb5e0ad5310> > 2022-10-17 08:37:56,135 - MainThread - botocore.regions - DEBUG - Creating > a regex based endpoint for sns, default > 2022-10-17 08:37:56,138 - MainThread - botocore.endpoint - DEBUG - Setting > sns timeout as (60, 60) > 2022-10-17 08:37:56,139 - MainThread - botocore.hooks - DEBUG - Event > provide-client-params.sns.CreateTopic: calling handler <function > base64_decode_input_blobs at 0x7fb5de420af0> > 2022-10-17 08:37:56,139 - MainThread - botocore.hooks - DEBUG - Event > before-parameter-build.sns.CreateTopic: calling handler <function > generate_idempotent_uuid at 0x7fb5e0801700> > 2022-10-17 08:37:56,139 - MainThread - botocore.hooks - DEBUG - Event > before-call.sns.CreateTopic: calling handler <function > inject_api_version_header_if_needed at 0x7fb5e0802f70> > 2022-10-17 08:37:56,139 - MainThread - botocore.endpoint - DEBUG - Making > request for OperationModel(name=CreateTopic) with params: {'url_path': '/', > 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': > 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': > 'aws-cli/2.7.34 Python/3.9.11 Linux/4.18.0-372.16.1.el8_6.x86_64 > docker/x86_64.amzn.2 prompt/off command/sns.create-topic'}, 'body': > {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': > 'processing-test-test', 'Attributes.entry.1.key': 'push-endpoint', > 'Attributes.entry.1.value': 'kafka://my-user:<redacted>@172.24.5.223:9094', > 'Attributes.entry.2.key': 'use-ssl', 'Attributes.entry.2.value': 'true', > 'Attributes.entry.3.key': 'ca-location', 'Attributes.entry.3.value': > 'cluster.crt'}, 'url': 'https://s3dfrgw.example.com/', 'context< > https://s3dfrgw.example.com/',%20'context>': {'client_region': 'default', > 'client_config': <botocore.config.Config object at 0x7fb5dd567b20>, > 'has_streaming_input': False, 'auth_type': None}} > 2022-10-17 08:37:56,139 - MainThread - botocore.hooks - DEBUG - Event > request-created.sns.CreateTopic: calling handler <bound method > RequestSigner.handler of <botocore.signers.RequestSigner object at > 0x7fb5dd5679d0>> > 2022-10-17 08:37:56,139 - MainThread - botocore.hooks - DEBUG - Event > choose-signer.sns.CreateTopic: calling handler <function > set_operation_specific_signer at 0x7fb5e08015e0> > 2022-10-17 08:37:56,139 - MainThread - botocore.auth - DEBUG - Calculating > signature using v4 auth. > 2022-10-17 08:37:56,140 - MainThread - botocore.auth - DEBUG - > CanonicalRequest: > POST > / > > content-type:application/x-www-form-urlencoded; charset=utf-8 > host:s3dfrgw.example.com<http://s3dfrgw.example.com> > x-amz-date:20221017T153756Z > > content-type;host;x-amz-date > 9cd4cabc69d03c0b45b51de3aed2764185b2d3e4a2c426d95624ee04127d2c89 > 2022-10-17 08:37:56,140 - MainThread - botocore.auth - DEBUG - > StringToSign: > AWS4-HMAC-SHA256 > 20221017T153756Z > 20221017/default/sns/aws4_request > 6118de5b5cc8466768a15393fba5f8c7ffa8ee212e2f3555086774d69b2493bc > 2022-10-17 08:37:56,140 - MainThread - botocore.auth - DEBUG - Signature: > 62d6119ffa70fcbffaa476dfc400402e174e6734e2d3174172c66c56e00b4a75 > 2022-10-17 08:37:56,140 - MainThread - botocore.endpoint - DEBUG - Sending > http request: <AWSPreparedRequest stream_output=False, method=POST, url= > https://s3dfrgw.example.com/, headers={'Content-Type': > b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': > b'aws-cli/2.7.34 Python/3.9.11 Linux/4.18.0-372.16.1.el8_6.x86_64 > docker/x86_64.amzn.2 prompt/off command/sns.create-topic', 'X-Amz-Date': > b'20221017T153756Z', 'Authorization': b'AWS4-HMAC-SHA256 > Credential=8PTGE4UNTO2ELRK3EFOR/20221017/default/sns/aws4_request, > SignedHeaders=content-type;host;x-amz-date, > Signature=62d6119ffa70fcbffaa476dfc400402e174e6734e2d3174172c66c56e00b4a75', > 'Content-Length': '326'}> > 2022-10-17 08:37:56,140 - MainThread - botocore.httpsession - DEBUG - > Certificate path: > /usr/local/aws-cli/v2/2.7.34/dist/awscli/botocore/cacert.pem > 2022-10-17 08:37:56,140 - MainThread - urllib3.connectionpool - DEBUG - > Starting new HTTPS connection (1): s3dfrgw.example.com:443< > http://s3dfrgw.example.com:443> > 2022-10-17 08:37:56,167 - MainThread - urllib3.connectionpool - DEBUG - > https://s3dfrgw.example.com:443 "POST / HTTP/1.1" 400 187 > 2022-10-17 08:37:56,167 - MainThread - botocore.parsers - DEBUG - Response > headers: {'Content-Length': '187', 'x-amz-request-id': > 'tx000005ec21219335dafaa-00634d76d4-2f74a-default', 'Accept-Ranges': > 'bytes', 'Content-Type': 'application/xml', 'Date': 'Mon, 17 Oct 2022 > 15:37:56 GMT'} > 2022-10-17 08:37:56,167 - MainThread - botocore.parsers - DEBUG - Response > body: > b'<?xml version="1.0" > encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx000005ec21219335dafaa-00634d76d4-2f74a-default</RequestId><HostId>2f74a-default-default</HostId></Error>' > 2022-10-17 08:37:56,168 - MainThread - botocore.hooks - DEBUG - Event > needs-retry.sns.CreateTopic: calling handler <bound method > RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object > at 0x7fb5dd99e400>> > 2022-10-17 08:37:56,168 - MainThread - botocore.retries.standard - DEBUG - > Not retrying request. > 2022-10-17 08:37:56,168 - MainThread - botocore.hooks - DEBUG - Event > after-call.sns.CreateTopic: calling handler <bound method > RetryQuotaChecker.release_retry_quota of > <botocore.retries.standard.RetryQuotaChecker object at 0x7fb5dd567f70>> > 2022-10-17 08:37:56,169 - MainThread - awscli.clidriver - DEBUG - > Exception caught in main() > Traceback (most recent call last): > File "awscli/clidriver.py", line 458, in main > File "awscli/clidriver.py", line 593, in __call__ > File "awscli/clidriver.py", line 769, in __call__ > File "awscli/clidriver.py", line 900, in invoke > File "awscli/clidriver.py", line 912, in _make_client_call > File "awscli/botocore/client.py", line 309, in _api_call > File "awscli/botocore/client.py", line 639, in _make_api_call > botocore.exceptions.ClientError: An error occurred (Unknown) when calling > the CreateTopic operation: Unknown > > An error occurred (Unknown) when calling the CreateTopic operation: Unknown > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |