Re: Openstack glance ceph rbd_store_user authentification problem

[hemant burman <hemant.burman@xxxxxxxxx>]

My best guess is the keyring is not added to the cluster which you are using for RBD, please do ceph auth list and check the key present there is also in your config

-Hemant

On Fri, Dec 27, 2013 at 8:17 PM, Karan Singh <ksingh@xxxxxx> wrote:

Hello Vikrant

Check permission of your of cinder.conf and glance-api.conf configuration file.

Many Thanks
Karan Singh

---

From: "Julien Calvet" <julien.calvet@xxxxxxxxxx>
To: "Vikrant Verma" <vikrantverma99@xxxxxxxxx>
Cc: "Steffen Thorhauer" <thorhaue@xxxxxxxxxxxxxxxxxxxxxxx>, ceph-users@xxxxxxxxxxxxxx
Sent: Friday, 27 December, 2013 2:29:53 PM
Subject: Re: [ceph-users] Openstack glance ceph rbd_store_user        authentification problem

Use sudo before your command

Julien

Le 27 déc. 2013 à 13:25, Vikrant Verma <vikrantverma99@xxxxxxxxx> a écrit :

Hi Steffen,

When i run the command  " rbd --id glance ls images" i am getting following error 

2013-12-27 17:44:56.493670 7fe56c754780  0 librados: client.glance authentication error (1) Operation not permitted

rbd: couldn't connect to the cluster!

and so is the case with cinder. Please let me know where i am wrong

I want to create a Ceph Boot Volume and launch an instance from this volume. I have posted my actual issue at - 

https://answers.launchpad.net/cinder/+question/241336

Please find  my glance-api.conf  as an attachment.

On Wed, Dec 11, 2013 at 2:54 PM, Steffen Thorhauer <thorhaue@xxxxxxxxxxxxxxxxxxxxxxx> wrote:

Hi Vikrant,
there is a difference between ceph openstack doc http://ceph.com/docs/master/rbd/rbd-openstack/ and your glance-api.conf (ubuntu) defaults.

rbd_store_ceph_conf = /etc/ceph/ceph.conf

# RADOS user to authenticate as (only applicable if using cephx)

rbd_store_user = glance

# RADOS pool in which images are stored

rbd_store_pool = images

I think you should change only the following line in your glance-api.conf
default_store = rbd


the default name of the rbd pool is images:
ceph osd pool create images 128

the default user is glance:
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'

on your glance server side you need then /etc/ceph/ceph.conf and /etc/ceph/ceph.client.glance.keyring

to test the authentication on your glance host:
   rbd --id glance ls images

and now it should work.

Regards,
  Steffen

On 12/11/2013 09:42 AM, Vikrant Verma wrote:

Hi Karan,

I want to set up a ceph boot volume, i am following the below link for that

http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-glance

Here is the glance-api.conf contents - 

[DEFAULT]

# Show more verbose log output (sets INFO log level output)

verbose = True

# Show debugging output in logs (sets DEBUG log level output)

debug = True

# Which backend scheme should Glance use by default is not specified

# in a request to add a new image to Glance? Known schemes are determined

# by the known_stores option below.

# Default: 'file'

default_store = file

# List of which store classes and store class locations are

# currently known to glance at startup.

#known_stores = glance.store.filesystem.Store,

#               glance.store.http.Store,

#               glance.store.rbd.Store,

#               glance.store.s3.Store,

#               glance.store.swift.Store,

#               glance.store.sheepdog.Store,

#               glance.store.cinder.Store,

# Maximum image size (in bytes) that may be uploaded through the

# Glance API server. Defaults to 1 TB.

# WARNING: this value should only be increased after careful consideration

# and must be set to a value under 8 EB (9223372036854775808).

#image_size_cap = 1099511627776

# Address to bind the API server

bind_host = 0.0.0.0

# Port the bind the API server to

bind_port = 9292

# Log to this file. Make sure you do not set the same log

# file for both the API and registry servers!

log_file = /var/log/glance/api.log

# Backlog requests when creating socket

backlog = 4096

# TCP_KEEPIDLE value in seconds when creating socket.

# Not supported on OS X.

#tcp_keepidle = 600

# API to use for accessing data. Default value points to sqlalchemy

# package, it is also possible to use: glance.db.registry.api

# data_api = glance.db.sqlalchemy.api

# SQLAlchemy connection string for the reference implementation

# registry server. Any valid SQLAlchemy connection string is fine.

# See: http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine

sql_connection = mysql://glance:intel123@vikrant/glance

# Period in seconds after which SQLAlchemy should reestablish its connection

# to the database.

#

# MySQL uses a default `wait_timeout` of 8 hours, after which it will drop

# idle connections. This can result in 'MySQL Gone Away' exceptions. If you

# notice this, you can lower this value to ensure that SQLAlchemy reconnects

# before MySQL can drop the connection.

sql_idle_timeout = 3600

# Number of Glance API worker processes to start.

# On machines with more than one CPU increasing this value

# may improve performance (especially if using SSL with

# compression turned on). It is typically recommended to set

# this value to the number of CPUs present on your machine.

workers = 1

# Role used to identify an authenticated user as administrator

#admin_role = admin

# Allow unauthenticated users to access the API with read-only

# privileges. This only applies when using ContextMiddleware.

#allow_anonymous_access = False

# Allow access to version 1 of glance api

#enable_v1_api = True

# Allow access to version 2 of glance api

#enable_v2_api = True

# Return the URL that references where the data is stored on

# the backend storage system.  For example, if using the

# file system store a URL of 'file:///path/to/image' will

# be returned to the user in the 'direct_url' meta-data field.

# The default value is false.

#show_image_direct_url = False

# Send headers containing user and tenant information when making requests to

# the v1 glance registry. This allows the registry to function as if a user is

# authenticated without the need to authenticate a user itself using the

# auth_token middleware.

# The default value is false.

#send_identity_headers = False

# Supported values for the 'container_format' image attribute

#container_formats=ami,ari,aki,bare,ovf

# Supported values for the 'disk_format' image attribute

#disk_formats=ami,ari,aki,vhd,vmdk,raw,qcow2,vdi,iso

# Directory to use for lock files. Default to a temp directory

# (string value). This setting needs to be the same for both

# glance-scrubber and glance-api.

#lock_path=<None>

#

# Property Protections config file

# This file contains the rules for property protections and the roles

# associated with it.

# If this config value is not specified, by default, property protections

# won't be enforced.

# If a value is specified and the file is not found, then an

# HTTPInternalServerError will be thrown.

#property_protection_file =

# Set a system wide quota for every user.  This value is the total number

# of bytes that a user can use across all storage systems.  A value of

# 0 means unlimited.

#user_storage_quota = 0

# ================= Syslog Options ============================

# Send logs to syslog (/dev/log) instead of to file specified

# by `log_file`

#use_syslog = False

# Facility to use. If unset defaults to LOG_USER.

#syslog_log_facility = LOG_LOCAL0

# ================= SSL Options ===============================

# Certificate file to use when starting API server securely

#cert_file = /path/to/certfile

# Private key file to use when starting API server securely

#key_file = /path/to/keyfile

# CA certificate file to use to verify connecting clients

#ca_file = /path/to/cafile

# ================= Security Options ==========================

# AES key for encrypting store 'location' metadata, including

# -- if used -- Swift or S3 credentials

# Should be set to a random string of length 16, 24 or 32 bytes

#metadata_encryption_key = <16, 24 or 32 char registry metadata key>

# ============ Registry Options ===============================

# Address to find the registry server

registry_host = 0.0.0.0

# Port the registry server is listening on

registry_port = 9191

# What protocol to use when connecting to the registry server?

# Set to https for secure HTTP communication

registry_client_protocol = http

# The path to the key file to use in SSL connections to the

# registry server, if any. Alternately, you may set the

# GLANCE_CLIENT_KEY_FILE environ variable to a filepath of the key file

#registry_client_key_file = /path/to/key/file

# The path to the cert file to use in SSL connections to the

# registry server, if any. Alternately, you may set the

# GLANCE_CLIENT_CERT_FILE environ variable to a filepath of the cert file

#registry_client_cert_file = /path/to/cert/file

# The path to the certifying authority cert file to use in SSL connections

# to the registry server, if any. Alternately, you may set the

# GLANCE_CLIENT_CA_FILE environ variable to a filepath of the CA cert file

#registry_client_ca_file = /path/to/ca/file

# When using SSL in connections to the registry server, do not require

# validation via a certifying authority. This is the registry's equivalent of

# specifying --insecure on the command line using glanceclient for the API

# Default: False

#registry_client_insecure = False

# The period of time, in seconds, that the API server will wait for a registry

# request to complete. A value of '0' implies no timeout.

# Default: 600

#registry_client_timeout = 600

# Whether to automatically create the database tables.

# Default: False

#db_auto_create = False

# Enable DEBUG log messages from sqlalchemy which prints every database

# query and response.

# Default: False

#sqlalchemy_debug = True

# ============ Notification System Options =====================

# Notifications can be sent when images are create, updated or deleted.

# There are three methods of sending notifications, logging (via the

# log_file directive), rabbit (via a rabbitmq queue), qpid (via a Qpid

# message queue), or noop (no notifications sent, the default)

notifier_strategy = noop

# Configuration options if sending notifications via rabbitmq (these are

# the defaults)

rabbit_host = localhost

rabbit_port = 5672

rabbit_use_ssl = false

rabbit_userid = guest

rabbit_password = guest

rabbit_virtual_host = /

rabbit_notification_exchange = glance

rabbit_notification_topic = notifications

rabbit_durable_queues = False

# Configuration options if sending notifications via Qpid (these are

# the defaults)

qpid_notification_exchange = glance

qpid_notification_topic = notifications

qpid_host = localhost

qpid_port = 5672

qpid_username =

qpid_password =

qpid_sasl_mechanisms =

qpid_reconnect_timeout = 0

qpid_reconnect_limit = 0

qpid_reconnect_interval_min = 0

qpid_reconnect_interval_max = 0

qpid_reconnect_interval = 0

qpid_heartbeat = 5

# Set to 'ssl' to enable SSL

qpid_protocol = tcp

qpid_tcp_nodelay = True

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store

# writes image data to

filesystem_store_datadir = /var/lib/glance/images/

# A path to a JSON file that contains metadata describing the storage

# system.  When show_multiple_locations is True the information in this

# file will be returned with any location that is contained in this

# store.

#filesystem_store_metadata_file = None

# ============ Swift Store Options =============================

# Version of the authentication service to use

# Valid versions are '2' for keystone and '1' for swauth and rackspace

swift_store_auth_version = 2

# Address where the Swift authentication service lives

# Valid schemes are 'http://' and 'https://'

# If no scheme specified,  default to 'https://'

# For swauth, use something like '127.0.0.1:8080/v1.0/'

swift_store_auth_address = 127.0.0.1:5000/v2.0/

# User to authenticate against the Swift authentication service

# If you use Swift authentication service, set it to 'account':'user'

# where 'account' is a Swift storage account and 'user'

# is a user in that account

swift_store_user = jdoe:jdoe

# Auth key for the user authenticating against the

# Swift authentication service

swift_store_key = a86850deb2742ec3cb41518e26aa2d89

# Container within the account that the account should use

# for storing images in Swift

swift_store_container = glance

# Do we create the container if it does not exist?

swift_store_create_container_on_put = False

# What size, in MB, should Glance start chunking image files

# and do a large object manifest in Swift? By default, this is

# the maximum object size in Swift, which is 5GB

swift_store_large_object_size = 5120

# When doing a large object manifest, what size, in MB, should

# Glance write chunks to Swift? This amount of data is written

# to a temporary disk buffer during the process of chunking

# the image file, and the default is 200MB

swift_store_large_object_chunk_size = 200

# Whether to use ServiceNET to communicate with the Swift storage servers.

# (If you aren't RACKSPACE, leave this False!)

#

# To use ServiceNET for authentication, prefix hostname of

# `swift_store_auth_address` with 'snet-'.

# Ex. https://example.com/v1.0/ -> https://snet-example.com/v1.0/

swift_enable_snet = False

# If set to True enables multi-tenant storage mode which causes Glance images

# to be stored in tenant specific Swift accounts.

#swift_store_multi_tenant = False

# A list of swift ACL strings that will be applied as both read and

# write ACLs to the containers created by Glance in multi-tenant

# mode. This grants the specified tenants/users read and write access

# to all newly created image objects. The standard swift ACL string

# formats are allowed, including:

# <tenant_id>:<username>

# <tenant_name>:<username>

# *:<username>

# Multiple ACLs can be combined using a comma separated list, for

# example: swift_store_admin_tenants = service:glance,*:admin

#swift_store_admin_tenants =

# The region of the swift endpoint to be used for single tenant. This setting

# is only necessary if the tenant has multiple swift endpoints.

#swift_store_region =

# If set to False, disables SSL layer compression of https swift requests.

# Setting to 'False' may improve performance for images which are already

# in a compressed format, eg qcow2. If set to True, enables SSL layer

# compression (provided it is supported by the target swift proxy).

#swift_store_ssl_compression = True

# ============ S3 Store Options =============================

# Address where the S3 authentication service lives

# Valid schemes are 'http://' and 'https://'

# If no scheme specified,  default to 'http://'

s3_store_host = 127.0.0.1:8080/v1.0/

# User to authenticate against the S3 authentication service

s3_store_access_key = <20-char AWS access key>

# Auth key for the user authenticating against the

# S3 authentication service

s3_store_secret_key = <40-char AWS secret key>

# Container within the account that the account should use

# for storing images in S3. Note that S3 has a flat namespace,

# so you need a unique bucket name for your glance images. An

# easy way to do this is append your AWS access key to "glance".

# S3 buckets in AWS *must* be lowercased, so remember to lowercase

# your AWS access key if you use it in your bucket name below!

s3_store_bucket = <lowercased 20-char aws access key>glance

# Do we create the bucket if it does not exist?

s3_store_create_bucket_on_put = False

# When sending images to S3, the data will first be written to a

# temporary buffer on disk. By default the platform's temporary directory

# will be used. If required, an alternative directory can be specified here.

#s3_store_object_buffer_dir = /path/to/dir

# When forming a bucket url, boto will either set the bucket name as the

# subdomain or as the first token of the path. Amazon's S3 service will

# accept it as the subdomain, but Swift's S3 middleware requires it be

# in the path. Set this to 'path' or 'subdomain' - defaults to 'subdomain'.

#s3_store_bucket_url_format = subdomain

# ============ RBD Store Options =============================

# Ceph configuration file path

# If using cephx authentication, this file should

# include a reference to the right keyring

# in a client.<USER> section

rbd_store_ceph_conf = /etc/ceph/ceph.conf

# RADOS user to authenticate as (only applicable if using cephx)

rbd_store_user = glance

# RADOS pool in which images are stored

rbd_store_pool = images

# Images will be chunked into objects of this size (in megabytes).

# For best performance, this should be a power of two

rbd_store_chunk_size = 8

# ============ Sheepdog Store Options =============================

sheepdog_store_address = localhost

sheepdog_store_port = 7000

# Images will be chunked into objects of this size (in megabytes).

# For best performance, this should be a power of two

sheepdog_store_chunk_size = 64

# ============ Cinder Store Options ===============================

# Info to match when looking for cinder in the service catalog

# Format is : separated values of the form:

# <service_type>:<service_name>:<endpoint_type> (string value)

#cinder_catalog_info = volume:cinder:publicURL

# Override service catalog lookup with template for cinder endpoint

# e.g. http://localhost:8776/v1/%(project_id)s (string value)

#cinder_endpoint_template = <None>

# Region name of this node (string value)

#os_region_name = <None>

# Location of ca certicates file to use for cinder client requests

# (string value)

#cinder_ca_certificates_file = <None>

# Number of cinderclient retries on failed http calls (integer value)

#cinder_http_retries = 3

# Allow to perform insecure SSL requests to cinder (boolean value)

#cinder_api_insecure = False

# ============ Delayed Delete Options =============================

# Turn on/off delayed delete

delayed_delete = False

# Delayed delete time in seconds

scrub_time = 43200

# Directory that the scrubber will use to remind itself of what to delete

# Make sure this is also set in glance-scrubber.conf

scrubber_datadir = /var/lib/glance/scrubber

# =============== Image Cache Options =============================

# Base directory that the Image Cache uses

image_cache_dir = /var/lib/glance/image-cache/

[keystone_authtoken]

auth_host = vikrant

auth_port = 35357

auth_protocol = http

admin_tenant_name = service

admin_user = glance

admin_password = intel123

[paste_deploy]

# Name of the paste configuration file that defines the available pipelines

#config_file = glance-api-paste.ini

# Partial name of a pipeline in your paste configuration file with the

# service name removed. For example, if your paste section name is

# [pipeline:glance-api-keystone], you would configure the flavor below

# as 'keystone'.

#flavor=

Regards,

Vikrant 

On Tue, Dec 10, 2013 at 7:04 PM, Karan Singh <ksingh@xxxxxx> wrote:

Hi Vikrant

Can you share  ceph auth list  and your glance-api.conf file output.

What are your plans with respect to configuration , what you want to achieve.

Many Thanks
Karan Singh

---

From: "Vikrant Verma" <vikrantverma99@xxxxxxxxx>
To: thorhaue@xxxxxxxxxxxxxxxxxxxxxxx
Cc: ceph-users@xxxxxxxxxxxxxx
Sent: Tuesday, 10 December, 2013 1:20:32 PM
Subject: Openstack glance ceph rbd_store_user authentification        problem

Hi Steffen,

WIth respect to your post as mentioned in the below link

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-August/003370.html

I am facing the same issue, here is my error log from api.log

"2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/dist-packages/rados.py", line 367, in connect

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils     raise make_ex(ret, "error calling connect")

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils ObjectNotFound: error calling connect

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils

2013-12-10 02:47:36.157 32509 DEBUG glance.registry.client.v1.api [5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Updating image metadata for image a66d0475-70fd-49f0-86c3-ddc47cffdcaf... update_image_metadata /usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py:168

2013-12-10 02:47:36.157 32509 DEBUG glance.common.client [5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Constructed URL: http://0.0.0.0:9191/images/a66d0475-70fd-49f0-86c3-ddc47cffdcaf _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:408

2013-12-10 02:47:36.221 32509 DEBUG glance.registry.client.v1.client [5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Registry request PUT /images/a66d0475-70fd-49f0-86c3-ddc47cffdcaf HTTP 200 request id req-8aba73e7-3390-4201-acdb-73ae1ddf80f7 do_request /usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py:115"

As per your suggestion "configured a client.glance auth and it worked with the "implicit" glance user"

Please let me know to configure it.

Regards,

Vikrant

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
______________________________________________________________________
Steffen Thorhauer

Department of Technichal and Business Information Systems (ITI)
Faculty of Computer Science (FIN)
  Otto von Guericke University Magdeburg
Universitaetsplatz 2
39106 Magdeburg, Germany

phone: 0391 67 52996
fax: 0391 67 12341
email: sth@xxxxxxxxxxxxxxxxxxxxxxx
url: http://wwwiti.cs.uni-magdeburg.de/~thorhaue        

<glance-api.conf>

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com