Re: Openstack glance ceph rbd_store_user authentification problem

[Vikrant Verma <vikrantverma99@xxxxxxxxx>]

Hi Steffen,

When i run the command  " rbd --id glance ls images" i am getting following error 

2013-12-27 17:44:56.493670 7fe56c754780  0 librados: client.glance authentication error (1) Operation not permitted

rbd: couldn't connect to the cluster!

and so is the case with cinder. Please let me know where i am wrong

I want to create a Ceph Boot Volume and launch an instance from this volume. I have posted my actual issue at - 

https://answers.launchpad.net/cinder/+question/241336

Please find  my glance-api.conf  as an attachment.

On Wed, Dec 11, 2013 at 2:54 PM, Steffen Thorhauer <thorhaue@xxxxxxxxxxxxxxxxxxxxxxx> wrote:

Hi Vikrant,
there is a difference between ceph openstack doc http://ceph.com/docs/master/rbd/rbd-openstack/ and your glance-api.conf (ubuntu) defaults.

rbd_store_ceph_conf = /etc/ceph/ceph.conf

# RADOS user to authenticate as (only applicable if using cephx)

rbd_store_user = glance

# RADOS pool in which images are stored

rbd_store_pool = images

I think you should change only the following line in your glance-api.conf
default_store = rbd


the default name of the rbd pool is images:
ceph osd pool create images 128

the default user is glance:
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'

on your glance server side you need then /etc/ceph/ceph.conf and /etc/ceph/ceph.client.glance.keyring

to test the authentication on your glance host:
   rbd --id glance ls images

and now it should work.

Regards,
  Steffen

On 12/11/2013 09:42 AM, Vikrant Verma wrote:

Hi Karan,

I want to set up a ceph boot volume, i am following the below link for that

http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-glance

Here is the glance-api.conf contents - 

[DEFAULT]

# Show more verbose log output (sets INFO log level output)

verbose = True

# Show debugging output in logs (sets DEBUG log level output)

debug = True

# Which backend scheme should Glance use by default is not specified

# in a request to add a new image to Glance? Known schemes are determined

# by the known_stores option below.

# Default: 'file'

default_store = file

# List of which store classes and store class locations are

# currently known to glance at startup.

#known_stores = glance.store.filesystem.Store,

#               glance.store.http.Store,

#               glance.store.rbd.Store,

#               glance.store.s3.Store,

#               glance.store.swift.Store,

#               glance.store.sheepdog.Store,

#               glance.store.cinder.Store,

# Maximum image size (in bytes) that may be uploaded through the

# Glance API server. Defaults to 1 TB.

# WARNING: this value should only be increased after careful consideration

# and must be set to a value under 8 EB (9223372036854775808).

#image_size_cap = 1099511627776

# Address to bind the API server

bind_host = 0.0.0.0

# Port the bind the API server to

bind_port = 9292

# Log to this file. Make sure you do not set the same log

# file for both the API and registry servers!

log_file = /var/log/glance/api.log

# Backlog requests when creating socket

backlog = 4096

# TCP_KEEPIDLE value in seconds when creating socket.

# Not supported on OS X.

#tcp_keepidle = 600

# API to use for accessing data. Default value points to sqlalchemy

# package, it is also possible to use: glance.db.registry.api

# data_api = glance.db.sqlalchemy.api

# SQLAlchemy connection string for the reference implementation

# registry server. Any valid SQLAlchemy connection string is fine.

# See: http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine

sql_connection = mysql://glance:intel123@vikrant/glance

# Period in seconds after which SQLAlchemy should reestablish its connection

# to the database.

#

# MySQL uses a default `wait_timeout` of 8 hours, after which it will drop

# idle connections. This can result in 'MySQL Gone Away' exceptions. If you

# notice this, you can lower this value to ensure that SQLAlchemy reconnects

# before MySQL can drop the connection.

sql_idle_timeout = 3600

# Number of Glance API worker processes to start.

# On machines with more than one CPU increasing this value

# may improve performance (especially if using SSL with

# compression turned on). It is typically recommended to set

# this value to the number of CPUs present on your machine.

workers = 1

# Role used to identify an authenticated user as administrator

#admin_role = admin

# Allow unauthenticated users to access the API with read-only

# privileges. This only applies when using ContextMiddleware.

#allow_anonymous_access = False

# Allow access to version 1 of glance api

#enable_v1_api = True

# Allow access to version 2 of glance api

#enable_v2_api = True

# Return the URL that references where the data is stored on

# the backend storage system.  For example, if using the

# file system store a URL of 'file:///path/to/image' will

# be returned to the user in the 'direct_url' meta-data field.

# The default value is false.

#show_image_direct_url = False

# Send headers containing user and tenant information when making requests to

# the v1 glance registry. This allows the registry to function as if a user is

# authenticated without the need to authenticate a user itself using the

# auth_token middleware.

# The default value is false.

#send_identity_headers = False

# Supported values for the 'container_format' image attribute

#container_formats=ami,ari,aki,bare,ovf

# Supported values for the 'disk_format' image attribute

#disk_formats=ami,ari,aki,vhd,vmdk,raw,qcow2,vdi,iso

# Directory to use for lock files. Default to a temp directory

# (string value). This setting needs to be the same for both

# glance-scrubber and glance-api.

#lock_path=<None>

#

# Property Protections config file

# This file contains the rules for property protections and the roles

# associated with it.

# If this config value is not specified, by default, property protections

# won't be enforced.

# If a value is specified and the file is not found, then an

# HTTPInternalServerError will be thrown.

#property_protection_file =

# Set a system wide quota for every user.  This value is the total number

# of bytes that a user can use across all storage systems.  A value of

# 0 means unlimited.

#user_storage_quota = 0

# ================= Syslog Options ============================

# Send logs to syslog (/dev/log) instead of to file specified

# by `log_file`

#use_syslog = False

# Facility to use. If unset defaults to LOG_USER.

#syslog_log_facility = LOG_LOCAL0

# ================= SSL Options ===============================

# Certificate file to use when starting API server securely

#cert_file = /path/to/certfile

# Private key file to use when starting API server securely

#key_file = /path/to/keyfile

# CA certificate file to use to verify connecting clients

#ca_file = /path/to/cafile

# ================= Security Options ==========================

# AES key for encrypting store 'location' metadata, including

# -- if used -- Swift or S3 credentials

# Should be set to a random string of length 16, 24 or 32 bytes

#metadata_encryption_key = <16, 24 or 32 char registry metadata key>

# ============ Registry Options ===============================

# Address to find the registry server

registry_host = 0.0.0.0

# Port the registry server is listening on

registry_port = 9191

# What protocol to use when connecting to the registry server?

# Set to https for secure HTTP communication

registry_client_protocol = http

# The path to the key file to use in SSL connections to the

# registry server, if any. Alternately, you may set the

# GLANCE_CLIENT_KEY_FILE environ variable to a filepath of the key file

#registry_client_key_file = /path/to/key/file

# The path to the cert file to use in SSL connections to the

# registry server, if any. Alternately, you may set the

# GLANCE_CLIENT_CERT_FILE environ variable to a filepath of the cert file

#registry_client_cert_file = /path/to/cert/file

# The path to the certifying authority cert file to use in SSL connections

# to the registry server, if any. Alternately, you may set the

# GLANCE_CLIENT_CA_FILE environ variable to a filepath of the CA cert file

#registry_client_ca_file = /path/to/ca/file

# When using SSL in connections to the registry server, do not require

# validation via a certifying authority. This is the registry's equivalent of

# specifying --insecure on the command line using glanceclient for the API

# Default: False

#registry_client_insecure = False

# The period of time, in seconds, that the API server will wait for a registry

# request to complete. A value of '0' implies no timeout.

# Default: 600

#registry_client_timeout = 600

# Whether to automatically create the database tables.

# Default: False

#db_auto_create = False

# Enable DEBUG log messages from sqlalchemy which prints every database

# query and response.

# Default: False

#sqlalchemy_debug = True

# ============ Notification System Options =====================

# Notifications can be sent when images are create, updated or deleted.

# There are three methods of sending notifications, logging (via the

# log_file directive), rabbit (via a rabbitmq queue), qpid (via a Qpid

# message queue), or noop (no notifications sent, the default)

notifier_strategy = noop

# Configuration options if sending notifications via rabbitmq (these are

# the defaults)

rabbit_host = localhost

rabbit_port = 5672

rabbit_use_ssl = false

rabbit_userid = guest

rabbit_password = guest

rabbit_virtual_host = /

rabbit_notification_exchange = glance

rabbit_notification_topic = notifications

rabbit_durable_queues = False

# Configuration options if sending notifications via Qpid (these are

# the defaults)

qpid_notification_exchange = glance

qpid_notification_topic = notifications

qpid_host = localhost

qpid_port = 5672

qpid_username =

qpid_password =

qpid_sasl_mechanisms =

qpid_reconnect_timeout = 0

qpid_reconnect_limit = 0

qpid_reconnect_interval_min = 0

qpid_reconnect_interval_max = 0

qpid_reconnect_interval = 0

qpid_heartbeat = 5

# Set to 'ssl' to enable SSL

qpid_protocol = tcp

qpid_tcp_nodelay = True

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store

# writes image data to

filesystem_store_datadir = /var/lib/glance/images/

# A path to a JSON file that contains metadata describing the storage

# system.  When show_multiple_locations is True the information in this

# file will be returned with any location that is contained in this

# store.

#filesystem_store_metadata_file = None

# ============ Swift Store Options =============================

# Version of the authentication service to use

# Valid versions are '2' for keystone and '1' for swauth and rackspace

swift_store_auth_version = 2

# Address where the Swift authentication service lives

# Valid schemes are 'http://' and 'https://'

# If no scheme specified,  default to 'https://'

# For swauth, use something like '127.0.0.1:8080/v1.0/'

swift_store_auth_address = 127.0.0.1:5000/v2.0/

# User to authenticate against the Swift authentication service

# If you use Swift authentication service, set it to 'account':'user'

# where 'account' is a Swift storage account and 'user'

# is a user in that account

swift_store_user = jdoe:jdoe

# Auth key for the user authenticating against the

# Swift authentication service

swift_store_key = a86850deb2742ec3cb41518e26aa2d89

# Container within the account that the account should use

# for storing images in Swift

swift_store_container = glance

# Do we create the container if it does not exist?

swift_store_create_container_on_put = False

# What size, in MB, should Glance start chunking image files

# and do a large object manifest in Swift? By default, this is

# the maximum object size in Swift, which is 5GB

swift_store_large_object_size = 5120

# When doing a large object manifest, what size, in MB, should

# Glance write chunks to Swift? This amount of data is written

# to a temporary disk buffer during the process of chunking

# the image file, and the default is 200MB

swift_store_large_object_chunk_size = 200

# Whether to use ServiceNET to communicate with the Swift storage servers.

# (If you aren't RACKSPACE, leave this False!)

#

# To use ServiceNET for authentication, prefix hostname of

# `swift_store_auth_address` with 'snet-'.

# Ex. https://example.com/v1.0/ -> https://snet-example.com/v1.0/

swift_enable_snet = False

# If set to True enables multi-tenant storage mode which causes Glance images

# to be stored in tenant specific Swift accounts.

#swift_store_multi_tenant = False

# A list of swift ACL strings that will be applied as both read and

# write ACLs to the containers created by Glance in multi-tenant

# mode. This grants the specified tenants/users read and write access

# to all newly created image objects. The standard swift ACL string

# formats are allowed, including:

# <tenant_id>:<username>

# <tenant_name>:<username>

# *:<username>

# Multiple ACLs can be combined using a comma separated list, for

# example: swift_store_admin_tenants = service:glance,*:admin

#swift_store_admin_tenants =

# The region of the swift endpoint to be used for single tenant. This setting

# is only necessary if the tenant has multiple swift endpoints.

#swift_store_region =

# If set to False, disables SSL layer compression of https swift requests.

# Setting to 'False' may improve performance for images which are already

# in a compressed format, eg qcow2. If set to True, enables SSL layer

# compression (provided it is supported by the target swift proxy).

#swift_store_ssl_compression = True

# ============ S3 Store Options =============================

# Address where the S3 authentication service lives

# Valid schemes are 'http://' and 'https://'

# If no scheme specified,  default to 'http://'

s3_store_host = 127.0.0.1:8080/v1.0/

# User to authenticate against the S3 authentication service

s3_store_access_key = <20-char AWS access key>

# Auth key for the user authenticating against the

# S3 authentication service

s3_store_secret_key = <40-char AWS secret key>

# Container within the account that the account should use

# for storing images in S3. Note that S3 has a flat namespace,

# so you need a unique bucket name for your glance images. An

# easy way to do this is append your AWS access key to "glance".

# S3 buckets in AWS *must* be lowercased, so remember to lowercase

# your AWS access key if you use it in your bucket name below!

s3_store_bucket = <lowercased 20-char aws access key>glance

# Do we create the bucket if it does not exist?

s3_store_create_bucket_on_put = False

# When sending images to S3, the data will first be written to a

# temporary buffer on disk. By default the platform's temporary directory

# will be used. If required, an alternative directory can be specified here.

#s3_store_object_buffer_dir = /path/to/dir

# When forming a bucket url, boto will either set the bucket name as the

# subdomain or as the first token of the path. Amazon's S3 service will

# accept it as the subdomain, but Swift's S3 middleware requires it be

# in the path. Set this to 'path' or 'subdomain' - defaults to 'subdomain'.

#s3_store_bucket_url_format = subdomain

# ============ RBD Store Options =============================

# Ceph configuration file path

# If using cephx authentication, this file should

# include a reference to the right keyring

# in a client.<USER> section

rbd_store_ceph_conf = /etc/ceph/ceph.conf

# RADOS user to authenticate as (only applicable if using cephx)

rbd_store_user = glance

# RADOS pool in which images are stored

rbd_store_pool = images

# Images will be chunked into objects of this size (in megabytes).

# For best performance, this should be a power of two

rbd_store_chunk_size = 8

# ============ Sheepdog Store Options =============================

sheepdog_store_address = localhost

sheepdog_store_port = 7000

# Images will be chunked into objects of this size (in megabytes).

# For best performance, this should be a power of two

sheepdog_store_chunk_size = 64

# ============ Cinder Store Options ===============================

# Info to match when looking for cinder in the service catalog

# Format is : separated values of the form:

# <service_type>:<service_name>:<endpoint_type> (string value)

#cinder_catalog_info = volume:cinder:publicURL

# Override service catalog lookup with template for cinder endpoint

# e.g. http://localhost:8776/v1/%(project_id)s (string value)

#cinder_endpoint_template = <None>

# Region name of this node (string value)

#os_region_name = <None>

# Location of ca certicates file to use for cinder client requests

# (string value)

#cinder_ca_certificates_file = <None>

# Number of cinderclient retries on failed http calls (integer value)

#cinder_http_retries = 3

# Allow to perform insecure SSL requests to cinder (boolean value)

#cinder_api_insecure = False

# ============ Delayed Delete Options =============================

# Turn on/off delayed delete

delayed_delete = False

# Delayed delete time in seconds

scrub_time = 43200

# Directory that the scrubber will use to remind itself of what to delete

# Make sure this is also set in glance-scrubber.conf

scrubber_datadir = /var/lib/glance/scrubber

# =============== Image Cache Options =============================

# Base directory that the Image Cache uses

image_cache_dir = /var/lib/glance/image-cache/

[keystone_authtoken]

auth_host = vikrant

auth_port = 35357

auth_protocol = http

admin_tenant_name = service

admin_user = glance

admin_password = intel123

[paste_deploy]

# Name of the paste configuration file that defines the available pipelines

#config_file = glance-api-paste.ini

# Partial name of a pipeline in your paste configuration file with the

# service name removed. For example, if your paste section name is

# [pipeline:glance-api-keystone], you would configure the flavor below

# as 'keystone'.

#flavor=

Regards,

Vikrant 

On Tue, Dec 10, 2013 at 7:04 PM, Karan Singh <ksingh@xxxxxx> wrote:

Hi Vikrant

Can you share  ceph auth list  and your glance-api.conf file output.

What are your plans with respect to configuration , what you want to achieve.

Many Thanks
Karan Singh

---

From: "Vikrant Verma" <vikrantverma99@xxxxxxxxx>
To: thorhaue@xxxxxxxxxxxxxxxxxxxxxxx
Cc: ceph-users@xxxxxxxxxxxxxx
Sent: Tuesday, 10 December, 2013 1:20:32 PM
Subject: Openstack glance ceph rbd_store_user authentification        problem

Hi Steffen,

WIth respect to your post as mentioned in the below link

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-August/003370.html

I am facing the same issue, here is my error log from api.log

"2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/dist-packages/rados.py", line 367, in connect

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils     raise make_ex(ret, "error calling connect")

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils ObjectNotFound: error calling connect

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils

2013-12-10 02:47:36.157 32509 DEBUG glance.registry.client.v1.api [5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Updating image metadata for image a66d0475-70fd-49f0-86c3-ddc47cffdcaf... update_image_metadata /usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py:168

2013-12-10 02:47:36.157 32509 DEBUG glance.common.client [5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Constructed URL: http://0.0.0.0:9191/images/a66d0475-70fd-49f0-86c3-ddc47cffdcaf _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:408

2013-12-10 02:47:36.221 32509 DEBUG glance.registry.client.v1.client [5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Registry request PUT /images/a66d0475-70fd-49f0-86c3-ddc47cffdcaf HTTP 200 request id req-8aba73e7-3390-4201-acdb-73ae1ddf80f7 do_request /usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py:115"

As per your suggestion "configured a client.glance auth and it worked with the "implicit" glance user"

Please let me know to configure it.

Regards,

Vikrant

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
______________________________________________________________________
Steffen Thorhauer

Department of Technichal and Business Information Systems (ITI)
Faculty of Computer Science (FIN)
  Otto von Guericke University Magdeburg
Universitaetsplatz 2
39106 Magdeburg, Germany

phone: 0391 67 52996
fax: 0391 67 12341
email: sth@xxxxxxxxxxxxxxxxxxxxxxx
url: http://wwwiti.cs.uni-magdeburg.de/~thorhaue        

Attachment: glance-api.conf
Description: Binary data

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com