Hi. I've been working on easy deployability and manageability of Ceph. This work is intended to be a complete replacement for mkcephfs, and integrate new product features instead of just automating the previous, clumsier, administration mechanisms. I'm using Chef to create and expand the cluster, but most of the new functionality is in making the OSDs more dynamic. The current work is in a branch called "ceph-1", and will be improved upon, but it is now at a stage where others should start looking at it. Here's a quick intro to what's there right now. Apologies for the formatting, I need to be on a plane fairly soon.. Rest assured, any command that looks clumsy is that way mostly because I haven't had time to make it prettier. I'll go through this with our QA and tech writer once the dust settles, to clean up the instructions. Limitations (all to be removed later): - supports only 1 monitor - journal is a file inside osd data directory - only supports 1 cluster (name hardcoded to "ceph"); later you will be able to run multiple clusters on the same hardware - no rgw, mds, or anything else but a RADOS/rbd cluster tested yet - no integration with e.g. OpenStack yet Open questions: - I removed the sysv-style init script (from the debian packaging). I'm not sure what to do with that. Older debs will still need it? - details of what goes where in e.g. the chef environment will change; input is welcome How to try it out: I need to leave to make to the airport in time, but the latest change is still compiling :( Wait till http://gitbuilder.ceph.com/ceph-deb-oneiric-x86_64-basic/ref/chef-1/sha1 says 4b75bccd52104d0ecd551e0656a30791b25fe032, hope for the best, and proceed: # create 3 vms running ubuntu 11.10 server; mine ended up being named chef02, inst03, inst04 # they need to be able to talk to each other, so do not use KVM's "user" networking, but NAT or bridged. (NAT is default for libvirt.) # make sure your vm has a unique hostname first, or it'll get confusing later; edit /etc/hostname, /etc/hosts, run "sudo hostname newname", re-login # source for this: http://wiki.opscode.com/display/chef/Installing+Chef+Server+on+Debian+or+Ubuntu+using+Packages # figure out the IP address of your chef server vm gpg --keyserver keys.gnupg.net --recv-keys 83EF826A gpg --export packages@xxxxxxxxxxx | sudo apt-key add - sudo tee /etc/apt/sources.list.d/chef.list <<EOF deb http://apt.opscode.com/ oneiric-0.10 main deb-src http://apt.opscode.com/ oneiric-0.10 main EOF sudo apt-get update sudo apt-get install chef # answer using the IP address of your chef server vm ┌───────────────────────┤ Configuring chef ├───────────────────────┐ │ This is the full URI that clients will use to connect to the │ │ server. │ │ . │ │ This will be used in /etc/chef/client.rb as 'chef_server_url'. │ │ │ │ URL of Chef Server (e.g., http://chef.example.com:4000): │ │ │ │ http://192.168.122.168:4000/____________________________________ │ │ │ │ <Ok> │ │ │ └──────────────────────────────────────────────────────────────────┘ sudo apt-get install chef-server # you MUST enter some password here or the installation will fail; no human will need to type this ever again ┌────────────────────────────┤ Configuring chef-solr ├─────────────────────────────┐ │ Set the password for the chef user in the AMQP server queue. Use │ │ RabbitMQ's rabbitmqctl program to set this password. The default user │ │ and vhost are assumed (chef and /chef, respectively). │ │ . │ │ RabbitMQ does not have the capability to read the password from a file, and │ │ this will be passed via "" on the command-line. As such, do not use shell │ │ meta-characters that could cause errors such as !. │ │ . │ │ This will be used in /etc/chef/solr.rb and /etc/chef/server.rb as 'amqp_pass'. │ │ │ │ New password for the 'chef' AMQP user in the RabbitMQ vhost "/chef": │ │ │ │ aipo6EY2i_______________________________________________________________________ │ │ │ │ <Ok> │ │ │ └──────────────────────────────────────────────────────────────────────────────────┘ # if you use the chef web ui, you'll be typing this in a lot.. "foo" is nice for demo-level security ┌───────────────────────┤ Configuring chef-server-webui ├───────────────────────┐ │ This sets a temporary first-use password to log into the Chef Server WebUI │ │ as the 'admin' user for the first time. Once logged in, the password should │ │ be changed immediately. │ │ . │ │ Once the chef-server-webui process is running, login using the username │ │ 'admin' using the password set here. │ │ . │ │ If a password is not entered, the webui default password for 'admin' will │ │ be used, which is displayed on the webui home page. The password must be │ │ at least 6 characters or the webui will not start properly. │ │ . │ │ This will be used in /etc/chef/webui.rb as 'web_ui_admin_default_password'. │ │ . │ │ │ │ New password for the 'admin' user in the Chef Server WebUI: │ │ │ │ foo__________________________________________________________________________ │ │ │ │ <Ok> │ │ │ └───────────────────────────────────────────────────────────────────────────────┘ # configure knife on the chef server ssh ubuntu@chef02 mkdir -p ~/.chef sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef sudo chown -R $USER ~/.chef knife configure -i WARNING: No knife configuration file found Where should I put the config file? [~/.chef/knife.rb] Please enter the chef server URL: [http://ubuntu:4000] http://localhost:4000 Please enter a clientname for the new client: [ubuntu] chef-admin Please enter the existing admin clientname: [chef-webui] Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem] .chef/webui.pem Please enter the validation clientname: [chef-validator] Please enter the location of the validation key: [/etc/chef/validation.pem] .chef/validation.pem Please enter the path to a chef repository (or leave blank): Creating initial API user... Created client[chef-admin] Configuration file written to /home/ubuntu/.chef/knife.rb ## OPTIONAL: if you want to use chef directly from your desktop, configure that now, following the instructions on the webpage given at the beginning # knife-block is a nice utility for connecting to multiple chef-servers, you might want that # http://os.greenandsecure.co.uk/2012/03/using-multiple-chef-servers-with-knife/ # configure knife on your desktop machine; this writes to ~/.gem gem1.9.1 install --no-rdoc --no-ri --user-install chef gem1.9.1 install --no-rdoc --no-ri --user-install knife-block ssh ubuntu@chef02 knife client create my-username -n -a -f /tmp/my-username.pem Created client[tv-test2] mkdir ~/.chef # grab the validation key cd ~/.chef scp ubuntu@chef02:/tmp/tv-test2.pem ~/.chef/my-username.pem knife block new This will create a new knife configuration file for you to use with knife-block Please provide a friendly name for the new configuration file: test2 Please enter the url to your Chef Server: http://192.168.122.168:4000/ Please enter an existing username or clientname for the API: [your-login] my-username Please enter the validation clientname: [chef-validator] Please enter the location of the validation key: [/etc/chef/validation.pem] .chef/validation-test2.pem Please enter the path to a chef repository (or leave blank): ***** You must place your client key in: /home/your-login/.chef/my-username.pem Before running commands with Knife! ***** You must place your validation key in: /etc/chef/validation.pem Before generating instance data with Knife! ***** Configuration file written to /home/your-login/.chef/knife-test2.rb /home/your-login/.chef/knife-test2.rb has been sucessfully created The available chef servers are: ... You are asking to change from test to test2. Are you sure? (Y/N) y The knife configuration has been updated to use test2 ## END OPTIONAL # now configure the actual ceph cluster vms # give them a data disk or two, in addition to the disk containing the root filesystem # (hint: for experiments ONLY, uncheck "Allocate entire disk now", choose "Device type: Virtio disk", "Cache mode: writeback" for best performance) # again, ensure unique hostnames # source for this: http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian # same stuff as beginning of the chef-server installation, repeat for every vm gpg --keyserver keys.gnupg.net --recv-keys 83EF826A gpg --export packages@xxxxxxxxxxx | sudo apt-key add - sudo tee /etc/apt/sources.list.d/chef.list <<EOF deb http://apt.opscode.com/ oneiric-0.10 main deb-src http://apt.opscode.com/ oneiric-0.10 main EOF sudo apt-get update sudo apt-get install chef # answer using the IP address of your chef server vm ┌───────────────────────┤ Configuring chef ├───────────────────────┐ │ This is the full URI that clients will use to connect to the │ │ server. │ │ . │ │ This will be used in /etc/chef/client.rb as 'chef_server_url'. │ │ │ │ URL of Chef Server (e.g., http://chef.example.com:4000): │ │ │ │ http://192.168.122.168:4000/____________________________________ │ │ │ │ <Ok> │ │ │ └──────────────────────────────────────────────────────────────────┘ # copy the validation key over, run on your desktop ssh chef02 sudo cat /etc/chef/validation.pem | ssh inst03 sudo tee /etc/chef/validation.pem >/dev/null # chef should now work on the new node! you can trigger a run manually: ssh inst03 sudo chef-client # if you set up your workstation as a knife client, you can do the following there; if not, you have to do this logged in to the chef server # doublecheck knife node list chef02 inst03 inst04 # clone the cookbook git clone https://github.com/ceph/ceph-cookbooks.git cd ceph-cookbooks # edit ~/.chef/knife-test2.rb (or knife.rb if you don't use knife-block) and add # cookbook_path ["/home/something/path/to/your/clone/ceph-cookbooks"] # currently the rgw cookbook forces us to have the apache2 cookbook available; do that # source: http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife knife cookbook site download apache2 tar xzf apache2-1.1.8.tar.gz knife cookbook upload apache2 # TODO cleanup the btrfs and parted cookbook remains knife cookbook upload btrfs ceph # let's create some roles (the cookbooks currently hardcode the "ceph-mon" role name, you must use that name for now) cat >ceph-mon.rb <<EOF name "ceph-mon" description "Ceph monitor server" run_list( 'recipe[ceph::single_mon]' ) EOF cat >ceph-osd.rb <<EOF name "ceph-osd" description "Ceph object store" run_list( 'recipe[ceph::bootstrap_osd]', ) EOF knife role from file ceph-mon.rb ceph-osd.rb # create a chef environment to contain the configuration, set the ceph branch to deploy (must be "chef-1" for now) knife environment create demo2 # "default_attributes": { # "ceph_branch": "chef-1" # }, knife node edit inst03 # "chef_environment": "demo2", # "run_list": [ # "recipe[ceph::apt]", # "role[ceph-mon]", # "role[ceph-osd]" # ] knife node edit inst04 # "chef_environment": "demo2", # "run_list": [ # "recipe[ceph::apt]", # "role[ceph-osd]" # ] # run chef-client a few rounds (first to get mon going, then to get the osd bootstrap files in place) ssh inst03 sudo chef-client ssh inst03 sudo chef-client ssh inst04 sudo chef-client # figure out what your ceph cluster uuid is; sorry, this is an eyeball operation right now ssh inst03 cat /var/lib/ceph/mon/ceph-single/log # prep a disk for osd use; this can be done anywhere, in advance, and a shelf of spare disks can be maintained at the data center # run on inst03, inst04 # create a gpt partition with the magic uuid; repeat for /dev/vdb vdc etc sudo apt-get install gdisk sudo gdisk /dev/vdb GPT fdisk (gdisk) version 0.6.14 Partition table scan: MBR: not present BSD: not present APM: not present GPT: not present Creating new GPT entries. Command (? for help): o This option deletes all partitions and creates a new protective MBR. Proceed? (Y/N): y Command (? for help): n Partition number (1-128, default 1): First sector (34-16777182, default = 34) or {+-}size{KMGTP}: Information: Moved requested sector from 34 to 2048 in order to align on 2048-sector boundaries. Use 'l' on the experts' menu to adjust alignment Last sector (2048-16777182, default = 16777182) or {+-}size{KMGTP}: Current type is 'Linux/Windows data' Hex code or GUID (L to show codes, Enter = 0700): 4fbd7e29-9d25-41b8-afd0-062c0ceff05d Changed type of partition to 'Unknown' Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed, possibly destroying your data? (Y/N): y OK; writing new GUID partition table (GPT). he operation has completed successfully. # mkfs and allocate disk to cluster; any filesystem is ok; use YOUR uuid from mon sudo mkfs -t ext4 /dev/vdb1 sudo mount -o user_xattr /dev/vdb1 /mnt sudo ceph-disk-prepare --cluster-uuid=07535b46-8429-4efd-9b2d-6b1170752448 /mnt sudo umount /mnt # simulate hotplug event sudo udevadm trigger --subsystem-match=block --action=add # see it go! sudo initctl list | grep ceph -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
