Re: Gluster testing matrix

[M S Vishwanath Bhat <msvbhat@xxxxxxxxx>]

More subcategories to add as suggested:

1. RDMA transport ptotocol

2. Openstatck-Manila integration

3. IPv6 support

New test categories which was missed in the last mail

1. Static Code analysis (Coverity/SonarQube)

2. Doc checks (Most likely be manual - All new feature or updates should have corresponding documentation)

Exit criterion for 

a) each patch -

   1. Smoke tests upon submission

   2. Functional/regression after review and before merging

b) Nightly builds

   1. Regression + Functional 

   2. Static Code Analysis

   3. performance regression

   4. Basic distro packaging

c) Minor releases (.x releases)

   1. Regression + Functional

   2. performance regression

   3. Full distro packaging sanity tests

   4. Longevity*

   5. Integration**

   6. Doc checks

   7. Upgrade testing

 

d) Major releases 

   1. Regression + Functional

   2. performance regression

   3. Full distro packaging sanity tests

   4. Longevity*

   5. Integration**

   6. Doc checks

   7. Upgrade testing

* We are not sure about longevity in minor releases considering the one month release schedule, and longevity tests has to run for at least 1-2 weeks.

** Currently integration tests are owned and maintained by the other projects (like swift/openstack-manila). Should we setup our own integration tests? Not sure how to qualify integration tests before releasing.

If we have missed anything or If you have any suggestions, please let us know.

Best Regards,

msvbhat & rtalur

On 25 November 2015 at 17:39, Raghavendra Talur <rtalur@xxxxxxxxxx> wrote:

Hi All,

Here is a table representing the current state of Gluster testing.

* Things in green are categories for which we have some kind of testing in place.

* Things in red are the ones which don't have any tests.

* Things in yellow are the ones which have no known tests or are not managed by Gluster community.

Test Category/Sub-Category
smoke	source build + Posix complaince + Dbench
functional	tests/basic
regression	tests/bugs
performance regression	N/A
integration	Backends/FS	Protocols	Consumers/Cloud Environments	Tools	libgfapi bindings	OS environment
	xfs	smb	qemu	gdeploy	java	firewalld
	ext4	nfs	openstack/cinder	heketi	python	ufw
	btrfs	swift	openshift/docker/containers		ruby	selinux
	zfs		aws		go	apparmor
			azure
			hadoop
update	major version upgrades	minor version upgrades
longevity	a. memory leaks b. log accumulation
distro packaging	a. pkg build + smoke b. sysvinit/systemd

I will send separate mails for each of the categories above highlighting plans for them.

Use this thread to indicate addition/deletion/changes to this matrix.

We will put this at gluster.org website once it is final.

Thanks,

Raghavendra Talur & MSV Bhat

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel