On Thu, Apr 27, 2023 at 06:32:42PM +0800, Ziyang Zhang wrote:
> It is very important to test ublk crash handling since the userspace
> part is not reliable. Especially we should test removing device, killing
> ublk daemons and user recovery feature.
Indeed, good job!
>
> Add five new test for ublk to cover these cases.
>
> Signed-off-by: Ziyang Zhang <ZiyangZhang@xxxxxxxxxxxxxxxxx>
> ---
> tests/ublk/001 | 39 +++++++++++++++++++++++++++
> tests/ublk/001.out | 2 ++
> tests/ublk/002 | 53 +++++++++++++++++++++++++++++++++++++
> tests/ublk/002.out | 2 ++
> tests/ublk/003 | 43 ++++++++++++++++++++++++++++++
> tests/ublk/003.out | 2 ++
> tests/ublk/004 | 63 +++++++++++++++++++++++++++++++++++++++++++
> tests/ublk/004.out | 2 ++
> tests/ublk/005 | 66 ++++++++++++++++++++++++++++++++++++++++++++++
> tests/ublk/005.out | 2 ++
> 10 files changed, 274 insertions(+)
> create mode 100644 tests/ublk/001
> create mode 100644 tests/ublk/001.out
> create mode 100644 tests/ublk/002
> create mode 100644 tests/ublk/002.out
> create mode 100644 tests/ublk/003
> create mode 100644 tests/ublk/003.out
> create mode 100644 tests/ublk/004
> create mode 100644 tests/ublk/004.out
> create mode 100644 tests/ublk/005
> create mode 100644 tests/ublk/005.out
Please run 'make check' and fix the following warning:
tests/ublk/002:35:13: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/002:36:17: note: Double quote to prevent globbing and word splitting. [SC2086]
tests/ublk/002:41:23: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/002:44:3: note: Instead of 'let expr', prefer (( expr )) . [SC2219]
tests/ublk/003:35:13: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/003:36:17: note: Double quote to prevent globbing and word splitting. [SC2086]
tests/ublk/004:35:13: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/004:36:17: note: Double quote to prevent globbing and word splitting. [SC2086]
tests/ublk/004:41:23: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/004:44:3: note: Instead of 'let expr', prefer (( expr )) . [SC2219]
tests/ublk/004:51:19: note: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?. [SC2181]
tests/ublk/004:53:17: note: Instead of 'let expr', prefer (( expr )) . [SC2219]
tests/ublk/004:55:15: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/005:35:13: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/005:36:17: note: Double quote to prevent globbing and word splitting. [SC2086]
tests/ublk/005:41:23: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/005:44:3: note: Instead of 'let expr', prefer (( expr )) . [SC2219]
tests/ublk/005:51:19: note: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?. [SC2181]
tests/ublk/005:53:17: note: Instead of 'let expr', prefer (( expr )) . [SC2219]
tests/ublk/005:55:15: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/005:58:13: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
tests/ublk/005:59:17: note: Double quote to prevent globbing and word splitting. [SC2086]
>
> diff --git a/tests/ublk/001 b/tests/ublk/001
> new file mode 100644
> index 0000000..fe158ba
> --- /dev/null
> +++ b/tests/ublk/001
> @@ -0,0 +1,39 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-3.0+
> +# Copyright (C) 2023 Ziyang Zhang
> +#
> +# Test ublk by deleting ublk device while running fio
> +
> +. tests/block/rc
> +. common/ublk
> +
> +DESCRIPTION="test ublk deletion"
> +QUICK=1
> +
> +requires() {
> + _have_ublk
> +}
> +
> +test() {
> + local ublk_prog="src/miniublk"
> +
> + echo "Running ${TEST_NAME}"
> +
> + if ! _init_ublk; then
> + return 1
> + fi
> +
> + ${ublk_prog} add -t null -n 0 > "$FULL"
> + udevadm settle
> + if ! ${ublk_prog} list -n 0 >> "$FULL"; then
> + echo "fail to list dev"
> + fi
> +
> + _run_fio_rand_io --filename=/dev/ublkb0 --time_based --runtime=30 > /dev/null 2>&1 &
> +
> + ${ublk_prog} del -n 0 >> "$FULL"
Indent.
Also probably, add one small delay before deleting device, then we can
make sure there are enough in-flight IOs before starting to delete.
> +
> + _exit_ublk
> +
> + echo "Test complete"
> +}
> diff --git a/tests/ublk/001.out b/tests/ublk/001.out
> new file mode 100644
> index 0000000..0d070b3
> --- /dev/null
> +++ b/tests/ublk/001.out
> @@ -0,0 +1,2 @@
> +Running ublk/001
> +Test complete
> diff --git a/tests/ublk/002 b/tests/ublk/002
> new file mode 100644
> index 0000000..25cad13
> --- /dev/null
> +++ b/tests/ublk/002
> @@ -0,0 +1,53 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-3.0+
> +# Copyright (C) 2023 Ziyang Zhang
> +#
> +# Test ublk by killing ublk daemon while running fio
> +# Delete the device after it is dead
> +
> +. tests/block/rc
> +. common/ublk
> +
> +DESCRIPTION="test ublk crash(1)"
> +QUICK=1
> +
> +requires() {
> + _have_ublk
> +}
> +
> +test() {
> + local ublk_prog="src/miniublk"
> +
> + echo "Running ${TEST_NAME}"
> +
> + if ! _init_ublk; then
> + return 1
> + fi
> +
> + ${ublk_prog} add -t null -n 0 > "$FULL"
> + udevadm settle
> + if ! ${ublk_prog} list -n 0 >> "$FULL"; then
> + echo "fail to list dev"
> + fi
> +
> + _run_fio_rand_io --filename=/dev/ublkb0 --time_based --runtime=30 > /dev/null 2>&1 &
> +
> + pid=`${ublk_prog} list -n 0 | grep "pid" | awk '{print $7}'`
> + kill -9 $pid
> +
> + sleep 2
> + secs=0
> + while [ $secs -lt 10 ]; do
> + state=`${ublk_prog} list -n 0 | grep "state" | awk '{print $11}'`
It could be cleaner to add one function for retrieving ublk state, given
the func is needed in many tests.
> + [ "$state" == "DEAD" ] && break
> + sleep 1
> + let secs++
> + done
Indent.
> + [ "$state" != "DEAD" ] && echo "device isn't dead after killing queue daemon"
> +
> + ${ublk_prog} del -n 0 >> "$FULL"
> +
> + _exit_ublk
> +
> + echo "Test complete"
> +}
> diff --git a/tests/ublk/002.out b/tests/ublk/002.out
> new file mode 100644
> index 0000000..93039b7
> --- /dev/null
> +++ b/tests/ublk/002.out
> @@ -0,0 +1,2 @@
> +Running ublk/002
> +Test complete
> diff --git a/tests/ublk/003 b/tests/ublk/003
> new file mode 100644
> index 0000000..34bce74
> --- /dev/null
> +++ b/tests/ublk/003
> @@ -0,0 +1,43 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-3.0+
> +# Copyright (C) 2023 Ziyang Zhang
> +#
> +# Test ublk by killing ublk daemon while running fio
> +# Delete the device immediately
> +
> +. tests/block/rc
> +. common/ublk
> +
> +DESCRIPTION="test ublk crash(2)"
> +QUICK=1
> +
> +requires() {
> + _have_ublk
> +}
> +
> +test() {
> + local ublk_prog="src/miniublk"
> +
> + echo "Running ${TEST_NAME}"
> +
> + if ! _init_ublk; then
> + return 1
> + fi
> +
> + ${ublk_prog} add -t null -n 0 > "$FULL"
> + udevadm settle
> + if ! ${ublk_prog} list -n 0 >> "$FULL"; then
> + echo "fail to list dev"
> + fi
> +
> + _run_fio_rand_io --filename=/dev/ublkb0 --time_based --runtime=30 > /dev/null 2>&1 &
> +
> + pid=`${ublk_prog} list -n 0 | grep "pid" | awk '{print $7}'`
> + kill -9 $pid
> +
> + ${ublk_prog} del -n 0 >> "$FULL"
Indent.
> +
> + _exit_ublk
> +
> + echo "Test complete"
> +}
> diff --git a/tests/ublk/003.out b/tests/ublk/003.out
> new file mode 100644
> index 0000000..90a3bfa
> --- /dev/null
> +++ b/tests/ublk/003.out
> @@ -0,0 +1,2 @@
> +Running ublk/003
> +Test complete
> diff --git a/tests/ublk/004 b/tests/ublk/004
> new file mode 100644
> index 0000000..c5d0694
> --- /dev/null
> +++ b/tests/ublk/004
> @@ -0,0 +1,63 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-3.0+
> +# Copyright (C) 2023 Ziyang Zhang
> +#
> +# Test ublk user recovery by run fio with dev recovery:
> +# (1)kill all ubq_deamon, (2)recover with new ubq_daemon,
> +# (3)delete dev
> +
> +. tests/block/rc
> +. common/ublk
> +
> +DESCRIPTION="test ublk recovery(1)"
> +QUICK=1
> +
> +requires() {
> + _have_ublk
> +}
> +
> +test() {
> + local ublk_prog="src/miniublk"
> +
> + echo "Running ${TEST_NAME}"
> +
> + if ! _init_ublk; then
> + return 1
> + fi
> +
> + ${ublk_prog} add -t null -n 0 -r 1 > "$FULL"
> + udevadm settle
> + if ! ${ublk_prog} list -n 0 >> "$FULL"; then
> + echo "fail to list dev"
> + fi
> +
> + _run_fio_rand_io --filename=/dev/ublkb0 --time_based --runtime=30 > /dev/null 2>&1 &
> + pid=`${ublk_prog} list -n 0 | grep "pid" | awk '{print $7}'`
> + kill -9 $pid
> +
> + sleep 2
> + secs=0
> + while [ $secs -lt 10 ]; do
> + state=`${ublk_prog} list -n 0 | grep "state" | awk '{print $11}'`
> + [ "$state" == "QUIESCED" ] && break
> + sleep 1
> + let secs++
> + done
Indent.
> + [ "$state" != "QUIESCED" ] && echo "device isn't quiesced after killing queue daemon"
> +
> + secs=0
> + while [ $secs -lt 10 ]; do
> + ${ublk_prog} recover -t null -n 0 >> "$FULL"
> + [ $? -eq 0 ] && break
> + sleep 1
> + let secs++
> + done
Do we need to send multiple recover commands?
> + state=`${ublk_prog} list -n 0 | grep "state" | awk '{print $11}'`
> + [ "$state" == "QUIESCED" ] && echo "failed to recover dev"
> +
> + ${ublk_prog} del -n 0 >> "$FULL"
> +
> + _exit_ublk
> +
> + echo "Test complete"
> +}
> diff --git a/tests/ublk/004.out b/tests/ublk/004.out
> new file mode 100644
> index 0000000..a92cd50
> --- /dev/null
> +++ b/tests/ublk/004.out
> @@ -0,0 +1,2 @@
> +Running ublk/004
> +Test complete
> diff --git a/tests/ublk/005 b/tests/ublk/005
> new file mode 100644
> index 0000000..23c0555
> --- /dev/null
> +++ b/tests/ublk/005
> @@ -0,0 +1,66 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-3.0+
> +# Copyright (C) 2023 Ziyang Zhang
> +#
> +# Test ublk user recovery by run fio with dev recovery:
> +# (1)kill all ubq_deamon, (2)recover with new ubq_daemon,
> +# (3)kill all ubq_deamon, (4)delete dev
> +
> +. tests/block/rc
> +. common/ublk
> +
> +DESCRIPTION="test ublk recovery(2)"
> +QUICK=1
> +
> +requires() {
> + _have_ublk
> +}
> +
> +test() {
> + local ublk_prog="src/miniublk"
> +
> + echo "Running ${TEST_NAME}"
> +
> + if ! _init_ublk; then
> + return 1
> + fi
> +
> + ${ublk_prog} add -t null -n 0 -r 1 > "$FULL"
> + udevadm settle
> + if ! ${ublk_prog} list -n 0 >> "$FULL"; then
> + echo "fail to list dev"
> + fi
> +
> + _run_fio_rand_io --filename=/dev/ublkb0 --time_based --runtime=30 > /dev/null 2>&1 &
> + pid=`${ublk_prog} list -n 0 | grep "pid" | awk '{print $7}'`
> + kill -9 $pid
> +
> + sleep 2
> + secs=0
> + while [ $secs -lt 10 ]; do
> + state=`${ublk_prog} list -n 0 | grep "state" | awk '{print $11}'`
> + [ "$state" == "QUIESCED" ] && break
> + sleep 1
> + let secs++
> + done
> + [ "$state" != "QUIESCED" ] && echo "device isn't quiesced after killing queue daemon"
Indent.
> +
> + secs=0
> + while [ $secs -lt 10 ]; do
> + ${ublk_prog} recover -t null -n 0 >> "$FULL"
> + [ $? -eq 0 ] && break
> + sleep 1
> + let secs++
> + done
Same question with above.
Thanks,
Ming
