Dear Jens Axboe and Steve Daniel (and the FIO mailing list),
My name is Mike O'Sullivan, a senior lecturer at the University of
Auckland, New Zealand.
I have been working on integrating the open source SPC-1 benchmark from
Steve Daniel into version 1.36 of fio. I now have this working, but the
best way to describe it is a "bit of a hack". Currently all the SPC-1
options are defined at the top of the spc1_wrapper.h file. Thus if I
want to run the SPC-1 benchmark for a different length of time or on
different devices I need to recompile. Also, the way the code works
right now is to use the open source SPC-1 code to generate all the
necessary IO, save it into memory (quite inefficiently at the moment,
just using big arrays) and then "emulate" an iolog file to get fio to
run the IO. It works by setting the right options in spc1_wrapper.h,
recompiling and running fio --spc. Currently, on my laptop and servers I
can generate IO from the SPC-1 benchmark for at least an hour, much
longer and I run into memory issues storing the SPC-1 IO. There are
options to use a single fio process or multiple processes/threads (one
for each BSU in the SPC-1 benchmark).
I'm emailing because I would like to improve this code, but I'm not sure
of the best way to proceed. Some immediate options that spring to mind are:
1) I think the command should be fio --spc <spc_param_file>, but I'm not
sure how to best implement this in fio or what the format of the param
file should be. Then the options could be removed from the top of
spc1_wrapper.h and put into a param file, so not more recompiling. This
would involve changes to the spc1_wrapper files, but I think it should
be straightforward. However, I struggle with the options in fio and
don't have a good idea as to the best format to use for the parameter file;
2) use dynamic storage (something like std::vector from C++) to store
the SPC-1 IO, thus only as much as needed is created and once it is used
it can be released;
3) remove the iolog emulation, so SPC-1 IO is automatically queued in
fio (I think there may be an issue with knowing the maximum block size
before starting IO...?). Initially I was generating iolog files from the
SPC-1 code, but this caused problems with too many files.
I have attached a patch showing (I hope, git newbie) the differences
between my code and the main fio-1.36 code as well as the SPC-1 wrapper
code. I have also attached my version of the SPC-1 open source code,
modified (very slightly if I remember correctly) from Steve Daniel's code.
Any help, comments, suggestions welcome.
Kind regards, Mike O'Sullivan
/*
Copyright 2005-2009 NetApp, Incorporated. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY NETAPP, INCORPORATED
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN
NO EVENT SHALL NETAPP, INC. BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/*
* Revision History
*
* Version 1.2
* August 28, 2009. Fixed an overflow bug for large ASUs
* Version 1.1
* June 5, 2005. Added multiple state blocks.
* Version 1.0
* May 1, 2005. First public version.
*
* $Id$
*/
/*
* This file defines an SPC-1 I/O operation.
*/
struct spc1_io_s {
unsigned int asu:2; /* which ASU? */
unsigned int dir:1; /* read=0, write=1 */
unsigned int len:7; /* length of transfer in units of 4KB */
unsigned int stream:3; /* which stream in the bsu? */
unsigned int bsu:16; /* which bsu? */
unsigned int pos; /* in units of 4KB */
unsigned int when; /* when to do this I/O in units of 0.1 milliseconds */
};
/*
* Error codes
*/
enum {
SPC1_ENOERR = 0, /* Success */
SPC1_ENOMEM = -1, /* Memory allocation failed */
SPC1_ESTYLE = -2, /* Illegal HRRW style */
SPC1_EHRRW = -3, /* Internal HRRW error */
SPC1_EASU = -4, /* Internal ASU error */
};
/*
* Generates the next operation.
*
* Parameters:
* context Which context block to use. Must
* be in the range [0, n_contexts-1]
*
*
* Returns:
* no error or one of the errors above.
* These errors "should never happen".
*
* Each I/O request generated will be requested either
* at the same time as the previous request in this context
* or at a later time. Within a context time never runs
* backwards.
*
* Running time is O(log(b))
*
* This routine is thread safe iff concurrent calls
* use different values of context.
*
*/
int spc1_next_op(struct spc1_io_s *, int context);
/*
* Generate the next operation in any context.
*
* Finds the next spc1 operation in any context.
* Linear in the number of contexts.
* Most assuredly *not* thread safe.
*/
int spc1_next_op_any(struct spc1_io_s *);
/*
* Initialize the SPC-1 I/O geneator.
*
* Parameters:
* m Name of the program. Used for error messages.
* b Number of BSUs.
* a1 Size of ASU 1 in 4K blocks.
* a2 Size of ASU 2 in 4K blocks.
* a3 Size of ASU 3 in 4K blocks.
* n_contexts
* The number of context blocks to allocate
* version An output buffer where a version string may
* be written. If NULL, no version is written.
* len The length of the output buffer.
*
* The only possible errors are internal programming errors or
* a failure to allocate enough memory. Memory requirement is
* O(b) and quite modest.
*
* No checks are made to ensure the ASUs are the right size
* relative to each other.
*
* No check is made if the ASUs are too small. The minimum
* size of ASU1 or ASU2 is 20 * 2**6 * 8 (= 10240) 4KB blocks
* or about 40 MB. Running with ASUs smaller than this
* will produce non-conforming output.
*
* Version 1 of the SPC-1 benchmark uses multiple java virtual
* machines (JVMs) to improve the benchmark's ability to scale to large
* workloads. Because the JVMs do not share state with each other
* each JVM is a distinct context. This workload generator
* has the ability to use multiple contexts as well.
* An implementation wishing close conformance with the specification
* should set n_contexts=1. An implementation wishing close
* conformance with version on of the SPC workload generator
* should set n_contexts = (b+99)/100.
*
*
* This routine is not thread safe.
*/
int spc1_init(char *m,
int b,
unsigned int a1,
unsigned int a2,
unsigned int a3,
int n_contexts,
char *version, int len);
# source files.
SRC = spc1.c
OBJ = $(SRC:.c=.o)
OUT = libspc1.a
# include directories
INCLUDES = -I.
# C++ compiler flags (-g -O2 -Wall)
CCFLAGS = -g
# compiler
CCC = gcc
.SUFFIXES: .c
default: $(OUT)
.c.o:
$(CCC) $(INCLUDES) $(CCFLAGS) -c $< -o $@
$(OUT): $(OBJ)
ar rcs $(OUT) $(OBJ)
clean:
rm -f $(OBJ) $(OUT)
/*
Copyright 2005-2009 NetApp, Incorporated. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY NETAPP, INCORPORATED
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN
NO EVENT SHALL NETAPP, INC. BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/*
* Implement the SPC-1 workload.
* Entry points are defined in spc1.h
*/
/*
* Revision History
*
* Version 1.4.
* August 28, 2009. Fixed an overflow bug for large ASUs
* Version 1.3.
* July 12, 2005. Better support for integer-only math.
* Version 1.2
* June 24, 2005. Better support for embedded systems.
* Version 1.1
* June 5, 2005. Added multiple state blocks.
* Version 1.0
* May 1, 2005. First public version.
*/
static char *Version = "V1.4: $Id$";
#define _XOPEN_SOURCE
#define _ISOC99_SOURCE
#define _XPG5
#ifdef _ONTAP_
#include "spc1_kernel.h"
#else
#define VALIDATE
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#endif
#include "spc1.h"
/*
* Parameters.
*/
#ifdef VALIDATE
static char *myname;
#endif
static int asu1_size;
static int asu2_size;
static int asu3_size;
static int asu1_mult;
static int asu2_mult;
static int asu3_mult;
#define IOPS_PER_BSU 50.
#ifndef TIME_UNITS_PER_SECOND
#define TIME_UNITS_PER_SECOND 10000
#endif
#ifndef SPC1_USE_INTEGER_MATH
#define SPC1_USE_INTEGER_MATH 0
#endif
#if SPC1_USE_INTEGER_MATH == 1
#define PERCENT(x,p) ((int)(((unsigned long)(x)*(p))/100))
/*
* These constants represent the stream intensities.
* The are scaled by a factor of 5000.= 100 * IOPS per BSU
*/
#define IN018 90
#define IN035 175
#define IN070 350
#define IN210 1050
#define IN281 1405
#else
#define PERCENT(x,p) ((int)((double)(x)*((p)/100.0)))
#define IN018 0.018
#define IN035 0.035
#define IN070 0.070
#define IN210 0.210
#define IN281 0.281
#endif
/*
* Some discussion of HRRW implemetations:
* CLASSIC is what is implemented in the version 1 SPC workload generator.
*
* V2 enlarges the tree and loops for out-of-bounds requests.
* The version two generator will do this, but with a bound of 100
* loops.
*
* FIXED uses a smaller tree, like classic, but fixes the data
* sharing model to do the right thing.
*/
#define HRRW_CLASSIC 1 /* As implemented */
#define HRRW_FIXED 2 /* Simple Fix */
#define HRRW_V2 3 /* As Proposed */
#define HRRW_BLOCKS_PER_LEAF 8
#define HRRW_V2_RETRY 100
//static int hrrw_style = HRRW_CLASSIC;
static int hrrw_style = HRRW_V2;
static int
rnd(int n);
/*
* An I/O Stream
*/
struct io_state_s {
unsigned int i_stream_id;
unsigned int i_next_time;
unsigned char i_op;
unsigned char i_len; /* units are 4K */
unsigned int i_block_addr; /* units are 4K */
unsigned int i_end_addr;
// for classic and fixed, this is the offset, in blocks,
// between the start of teh hot spot and the start of
// the hrrw tree. For v2, this is zero.
unsigned int i_hrrw_offset;
char i_rewrite; /* boolean */
unsigned int i_rewrite_block;
};
/*
* Stuff for the HRRW
*/
struct hrrw_s {
unsigned int h_min_block; // the start of the region
// for classic and fixed, size of the tree in blocks.
// for v2, size of the region in blocks.
unsigned int h_tree_size;
unsigned int h_n_levels;
unsigned char *h_leaf_state;
// for classic and fixed, delta is the difference,
// in blocks, between the size of the hot spot
// and the size of the hrrw tree..
// For v2, 0.
unsigned int h_delta;
};
/*
* Context blocks. (Eumlating multiple JVMs.
*/
static int n_state_blocks;
struct state_block_s {
int stream_count;
int bsu_count;
struct io_state_s *io_heap;
struct hrrw_s hrrw1, hrrw2, hrrw3;
};
static struct state_block_s *states;
/* legal stream ids: */
#define ASU1_1 0
#define ASU1_2 1
#define ASU1_3 2
#define ASU1_4 3
#define ASU2_1 4
#define ASU2_2 5
#define ASU2_3 6
#define ASU3_1 7
#define BSU_STREAMS 8
int stream_id_to_asu(int stream_id) {
unsigned index = stream_id % BSU_STREAMS, asu;
switch (index) {
case ASU1_1: case ASU1_2: case ASU1_3: case ASU1_4:
asu = 1;
break;
case ASU2_1: case ASU2_2: case ASU2_3:
asu = 2;
break;
case ASU3_1:
asu = 3;
break;
default:
asu = SPC1_EASU;
}
return asu;
}
int stream_id_to_bsu(int stream_id) {
// printf("In stream_id_to_bsu: stream_id = %d, bsu = %d\n", stream_id, stream_id / BSU_STREAMS);
return stream_id / BSU_STREAMS;
}
/* legal operations */
#define OP_READ 0
#define OP_WRITE 1
static int
hrrw_init(struct hrrw_s *hp, int pos, int size)
{
int i;
unsigned int n_leaves;
unsigned int array_size;
hp->h_min_block = pos;
n_leaves = size / HRRW_BLOCKS_PER_LEAF;
array_size = n_leaves * 2;
hp->h_leaf_state = (unsigned char *)malloc(array_size);
if (hp->h_leaf_state == (unsigned char *)0) {
return SPC1_ENOMEM;
}
for (i = 0; i < array_size; i++)
hp->h_leaf_state[i] = 0;
hp->h_n_levels = 0;
for (i = 1; i < n_leaves; i <<= 1)
hp->h_n_levels++;
/*
* At this point, i is the smallest power of 2
* equal to or larger than the HRRW region.
*/
switch (hrrw_style) {
case HRRW_CLASSIC:
case HRRW_FIXED:
if (i > n_leaves) {
hp->h_n_levels--;
i >>= 1;
}
/* Now i is largest equal to or smaller than */
hp->h_delta = HRRW_BLOCKS_PER_LEAF * (n_leaves - i);
hp->h_tree_size = size - hp->h_delta;
break;
case HRRW_V2:
hp->h_delta = 0;
hp->h_tree_size = size;
break;
default:
return SPC1_ESTYLE;
}
return SPC1_ENOERR;
}
static int
hrrw_per_stream(struct state_block_s *sp, struct hrrw_s *hp)
{
int r;
switch (hrrw_style) {
case HRRW_CLASSIC:
sp->io_heap->i_hrrw_offset =
hp->h_min_block + rnd(hp->h_delta + 1);
sp->io_heap->i_hrrw_offset &= ~0x07; /* 32KB (leaf) boundary */
break;
case HRRW_FIXED:
r = rnd(hp->h_delta + 1);
sp->io_heap->i_hrrw_offset = hp->h_min_block +
r - (r % HRRW_BLOCKS_PER_LEAF);
break;
case HRRW_V2:
sp->io_heap->i_hrrw_offset = hp->h_min_block;
break;
default:
return SPC1_ESTYLE;
}
sp->io_heap->i_block_addr = sp->io_heap->i_hrrw_offset +
rnd(hp->h_tree_size);
sp->io_heap->i_rewrite = 0;
return SPC1_ENOERR;
}
static int
init(struct state_block_s *sp, int bsu_count)
{
int i;
struct io_state_s *ip;
int retcode;
sp->bsu_count = bsu_count;
sp->stream_count = sp->bsu_count * BSU_STREAMS;
sp->io_heap = (struct io_state_s *)malloc(sp->stream_count *
(sizeof (struct io_state_s)));
if (sp->io_heap == (struct io_state_s *)0) {
return SPC1_ENOMEM;
}
ip = sp->io_heap;
for (i = 0; i < sp->stream_count; i++) {
ip->i_next_time = 0; /* flag value, means not init */
ip++;
}
ip = sp->io_heap;
for (i = 0; i < sp->bsu_count; i++) {
ip->i_stream_id = i * BSU_STREAMS + ASU1_1;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU1_2;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU1_3;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU1_4;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU2_1;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU2_2;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU2_3;
ip++;
ip->i_stream_id = i * BSU_STREAMS + ASU3_1;
ip++;
}
retcode = hrrw_init(&(sp->hrrw1), PERCENT(asu1_size,15), asu1_size/20);
if (retcode)
return retcode;
retcode = hrrw_init(&(sp->hrrw2), PERCENT(asu1_size,70), asu1_size/20);
if (retcode)
return retcode;
retcode = hrrw_init(&(sp->hrrw3), PERCENT(asu2_size,47), asu2_size/20);
if (retcode)
return retcode;
return SPC1_ENOERR;
}
/*
* Call requeue after updating the time on the
* stream pointed to by io_heap.
*
* This is the guts of the heap algorithm.
*
* The array represents a binary tree.
* The tree has the property that tree[i] has children at
* tree[2i+1] and [2i+2].
*
*/
static void
requeue_i(struct state_block_s *sp, int n)
{
struct io_state_s it;
struct io_state_s *root, *left, *right;
root = sp->io_heap + n;
left = sp->io_heap + (2 * n + 1);
right = left + 1;
if (2 * n + 1 >= sp->stream_count)
left = (struct io_state_s *)0;
if (2 * n + 2 >= sp->stream_count)
right = (struct io_state_s *)0;
/* If we are already a head, return */
if ((!left || root->i_next_time <= left->i_next_time) &&
(!right || root->i_next_time <= right->i_next_time))
return;
it = *root;
if (!right || left->i_next_time <= right->i_next_time) {
/* use left tree */
*root = *left;
*left = it;
requeue_i(sp, 2 * n + 1);
} else {
/* use right tree */
*root = *right;
*right = it;
requeue_i(sp, 2 * n + 2);
}
}
static void
requeue(struct state_block_s *sp)
{
requeue_i(sp, 0);
}
/*
* These routines generate the next I/O request.
*/
static int
rnd(int n)
{
int check;
#ifdef VALIDATE
if (n <= 1) {
fprintf(stderr, "%s: INTERNAL ERROR rnd(%d)\n",
myname, n);
exit(1);
}
#endif
return lrand48() % n;
}
static int smix_lengths[] =
{ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2,
4, 4, 4, 4, 4,
8, 8,
16, 16,
};
static int
smix(void)
{
return smix_lengths[rnd(sizeof smix_lengths / sizeof (int))];
}
#if SPC1_USE_INTEGER_MATH == 1
#define RAND ((unsigned)lrand48()*2)
#define T24 16777216
#define T23 8388608
#define B 346574 // 50 * BF * -ln(.5)
#define BF 10000 // factor for time unit correction of B
#define AN 7 // number of coefficients in our expansion
#if AN == 5
long long c[AN] = { // chebyshev coeffcients scaled by 2^24
-29137993,
47093318,
-24414117,
7390409,
-930452
};
#endif
#if AN == 7
long long c[AN] = { // chebyshev coeffcients scaled by 2^24
-35289639,
70819884,
-61653999,
37788306,
-14512655,
3140287,
-292159
};
#endif
/*
* v is any non-zero unsigned integer.
*
* returns 50 * BF * log(v/(2**32)) as a signed 64 bit integer
*/
static long long
ilog(unsigned int v)
{
int i;
int k;
long long acc;
long long vv;
if (v == 0)
return -100; /* punt */
k = 0;
while (v > T24) {
v >>= 1;
k--;
}
while (v <= T23) {
v <<= 1;
k++;
}
vv = (long long)v;
acc = c[0] * T23;
for (i = 1; i < AN; i++) {
acc += c[i] * vv;
vv = ((vv * (long long)v) >> 23);
}
acc >>= 8; /* get rid of some fractional part
* before the multiply by BF */
acc = (acc * (long long)(50 * BF)) >> (23+24-8);
acc -= (k+9) * B;
return acc;
}
/*
* Call tnext with one of the intensities and get
* back the number of milliseconds to the next I/O request
*
* Assumes there exists a RAND function that returns
* an unsigned 32-bit random integer.
*/
static int
tnext(int intensity)
{
unsigned int v;
int factor = BF / TIME_UNITS_PER_SECOND;
int result;
do {
v = RAND;
} while (v == 0);
/* When compared with the floating point version at 10000 and
* 1000 time units per second, result will be identical about
* 50% of the time, off by negative one about 25% of the time,
* and off by positive one about 25% of the time. (Tested with
* 5m random numbers and an observed variation of about 1%.) */
if (factor == 1) {
result = (-ilog(v) + intensity/2) / intensity;
result *= 2;
} else {
result = -ilog(v) / intensity;
result = (result + factor/2) / factor;
result *= 2;
}
#ifdef VALIDATE
if (result < 0)
fprintf(stderr, "%s: INTERNAL ERROR tnext(%d) returning %d\n",
myname, intensity, result);
#endif
return result;
}
#else /* SPC1_USE_INTEGER_MATH (floating point version follows) */
static double
exponential(double average)
{
double d;
d = drand48();
if (d < 1.0E-8)
d = 1.0E-8;
return -log(d) * average;
}
/* how long till the next op at the stated intensity? */
static int
tnext(double intensity)
{
double d;
d = exponential(1. / (IOPS_PER_BSU * intensity));
return (int) (d * (double)TIME_UNITS_PER_SECOND + .5);
}
#endif /* SPC1_USE_INTEGER_MATH */
static int
asu1_1(struct state_block_s *sp)
{
sp->io_heap->i_next_time += tnext(IN035);
sp->io_heap->i_op = (rnd(10) < 5)? OP_READ: OP_WRITE;
sp->io_heap->i_len = 1;
sp->io_heap->i_block_addr = rnd(asu1_size);
return SPC1_ENOERR;
}
static int
asu1_2(struct state_block_s *sp)
{
struct hrrw_s *hp = &(sp->hrrw1);
if (sp->io_heap->i_next_time == 0) {
int retcode = hrrw_per_stream(sp, &(sp->hrrw1));
if (retcode)
return retcode;
}
sp->io_heap->i_next_time += tnext(IN281);
sp->io_heap->i_op = (rnd(10) < 5)? OP_READ: OP_WRITE;
sp->io_heap->i_len = 1;
/* the hrrw stuff is computed at launch time, not now */
return SPC1_ENOERR;
}
static int
asu1_3(struct state_block_s *sp)
{
sp->io_heap->i_op = OP_READ;
sp->io_heap->i_block_addr += sp->io_heap->i_len;
sp->io_heap->i_len = smix();
if (sp->io_heap->i_next_time == 0 ||
sp->io_heap->i_block_addr + sp->io_heap->i_len >= sp->io_heap->i_end_addr) {
/* must initialize the seq read stream */
sp->io_heap->i_block_addr =
PERCENT(asu1_size, 20) + rnd(PERCENT(asu1_size, 40));
sp->io_heap->i_end_addr =
sp->io_heap->i_block_addr + PERCENT(asu1_size, 10);
if (hrrw_style == HRRW_CLASSIC) { /* 64KB boundary */
sp->io_heap->i_block_addr &= ~0x0f;
sp->io_heap->i_end_addr &= ~0x0f;
}
}
sp->io_heap->i_next_time += tnext(IN070);
return SPC1_ENOERR;
}
static int
asu1_4(struct state_block_s *sp)
{
if (sp->io_heap->i_next_time == 0) {
int retcode = hrrw_per_stream(sp, &(sp->hrrw2));
if (retcode)
return retcode;
}
sp->io_heap->i_next_time += tnext(IN210);
sp->io_heap->i_op = (rnd(10) < 5)? OP_READ: OP_WRITE;
sp->io_heap->i_len = 1;
/* the hrrw stuff is computed at launch time, not now */
return SPC1_ENOERR;
}
static int
asu2_1(struct state_block_s *sp)
{
sp->io_heap->i_next_time += tnext(IN018);
sp->io_heap->i_op = (rnd(10) < 3)? OP_READ: OP_WRITE;
sp->io_heap->i_len = 1;
if (hrrw_style == HRRW_CLASSIC) /* XXX FIXME do this always? */
sp->io_heap->i_block_addr = rnd(asu2_size);
else
sp->io_heap->i_block_addr = (rnd(asu2_size) / 2) * 2;
return SPC1_ENOERR;
}
static int
asu2_2(struct state_block_s *sp)
{
if (sp->io_heap->i_next_time == 0) {
int retcode = hrrw_per_stream(sp, &(sp->hrrw3));
if (retcode)
return retcode;
}
sp->io_heap->i_next_time += tnext(IN070);
sp->io_heap->i_op = (rnd(10) < 3)? OP_READ: OP_WRITE;
sp->io_heap->i_len = 1;
/* the hrrw stuff is computed at launch time, not now */
return SPC1_ENOERR;
}
static int
asu2_3(struct state_block_s *sp)
{
sp->io_heap->i_op = OP_READ;
sp->io_heap->i_block_addr += sp->io_heap->i_len;
sp->io_heap->i_len = smix();
if (sp->io_heap->i_next_time == 0 ||
sp->io_heap->i_block_addr + sp->io_heap->i_len >= sp->io_heap->i_end_addr) {
/* must initialize the seq read stream */
sp->io_heap->i_block_addr =
PERCENT(asu2_size, 20) + rnd(PERCENT(asu2_size, 40));
sp->io_heap->i_end_addr =
sp->io_heap->i_block_addr + PERCENT(asu2_size, 10);
if (hrrw_style == HRRW_CLASSIC) { /* 64KB boundary */
sp->io_heap->i_block_addr &= ~0x0f;
sp->io_heap->i_end_addr &= ~0x0f;
}
}
sp->io_heap->i_next_time += tnext(IN035);
return SPC1_ENOERR;
}
static int
asu3_1(struct state_block_s *sp)
{
sp->io_heap->i_op = OP_WRITE;
sp->io_heap->i_block_addr += sp->io_heap->i_len;
sp->io_heap->i_len = smix();
if (sp->io_heap->i_next_time == 0 ||
sp->io_heap->i_block_addr + sp->io_heap->i_len >= sp->io_heap->i_end_addr) {
/* must initialize the seq write stream */
sp->io_heap->i_block_addr = rnd(PERCENT(asu3_size, 70));
sp->io_heap->i_end_addr =
sp->io_heap->i_block_addr + PERCENT(asu3_size, 30);
if (hrrw_style == HRRW_CLASSIC) { /* 64KB boundary */
sp->io_heap->i_block_addr &= ~0x0f;
sp->io_heap->i_end_addr &= ~0x0f;
}
}
sp->io_heap->i_next_time += tnext(IN281);
return SPC1_ENOERR;
}
/*
* Implement the actual hierarchical resuse random walk!
*/
static int
hrrw(struct hrrw_s *hp, struct io_state_s *ip)
{
unsigned int h, th;
unsigned int old_leaf;
unsigned int new_leaf;
unsigned int block;
int retry_count;
/* if mode is write and we need to repeat, do so */
if (ip->i_op == OP_WRITE && ip->i_rewrite) {
ip->i_rewrite = 0;
ip->i_block_addr = ip->i_rewrite_block;
return SPC1_ENOERR;
}
if (ip->i_block_addr < ip->i_hrrw_offset) {
return SPC1_EHRRW;
}
old_leaf = (ip->i_block_addr - ip->i_hrrw_offset) /
HRRW_BLOCKS_PER_LEAF;
retry_count = HRRW_V2_RETRY;
again:
h = 6;
th = 64; // 2 ** h
if (hrrw_style == HRRW_CLASSIC) { /* k=7 */
++h;
th *= 2;
}
while (h < hp->h_n_levels && rnd(100) < 44) {
h++;
th *= 2;
}
new_leaf = th * (old_leaf / th) + rnd(th);
if (ip->i_op == OP_WRITE)
new_leaf -= new_leaf % 8;
/*
* At this point, new_leaf is a position in the binary
* tree. Now we need to convert it into an index
* into the leaf state array.
*/
switch (hrrw_style) {
case HRRW_CLASSIC:
// no conversion necessary
break;
case HRRW_FIXED:
new_leaf += (ip->i_hrrw_offset - hp->h_min_block) /
HRRW_BLOCKS_PER_LEAF;
break;
case HRRW_V2:
// truncate the distribution
if (new_leaf > hp->h_tree_size / HRRW_BLOCKS_PER_LEAF) {
old_leaf = new_leaf;
if (retry_count -- > 0)
goto again;
new_leaf = rnd(hp->h_tree_size / HRRW_BLOCKS_PER_LEAF);
}
break;
}
if (ip->i_op == OP_READ) {
// on read, cycle through the 8 blocks
block = hp->h_leaf_state[new_leaf];
hp->h_leaf_state[new_leaf] = ((block + 1) %
HRRW_BLOCKS_PER_LEAF);
} else {
// op is write
if (rnd(100) < 50)
// 50% of the time pick a random block
block = rnd(HRRW_BLOCKS_PER_LEAF);
else {
// 50% of the time use the last read block
// (*not* the next read block)
block = hp->h_leaf_state[new_leaf];
if (hrrw_style != HRRW_CLASSIC) {
if (block == 0)
block = HRRW_BLOCKS_PER_LEAF;
block--;
}
}
if (rnd(100) < 15)
// 15% of the time do two writes
ip->i_rewrite = 1;
}
if (hrrw_style == HRRW_FIXED)
block += new_leaf * HRRW_BLOCKS_PER_LEAF + hp->h_min_block;
else
block += new_leaf * HRRW_BLOCKS_PER_LEAF + ip->i_hrrw_offset;
if (ip->i_op == OP_WRITE)
ip->i_rewrite_block = block;
ip->i_block_addr = block;
return SPC1_ENOERR;
}
/*
* Generate one I/O
*/
static int
gen_io_i(struct spc1_io_s *spc1_io, struct state_block_s *sp)
{
#ifdef VALIDATE
int ignore;
int s;
#endif
int retcode = SPC1_ENOERR;
memset(spc1_io, 0, sizeof(*spc1_io));
/*
* Skip the I/O if we are not initialized.
*/
#ifdef VALIDATE
again:
ignore = 0;
#endif
if (sp->io_heap->i_next_time) {
/*
* For HRRW, take the HRRW step now
*/
switch(sp->io_heap->i_stream_id) {
case ASU1_2:
retcode = hrrw(&(sp->hrrw1), sp->io_heap);
break;
case ASU1_4:
retcode =hrrw(&(sp->hrrw2), sp->io_heap);
break;
case ASU2_2:
retcode = hrrw(&(sp->hrrw3), sp->io_heap);
break;
default:
break;
}
if (retcode)
return retcode;
#ifdef VALIDATE
switch(stream_id_to_asu(sp->io_heap->i_stream_id)) {
case 3:
s = asu3_size;
break;
case 2:
s = asu2_size;
break;
case 1:
s = asu1_size;
break;
default:
fprintf(stderr, "%s: VALIDATE ERROR asu=%d\n",
myname,
stream_id_to_asu(sp->io_heap->i_stream_id) );
exit(1);
}
ignore = 0;
if (s < sp->io_heap->i_len + sp->io_heap->i_block_addr) {
fprintf(stderr, "%s: VALIDATE ERROR asu=%d\n",
myname,
stream_id_to_asu(sp->io_heap->i_stream_id) );
fprintf(stderr, "\tstream = %d\n", sp->io_heap->i_stream_id);
fprintf(stderr, "\tsize = %d\n", s);
fprintf(stderr, "\tlen = %d\n", sp->io_heap->i_len);
fprintf(stderr, "\tpos = %d\n", sp->io_heap->i_block_addr);
ignore = 1;
}
#endif
}
switch(sp->io_heap->i_stream_id % BSU_STREAMS) {
case ASU1_1:
retcode = asu1_1(sp);
break;
case ASU1_2:
retcode = asu1_2(sp);
break;
case ASU1_3:
retcode = asu1_3(sp);
break;
case ASU1_4:
retcode = asu1_4(sp);
break;
case ASU2_1:
retcode = asu2_1(sp);
break;
case ASU2_2:
retcode = asu2_2(sp);
break;
case ASU2_3:
retcode = asu2_3(sp);
break;
case ASU3_1:
retcode = asu3_1(sp);
break;
default:
return SPC1_EASU;
}
spc1_io->asu = stream_id_to_asu(sp->io_heap->i_stream_id);
spc1_io->dir = sp->io_heap->i_op;
spc1_io->len = sp->io_heap->i_len;
spc1_io->bsu = stream_id_to_bsu(sp->io_heap->i_stream_id);
spc1_io->stream = sp->io_heap->i_stream_id;
spc1_io->pos = sp->io_heap->i_block_addr;
spc1_io->when = sp->io_heap->i_next_time;
#ifdef VALIDATE
if (ignore)
goto again;
#endif
/*
* Do a fixup for small ASU sizes. This allows the code to run,
* even though the results may not be representative of SPC-1
* results.
*/
switch(stream_id_to_asu(sp->io_heap->i_stream_id)) {
case 1: spc1_io->pos /= asu1_mult; break;
case 2: spc1_io->pos /= asu2_mult; break;
case 3: spc1_io->pos /= asu3_mult; break;
default: return SPC1_EASU;
}
return retcode;
}
int
spc1_next_op(struct spc1_io_s *s, int context)
{
struct state_block_s *sp;
int i;
int retcode;
/*
* context is supposed to be in the range [0, n_state_blocks-1].
* However, it is simpler to fix it than to throw an error.
*/
if (context < 0)
context = 0;
sp = states + (context % n_state_blocks);
retcode = gen_io_i(s, sp);
// Correct the bsu index to account for all contexts
for (i = 0; i < context; i++) {
sp = states + (i % n_state_blocks);
s->bsu += sp->bsu_count;
}
printf("In spc1_next_op: context = %d, bsu = %d\n", context, s->bsu);
requeue(sp);
return retcode;
}
int
spc1_next_op_any(struct spc1_io_s *s)
{
struct state_block_s *sp;
static int last_context = 0;
int i, best_context;
int f, best_time;
int retcode;
/*
* Loop over all contexts finding the next op
* The complexity is added to break ties in a round-robin
* fashion.
*/
f = 0;
best_time = 0; /* suppress warnings */
best_context = 0; /* suppress warnings */
for (i = 0; i < n_state_blocks; i++) {
sp = states + ((i + last_context + 1) % n_state_blocks);
if (!f || best_time > sp->io_heap->i_next_time) {
f = 1;
best_time = sp->io_heap->i_next_time;
best_context = sp - states;
}
}
last_context = best_context;
sp = states + best_context;
retcode = gen_io_i(s, sp);
// Correct the bsu index to account for all contexts
for (i = 0; i < best_context; i++) {
sp = states + (i % n_state_blocks);
s->bsu += sp->bsu_count;
}
requeue(sp);
return retcode;
}
/*
* ASU1 and ASU2 have a minimum size or the hot spots do not work.
* If the requested ASU is smaller than the legal minimum, we make
* the ASU bigger by some mulitplier and then divide the answers by
* the same multiplier.
* This algorithm does not comply with the letter of the specification,
* but this is better than throwing an error and more practical than
* fixing all the HRRW walk code to deal with very small regions.
*/
static int
spc1_compute_multiplier(int size, int min)
{
int mult = 1;
int total = size;
if (total >= min)
return 1;
while (total * 10 < min) {
total += 10 * size;
mult += 10;
}
while (total < min) {
total += size;
++mult;
}
return mult;
}
int
spc1_init(char *m,
int b,
unsigned int a1,
unsigned int a2,
unsigned int a3,
int n_contexts,
char *version,
int len)
{
int i, j;
int mbc, rbc;
unsigned u;
struct state_block_s *sp;
int retcode;
#ifdef VALIDATE
myname = m;
#endif
if (n_contexts < 1)
n_contexts = 1;
n_state_blocks = n_contexts;
/* Provide a version string even if we fail early */
if (version)
snprintf(version, len,
"%s [bsu=%d asu1=%d asu2=%d asu3=%d ctx=%d mth=%s t=%d]",
Version, b, a1, a2, a3, n_contexts,
SPC1_USE_INTEGER_MATH ? "i" : "f",
TIME_UNITS_PER_SECOND);
u = (unsigned)(n_state_blocks * sizeof (struct state_block_s));
states = (struct state_block_s *)malloc(u);
if (states == (struct state_block_s *)0) {
return SPC1_ENOMEM;
}
asu1_mult = spc1_compute_multiplier(a1, 128 * 8 * 20); /* 2^7 * 32k / 5% */
asu2_mult = spc1_compute_multiplier(a2, 128 * 8 * 20); /* 2^7 * 32k / 5% */
asu3_mult = 1;
asu1_size = a1 * asu1_mult;
asu2_size = a2 * asu2_mult;
asu3_size = a3 * asu3_mult;
mbc = b / n_state_blocks;
rbc = b % n_state_blocks;
/* Provide a full version string after multipliers are calculated */
if (version)
snprintf(version, len,
"%s [bsu=%d asu1=%d*%d asu2=%d*%d asu3=%d*%d"
" ctx=%d mth=%s t=%d]",
Version, b, a1, asu1_mult, a2, asu2_mult, a3,
asu3_mult, n_contexts,
SPC1_USE_INTEGER_MATH ? "i" : "f",
TIME_UNITS_PER_SECOND);
for (i = 0; i < n_state_blocks; i++) {
sp = states + i;
/*
* The mbc/rbc stuff distributes the streams evenly
* over the state blocks.
*/
retcode = init(sp, mbc + (rbc-- > 0? 1: 0));
if (retcode)
return retcode;
/*
* Initialize each of the streams
*/
for (j = 0; j < sp->stream_count; j++) {
struct spc1_io_s spc1_io;
retcode = gen_io_i(&spc1_io, sp);
if (retcode)
return retcode;
requeue(sp);
}
}
return SPC1_ENOERR;
}
#ifndef SPC1_WRAPPER_H_
#define SPC1_WRAPPER_H_
/* Can change these parameters */
#define SINGLE
//#define PROCS
//#define THREADS
#define BSU 198 // # Max working with PROCS = 25, with SINGLE = 400
#define BEGIN_CONTEXT 0 /* This limits the BSUs to be those running on context = 0 */
#define END_CONTEXT 0 /* and context = 0 */
//#define BSU 99 // # Max working with PROCS = 25, with SINGLE = 400
//#define BEGIN_CONTEXT 0 /* This limits the BSUs to be those running on context = 0 */
//#define END_CONTEXT 0 /* and context = 1 */
#define MIN_IOPS_GENERATED 100
#define RUNTIME_HRS 0
#define RUNTIME_MINS 30
#define RUNTIME_SECS 0
#define ASU1 "/dev/mapper/test-asu1"
#define ASU2 "/dev/mapper/test-asu2"
#define ASU3 "/dev/mapper/test-asu3"
#define ASU1_SIZE_GB 20
#define ASU2_SIZE_GB 20
#define ASU3_SIZE_GB 20
#define BLOCK_SIZE_KB 4
#define PNAME "fio (v1.36)"
#define GIGABYTE 1073741824
#define MEGABYTE 1048576
#define KILOBYTE 1024
#define GB_TO_BLOCK( g ) ( (g)*(GIGABYTE/KILOBYTE)*BLOCK_SIZE_KB )
#define BLOCK_TO_B( b ) ( (b)*BLOCK_SIZE_KB*KILOBYTE )
#define BUFLEN 1000
#define IOPS_INFO 100000
#define IOPS_ALL 27900000
/* No more changes required below here */
#define BSU_ID_FOR_SINGLE -1
#define STR_ID_FOR_SINGLE -1
#define ASU 3
#define STREAMS 8
#define AVG_IOPS 50
#define TOTAL_TIME_SECS( h, m, s ) ( 60*60*(h) + 60*(m) + (s) )
#define IN_MILLIS( s ) ( 1000*(s) )
#define TOTAL_RUNTIME_SECS TOTAL_TIME_SECS(RUNTIME_HRS, RUNTIME_MINS, RUNTIME_SECS)
#define TOTAL_RUNTIME_MILLIS IN_MILLIS(TOTAL_RUNTIME_SECS)
#ifndef _SPC1_H
#define _SPC1_H
#include "spc1.h"
#endif
#include "fio.h"
extern short use_spc1;
extern struct spc1_io_s *** iostore;
extern unsigned ** iocount;
extern unsigned spc1_context_from_bsu(unsigned bsu);
extern int gen_spc1_ios();
extern char* gen_spc1_file(char *string, unsigned int *global_addr,
unsigned int *line_addr, int *bsu_addr, int *str_addr);
extern int init_spc1_io(struct thread_data *td);
extern int fin_spc1_io(struct thread_data *td);
extern int get_spc1_io(struct thread_data *td, struct io_u *io_u_addr);
extern void spc1_io_debug_info(const char *pre, struct spc1_io_s *spc1_io_s_addr);
extern void fio_io_debug_info(const char *pre, struct io_u *io_u_addr);
extern unsigned short spc1_ios_left(struct thread_data *td);
#endif /*SPC1_WRAPPER_H_*/
From 73d246dd4dcd7ae18c1c6e0ba16737fcd67bad3c Mon Sep 17 00:00:00 2001
From: unknown <mosu001@.(none)>
Date: Wed, 10 Feb 2010 11:06:38 +1300
Subject: [PATCH] Initial integration of open SPC-1 code into fio
---
spc1_wrapper.c | 660 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
spc1_wrapper.h | 93 ++++++++
2 files changed, 753 insertions(+), 0 deletions(-)
create mode 100644 spc1_wrapper.c
create mode 100644 spc1_wrapper.h
diff --git a/spc1_wrapper.c b/spc1_wrapper.c
new file mode 100644
index 0000000..3c59de5
--- /dev/null
+++ b/spc1_wrapper.c
@@ -0,0 +1,660 @@
+/*
+ * spc1_wrapper.c
+ *
+ * Created on: 23/12/2009
+ * Author: Michael
+ */
+
+#ifdef _USE_SPC1
+
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#ifndef _SPC1_H
+#define _SPC1_H
+#include "spc1.h"
+#endif
+
+#include "fio.h"
+
+#include "spc1_wrapper.h"
+
+short use_spc1;
+
+struct spc1_io_s *** iostore;
+
+unsigned ** iocount;
+
+void spc1_io_debug_info(const char *pre, struct spc1_io_s *spc1_io_s_addr) {
+ printf("%s", pre);
+ printf(": address = %d, ASU = %d, R/W = %d, length = %d, bsu stream = %d, bsu = %d, pos = %d, time = %d\n",
+ spc1_io_s_addr,
+ spc1_io_s_addr->asu,
+ spc1_io_s_addr->dir,
+ spc1_io_s_addr->len,
+ spc1_io_s_addr->stream,
+ spc1_io_s_addr->bsu,
+ spc1_io_s_addr->pos,
+ spc1_io_s_addr->when); // .when in 0.1 milliseconds
+ printf("pid = %d\n", getpid());
+ fflush(stdout);
+}
+void fio_io_debug_info(const char *pre, struct io_u *io_u_addr) {
+ printf("%s", pre);
+ printf(": address = %d, R/W = %d, offset = %lld, length = %ld, file %d, ",
+ io_u_addr,
+ io_u_addr->ddir,
+ io_u_addr->offset,
+ io_u_addr->buflen,
+ io_u_addr->file);
+ printf("pid = %d\n", getpid());
+ fflush(stdout);
+}
+
+unsigned spc1_context_from_bsu(unsigned bsu) {
+ int retcode;
+
+ retcode = (bsu + 1) / 100;
+
+ return retcode;
+}
+
+int gen_spc1_ios() {
+ int numContexts = (BSU + 1) / 100;
+ char vbuf[BUFLEN];
+ char pname[] = PNAME;
+
+ struct spc1_io_s nextio;
+ int retcode;
+ unsigned int i, j, bsu, stream, totalio, iops;
+ unsigned long long whenMillis, finalMillis = TOTAL_RUNTIME_MILLIS;
+ char message[BUFLEN];
+ unsigned numStarted, numFinished;
+ int status[BSU][STREAMS];
+
+ printf("Generating SPC-1 workload, num contexts = %d...\n", numContexts);
+
+ printf("Initialising data structures...\n");
+
+ assert( (BEGIN_CONTEXT >= 0) && (BEGIN_CONTEXT <= END_CONTEXT) && (END_CONTEXT <= numContexts) );
+
+ iostore = (struct spc1_io_s ***)malloc(BSU * sizeof(struct spc1_io_s **));
+ iocount = (unsigned **)malloc(BSU * sizeof(unsigned *));
+ for (i=0; i<BSU; i++) {
+ if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
+ (spc1_context_from_bsu(i) <= END_CONTEXT) ) {
+ iostore[i] = (struct spc1_io_s **)malloc(STREAMS * sizeof(struct spc_io_u *));
+ iocount[i] = (unsigned *)malloc(STREAMS * sizeof(unsigned));
+ for (j=0; j<STREAMS; j++) {
+ iostore[i][j] = (struct spc1_io_s *)malloc(AVG_IOPS*TOTAL_RUNTIME_SECS * sizeof(struct spc1_io_s));
+ iocount[i][j] = 0;
+ }
+ }
+ }
+ memset(status, 0, BSU * STREAMS* sizeof(int)); /* Flags to see if stream started/finished */
+
+
+#ifdef _SPC1_DEBUG
+ printf("SPC-1 checking value of iocount, pid = %d\n", getpid());
+ for (i=0; i<BSU; i++)
+ if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
+ (spc1_context_from_bsu(i) <= END_CONTEXT) )
+ for (j=0; j<STREAMS; j++)
+ printf("iocount[%d][%d] = %d\n", i, j, iocount[i][j]);
+ fflush(stdout);
+#endif
+
+ printf("Initialising SPC-1 benchmark generator...\n");
+
+ retcode = spc1_init(pname, // char *m Name of the program. Used for error messages.
+ BSU, // int b Number of BSUs.
+ GB_TO_BLOCK(ASU1_SIZE_GB), // unsigned int a1 Size of ASU 1 in 4K blocks.
+ GB_TO_BLOCK(ASU2_SIZE_GB), // unsigned int a2 Size of ASU 2 in 4K blocks.
+ GB_TO_BLOCK(ASU3_SIZE_GB), // unsigned int a3 Size of ASU 3 in 4K blocks.
+ numContexts, // int n_contexts The number of context blocks to allocate
+ vbuf, // version An output buffer where a version string may be written. If NULL, no version is written.
+ BUFLEN); // len The length of the output buffer.
+
+ if (retcode != SPC1_ENOERR) {
+ printf("Error initialising SPC-1 workload, errcode = %d\n", retcode);
+ return 1;
+ }
+
+ numStarted = numFinished = 0; /* No stream from any BSU has started/finished generating load */
+
+ printf("Initialisation complete!\n");
+
+ /* Generate SPC-1 IOs until the set time has elapsed */
+ printf("Creating SPC-1 workload...\n");
+
+ iops = 0;
+ do {
+ retcode = spc1_next_op_any(&nextio);
+ iops++;
+
+ if (retcode == SPC1_ENOERR) {
+#ifdef _SPC1_DEBUG
+ if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
+ sprintf(message, "IOP #%d, context = %d", iops, spc1_context_from_bsu(nextio.bsu));
+ spc1_io_debug_info(message, &nextio);
+ }
+#endif
+ if ( (spc1_context_from_bsu(nextio.bsu) < BEGIN_CONTEXT) ||
+ (spc1_context_from_bsu(nextio.bsu) > END_CONTEXT) )
+ continue; /* This bsu is outside the contexts considered */
+
+ bsu = nextio.bsu;
+ stream = nextio.stream;
+ whenMillis = nextio.when / 10;
+
+ if (status[bsu][stream] == -1)
+ continue; /* Nothing to do for this (bsu, stream), go to next IO */
+ else {
+ if (status[bsu][stream] == 0) { /* (bsu, stream) not started yet */
+ status[bsu][stream] = 1; /* Start (bsu, stream) */
+ numStarted++; /* One more (bsu, steam) started */
+ }
+
+#ifdef _SPC1_DEBUG
+ if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
+ printf("current = %lld, max = %lld\n", whenMillis, finalMillis);
+ fflush(stdout);
+ }
+#endif
+ if (whenMillis > finalMillis) { /* IO happens after time limit */
+ status[bsu][stream] = -1; /* Stop generating for this (bsu, stream) */
+ numFinished++; /* One more (bsu, steam) finished */
+ } else {
+#ifdef _SPC1_DEBUG
+ if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
+ printf("Adding SPC-1 IO\n");
+ fflush(stdout);
+ }
+#endif
+ iostore[bsu][stream][iocount[bsu][stream]] = nextio; /* Store IO for (bsu, stream) */
+ iocount[bsu][stream]++; /* One more IO for (bsu, stream) */
+ }
+ }
+
+#ifdef _SPC1_DEBUG
+ if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
+ printf("started = %d, finished = %d\n", numStarted, numFinished);
+ fflush(stdout);
+ }
+#endif
+ } else {
+ printf("Problem creating workload.");
+ return 1;
+ }
+
+ } while ( (numStarted > numFinished) || (iops <= MIN_IOPS_GENERATED) );
+
+ totalio = 0;
+ for (i=0; i<BSU; i++)
+ if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
+ (spc1_context_from_bsu(i) <= END_CONTEXT) )
+ for (j=0; j<STREAMS;j++) {
+ totalio += iocount[i][j];
+#ifdef _SPC1_DEBUG
+ printf("iocount[%d][%d] = %d\n", i, j, iocount[i][j]);
+ fflush(stdout);
+#endif
+ }
+ printf("totalio = %d, totaltime = %d, average = %g\n",
+ totalio, TOTAL_RUNTIME_SECS, (double)totalio / TOTAL_RUNTIME_SECS / BSU);
+
+ printf("Workload created!\n");
+
+ return 0;
+}
+
+extern char* gen_spc1_file(char *string, unsigned int *global_addr,
+ unsigned int *line_addr, int *bsu_addr, int *str_addr) {
+
+ if (*global_addr) {
+#ifdef _SPC1_DEBUG
+ printf("SPC-1, global section, line = %d\n", *line_addr);
+#endif
+ switch (*line_addr) {
+ case 0:
+ sprintf(string, "[global]");
+ (*line_addr)++;
+ break;
+ case 1:
+ sprintf(string, "rw=randrw");
+ *global_addr = 0;
+ *line_addr = 0;
+#ifdef SINGLE
+ *bsu_addr = BSU_ID_FOR_SINGLE;
+#else
+ *bsu_addr = 0;
+#endif
+ *str_addr = 0;
+ break;
+ }
+
+ } else {
+
+ if (*bsu_addr == BSU_ID_FOR_SINGLE) {
+
+ switch (*line_addr) {
+ case 0:
+ sprintf(string, "[spc_all]");
+ (*line_addr)++;
+ break;
+ case 1:
+ sprintf(string, "write_bw_log=spc_all");
+ (*line_addr)++;
+ break;
+ case 2:
+ sprintf(string, "write_lat_log=spc_all");
+ *str_addr = STREAMS;
+ *bsu_addr = BSU;
+ *line_addr = 0;
+ break;
+ }
+
+ } else if (*bsu_addr == BSU) {
+#ifdef _SPC1_DEBUG
+ printf("SPC-1 generate null string\n");
+#endif
+ return NULL;
+ } else if (iocount[*bsu_addr][*str_addr] > 0) {
+#ifdef _SPC1_DEBUG
+ printf("SPC-1, bsu%d_str%d section, line = %d\n", *bsu_addr, *str_addr, *line_addr);
+#endif
+ switch (*line_addr) {
+ case 0:
+ sprintf(string, "[bsu%d_str%d]", *bsu_addr, *str_addr);
+ (*line_addr)++;
+ break;
+ case 1:
+ sprintf(string, "write_bw_log=spc_bsu%d", *bsu_addr);
+ (*line_addr)++;
+ break;
+ case 2:
+ sprintf(string, "write_lat_log=spc_bsu%d", *bsu_addr);
+ (*str_addr)++;
+ if (*str_addr == STREAMS) {
+ (*bsu_addr)++;
+ *str_addr = 0;
+ }
+ *line_addr = 0;
+ break;
+ }
+ } else {
+ (*str_addr)++;
+ if (*str_addr == STREAMS) {
+ (*bsu_addr)++;
+ *str_addr = 0;
+ }
+ *line_addr = 0;
+ }
+ }
+
+#ifdef _SPC1_DEBUG
+ printf("SPC-1 generate string %s\n", string);
+#endif
+ return string;
+}
+
+int init_spc1_io(struct thread_data *td) {
+ int i, j, k;
+ int fileno;
+
+ if ( (td->bsu >= BSU) || (td->str >= STREAMS) ) return 1;
+#ifdef SINGLE
+#ifdef _SPC1_DEBUG
+ printf("Initialising max_bs...\n");
+#endif
+ td->single_iopos = (unsigned **) malloc(BSU * sizeof(unsigned *));
+ for (i=0; i<BSU; i++)
+ if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
+ (spc1_context_from_bsu(i) <= END_CONTEXT) ) {
+ td->single_iopos[i] = (unsigned *) malloc(STREAMS * sizeof(unsigned));
+ for (j=0; j<STREAMS; j++) {
+#ifdef _SPC1_DEBUG
+ printf("Initialising position, bsu = %d, str = %d\n", i, j);
+ fflush(stdout);
+#endif
+ td->single_iopos[i][j] = 0;
+ if (td->single_iocount[i][j] < 0) return 1;
+#ifdef _SPC1_DEBUG
+ printf("Checking max size, bsu = %d, str = %d\n", i, j);
+ fflush(stdout);
+// printf("for %d positions\n", td->single_iocount[i][j]);
+// fflush(stdout);
+#endif
+ for (k=0; k<td->single_iocount[i][j]; k++) {
+ int rw = td->single_iostore[i][j][k].dir;
+ int bytes = BLOCK_TO_B(td->single_iostore[i][j][k].len);
+
+#ifdef _SPC1_DEBUG
+ printf("IO from bsu = %d, str = %d, pos = %d has size %d\n", i, j, k, bytes);
+#endif
+ if (bytes > td->o.max_bs[rw])
+ td->o.max_bs[rw] = bytes;
+ }
+ }
+ }
+
+#ifdef _SPC1_DEBUG
+ printf("Adding ASU1 file...\n");
+#endif
+ fileno = get_fileno(td, ASU1);
+ if (fileno == -1) {
+ td->o.nr_files++;
+ fileno = add_file(td, ASU1);
+ }
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("ASU1 file could not be added...\n");
+#endif
+ return 1;
+ }
+ if (td_io_open_file(td, td->files[fileno])) {
+#ifdef _SPC1_DEBUG
+ printf("ASU1 file could not be opened...\n");
+#endif
+ return 1;
+ }
+#ifdef _SPC1_DEBUG
+ printf("Adding ASU2 file...\n");
+#endif
+ fileno = get_fileno(td, ASU2);
+ if (fileno == -1) {
+ td->o.nr_files++;
+ fileno = add_file(td, ASU2);
+ }
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("ASU2 file could not be added...\n");
+#endif
+ return 1;
+ }
+ if (td_io_open_file(td, td->files[fileno])) {
+#ifdef _SPC1_DEBUG
+ printf("ASU2 file could not be opened...\n");
+#endif
+ return 1;
+ }
+#ifdef _SPC1_DEBUG
+ printf("Adding ASU3 file...\n");
+#endif
+ fileno = get_fileno(td, ASU3);
+ if (fileno == -1) {
+ td->o.nr_files++;
+ fileno = add_file(td, ASU3);
+ }
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("ASU3 file could not be added...\n");
+#endif
+ return 1;
+ }
+ if (td_io_open_file(td, td->files[fileno])) {
+#ifdef _SPC1_DEBUG
+ printf("ASU3 file could not be opened...\n");
+#endif
+ return 1;
+ }
+
+#else
+ if (td->iocount <= 0) return 1;
+
+ td->iopos = 0;
+ // Find the maximum block size for creating buffers
+ for (i=0; i<td->iocount; i++) {
+ int rw = td->iostore[i].dir;
+ int bytes = BLOCK_TO_B(td->iostore[i].len);
+
+ if (bytes > td->o.max_bs[rw])
+ td->o.max_bs[rw] = bytes;
+ }
+#endif
+
+#ifdef _SPC1_DEBUG
+ printf("Initialisation finished for bsu = %d, str = %d, pid = %d\n", td->bsu, td->str, getpid());
+ fflush(stdout);
+#endif
+
+ return 0;
+}
+
+int fin_spc1_io(struct thread_data *td) {
+ int fileno;
+
+ if ( (td->bsu >= BSU) || (td->str >= STREAMS) ) return 1;
+#ifdef SINGLE
+
+ fileno = get_fileno(td, ASU1);
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("ASU1 file not open when closing...\n");
+#endif
+ return 1;
+ }
+ td_io_close_file(td, td->files[fileno]);
+ fileno = get_fileno(td, ASU2);
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("ASU2 file not open when closing...\n");
+#endif
+ return 1;
+ }
+ td_io_close_file(td, td->files[fileno]);
+ fileno = get_fileno(td, ASU3);
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("ASU3 file not open when closing...\n");
+#endif
+ return 1;
+ }
+ td_io_close_file(td, td->files[fileno]);
+
+#else
+ if (td->iocount <= 0) return 1;
+#endif
+
+#ifdef _SPC1_DEBUG
+ printf("Finalisation finished for bsu = %d, str = %d, pid = %d\n", td->bsu, td->str, getpid());
+ fflush(stdout);
+#endif
+ return 0;
+}
+
+int get_spc1_io(struct thread_data *td, struct io_u *io_u_addr) {
+
+ struct spc1_io_s spc1_io;
+ int i, j, bsu, str, pos, fileno;
+ unsigned long elapsed, whenMillis, least;
+ unsigned short found;
+
+#ifdef _SPC1_DEBUG
+ printf("Reading from SPC-1 IOs, pid = %d\n", getpid());
+ fflush(stdout);
+#endif
+
+#ifdef SINGLE
+ least = TOTAL_RUNTIME_MILLIS;
+ found = 0;
+ for (i=0; i<BSU; i++)
+ if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
+ (spc1_context_from_bsu(i) <= END_CONTEXT) ) {
+ for (j=0; j<STREAMS; j++) {
+#ifdef _SPC1_DEBUG
+ printf("single_iocount[%d][%d] = %d, single_iopos[%d][%d] = %d\n",
+ i, j, td->single_iocount[i][j],
+ i, j, td->single_iopos[i][j]
+ );
+#endif
+ if (td->single_iopos[i][j] < td->single_iocount[i][j])
+ if (td->single_iostore[i][j][td->single_iopos[i][j]].when / 10 < least) {
+ bsu = i;
+ str = j;
+ pos = td->single_iopos[i][j];
+#ifdef _SPC1_DEBUG
+ printf("Earlier SPC-1 IOs, bsu = %d, str = %d, pos = %d, pid = %d\n", bsu, str, pos, getpid());
+ fflush(stdout);
+#endif
+ least = td->single_iostore[bsu][str][pos].when / 10;
+ found = 1;
+ }
+ }
+ }
+ if (!found) {
+ td->done = 1;
+ return 1;
+ }
+ spc1_io = td->single_iostore[bsu][str][pos];
+
+#else
+ bsu = td->bsu;
+ str = td->str;
+
+ pos = td->iopos;
+
+ if (pos == td->iocount) {
+ printf("IOs exhausted... bsu = %d, str = %d, pid = %d\n", bsu, str, getpid());
+ /* No more IOs from this (bsu, stream) */
+ td->done = 1;
+ return 1;
+ }
+
+#ifdef _SPC1_DEBUG
+ printf("IO # %d from bsu = %d, str = %d, pid = %d\n", pos, bsu, str, getpid());
+#endif
+ fflush(stdout);
+ spc1_io = td->iostore[pos];
+#endif
+
+#ifdef _SPC1_DEBUG
+ spc1_io_debug_info("SPC-1 IOP", &spc1_io);
+ fflush(stdout);
+#endif
+
+ elapsed = mtime_since_genesis();
+ whenMillis = spc1_io.when / 10;
+#ifdef _SPC1_DEBUG
+ printf("SPC-1 IO: elapsed = %ld, when = %ld, pid = %d\n", elapsed, whenMillis, getpid());
+ fflush(stdout);
+#endif
+ if (whenMillis > elapsed) {
+ usec_sleep(td, (whenMillis - elapsed) * 1000);
+ }
+
+#ifdef _SPC1_DEBUG
+ printf("SPC-1 IO: wait over\n");
+ fflush(stdout);
+#endif
+
+ io_u_addr->ddir = spc1_io.dir;
+ io_u_addr->offset = spc1_io.pos;
+// io_u_addr->offset = BLOCK_TO_B(spc1_io.pos);
+ io_u_addr->buflen = BLOCK_TO_B(spc1_io.len);
+
+#ifdef _SPC1_DEBUG
+ printf("IO # %d from bsu = %d, str = %d, pid = %d, identifying file...\n", pos, bsu, str, getpid());
+ fflush(stdout);
+#endif
+ /* Get file to read from/write to */
+ switch (td->str) {
+ case 0: case 1: case 2: case 3:
+ fileno = get_fileno(td, ASU1);
+ if (fileno == -1) {
+ td->o.nr_files++;
+ fileno = add_file(td, ASU1);
+ }
+ break;
+ case 4: case 5: case 6:
+ fileno = get_fileno(td, ASU2);
+ if (fileno == -1) {
+ td->o.nr_files++;
+ fileno = add_file(td, ASU2);
+ }
+ break;
+ default: /* case 7: */
+ fileno = get_fileno(td, ASU3);
+ if (fileno == -1) {
+ td->o.nr_files++;
+ fileno = add_file(td, ASU3);
+ }
+ break;
+ }
+ if (fileno == -1) {
+#ifdef _SPC1_DEBUG
+ printf("IO file has not been added... bsu = %d, str = %d, pid = %d\n", bsu, str, getpid());
+#endif
+ return 1;
+ }
+
+ io_u_addr->file = td->files[fileno];
+
+#ifndef SINGLE
+
+ if (pos == 0) {
+#ifdef _SPC1_DEBUG
+ printf("IO # %d from bsu = %d, str = %d, pid = %d, opening file # %d...\n", pos, bsu, str, getpid(), fileno);
+#endif
+ if (td_io_open_file(td, io_u_addr->file)) {
+#ifdef _SPC1_DEBUG
+ printf("IO file could not be opened... bsu = %d, str = %d, pid = %d\n", bsu, str, getpid());
+#endif
+ return 1;
+ }
+ }
+
+ if (pos == td->iocount) {
+#ifdef _SPC1_DEBUG
+ printf("IO # %d from bsu = %d, str = %d, pid = %d, closing file # %d...\n", pos, bsu, str, getpid(), fileno);
+#endif
+ td_io_close_file(td, io_u_addr->file);
+ }
+
+#endif
+
+#ifdef _SPC1_DEBUG
+ printf("IO # %d from bsu = %d, str = %d, pid = %d, getting file %d...\n", pos, bsu, str, getpid(), io_u_addr->file);
+ fflush(stdout);
+#endif
+ get_file(io_u_addr->file);
+
+#ifdef _SPC1_DEBUG
+ fio_io_debug_info("SPC-1 io_u", io_u_addr);
+ fflush(stdout);
+#endif
+
+#ifdef SINGLE
+ td->single_iopos[bsu][str]++;
+#else
+ td->iopos++;
+#endif
+
+#ifdef _SPC1_DEBUG
+ printf("Finished reading IO # %d from bsu = %d, str = %d, pid = %d\n", pos, bsu, str, getpid());
+ fflush(stdout);
+#endif
+
+ return 0;
+}
+
+unsigned short spc1_ios_left(struct thread_data *td) {
+ unsigned short left = 0;
+ int i, j;
+
+ for (i=0; i<BSU; i++)
+ if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
+ (spc1_context_from_bsu(i) <= END_CONTEXT) ) {
+ for (j=0; j<STREAMS; j++)
+ if (td->single_iopos[i][j] < td->single_iocount[i][j]) {
+ left = 1;
+ break;
+ }
+ if (left) break;
+ }
+
+ return left;
+}
+
+#endif
diff --git a/spc1_wrapper.h b/spc1_wrapper.h
new file mode 100644
index 0000000..9d30dde
--- /dev/null
+++ b/spc1_wrapper.h
@@ -0,0 +1,93 @@
+#ifndef SPC1_WRAPPER_H_
+#define SPC1_WRAPPER_H_
+
+/* Can change these parameters */
+#define SINGLE
+//#define PROCS
+//#define THREADS
+
+#define BSU 198 // # Max working with PROCS = 25, with SINGLE = 400
+#define BEGIN_CONTEXT 0 /* This limits the BSUs to be those running on context = 0 */
+#define END_CONTEXT 0 /* and context = 0 */
+
+//#define BSU 99 // # Max working with PROCS = 25, with SINGLE = 400
+//#define BEGIN_CONTEXT 0 /* This limits the BSUs to be those running on context = 0 */
+//#define END_CONTEXT 0 /* and context = 1 */
+
+#define MIN_IOPS_GENERATED 100
+
+#define RUNTIME_HRS 0
+#define RUNTIME_MINS 30
+#define RUNTIME_SECS 0
+
+#define ASU1 "/dev/mapper/test-asu1"
+#define ASU2 "/dev/mapper/test-asu2"
+#define ASU3 "/dev/mapper/test-asu3"
+
+#define ASU1_SIZE_GB 20
+#define ASU2_SIZE_GB 20
+#define ASU3_SIZE_GB 20
+
+#define BLOCK_SIZE_KB 4
+
+#define PNAME "fio (v1.36)"
+
+#define GIGABYTE 1073741824
+#define MEGABYTE 1048576
+#define KILOBYTE 1024
+
+#define GB_TO_BLOCK( g ) ( (g)*(GIGABYTE/KILOBYTE)*BLOCK_SIZE_KB )
+#define BLOCK_TO_B( b ) ( (b)*BLOCK_SIZE_KB*KILOBYTE )
+
+#define BUFLEN 1000
+
+#define IOPS_INFO 100000
+#define IOPS_ALL 27900000
+
+/* No more changes required below here */
+#define BSU_ID_FOR_SINGLE -1
+#define STR_ID_FOR_SINGLE -1
+
+#define ASU 3
+
+#define STREAMS 8
+
+#define AVG_IOPS 50
+
+#define TOTAL_TIME_SECS( h, m, s ) ( 60*60*(h) + 60*(m) + (s) )
+#define IN_MILLIS( s ) ( 1000*(s) )
+
+#define TOTAL_RUNTIME_SECS TOTAL_TIME_SECS(RUNTIME_HRS, RUNTIME_MINS, RUNTIME_SECS)
+#define TOTAL_RUNTIME_MILLIS IN_MILLIS(TOTAL_RUNTIME_SECS)
+
+#ifndef _SPC1_H
+#define _SPC1_H
+#include "spc1.h"
+#endif
+
+#include "fio.h"
+
+extern short use_spc1;
+
+extern struct spc1_io_s *** iostore;
+
+extern unsigned ** iocount;
+
+extern unsigned spc1_context_from_bsu(unsigned bsu);
+
+extern int gen_spc1_ios();
+
+extern char* gen_spc1_file(char *string, unsigned int *global_addr,
+ unsigned int *line_addr, int *bsu_addr, int *str_addr);
+
+extern int init_spc1_io(struct thread_data *td);
+extern int fin_spc1_io(struct thread_data *td);
+
+extern int get_spc1_io(struct thread_data *td, struct io_u *io_u_addr);
+
+extern void spc1_io_debug_info(const char *pre, struct spc1_io_s *spc1_io_s_addr);
+extern void fio_io_debug_info(const char *pre, struct io_u *io_u_addr);
+
+extern unsigned short spc1_ios_left(struct thread_data *td);
+
+#endif /*SPC1_WRAPPER_H_*/
--
1.6.5.1.1367.gcd48
/*
* spc1_wrapper.c
*
* Created on: 23/12/2009
* Author: Michael
*/
#ifdef _USE_SPC1
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#ifndef _SPC1_H
#define _SPC1_H
#include "spc1.h"
#endif
#include "fio.h"
#include "spc1_wrapper.h"
short use_spc1;
struct spc1_io_s *** iostore;
unsigned ** iocount;
void spc1_io_debug_info(const char *pre, struct spc1_io_s *spc1_io_s_addr) {
printf("%s", pre);
printf(": address = %d, ASU = %d, R/W = %d, length = %d, bsu stream = %d, bsu = %d, pos = %d, time = %d\n",
spc1_io_s_addr,
spc1_io_s_addr->asu,
spc1_io_s_addr->dir,
spc1_io_s_addr->len,
spc1_io_s_addr->stream,
spc1_io_s_addr->bsu,
spc1_io_s_addr->pos,
spc1_io_s_addr->when); // .when in 0.1 milliseconds
printf("pid = %d\n", getpid());
fflush(stdout);
}
void fio_io_debug_info(const char *pre, struct io_u *io_u_addr) {
printf("%s", pre);
printf(": address = %d, R/W = %d, offset = %lld, length = %ld, file %d, ",
io_u_addr,
io_u_addr->ddir,
io_u_addr->offset,
io_u_addr->buflen,
io_u_addr->file);
printf("pid = %d\n", getpid());
fflush(stdout);
}
unsigned spc1_context_from_bsu(unsigned bsu) {
int retcode;
retcode = (bsu + 1) / 100;
return retcode;
}
int gen_spc1_ios() {
int numContexts = (BSU + 1) / 100;
char vbuf[BUFLEN];
char pname[] = PNAME;
struct spc1_io_s nextio;
int retcode;
unsigned int i, j, bsu, stream, totalio, iops;
unsigned long long whenMillis, finalMillis = TOTAL_RUNTIME_MILLIS;
char message[BUFLEN];
unsigned numStarted, numFinished;
int status[BSU][STREAMS];
printf("Generating SPC-1 workload, num contexts = %d...\n", numContexts);
printf("Initialising data structures...\n");
assert( (BEGIN_CONTEXT >= 0) && (BEGIN_CONTEXT <= END_CONTEXT) && (END_CONTEXT <= numContexts) );
iostore = (struct spc1_io_s ***)malloc(BSU * sizeof(struct spc1_io_s **));
iocount = (unsigned **)malloc(BSU * sizeof(unsigned *));
for (i=0; i<BSU; i++) {
if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
(spc1_context_from_bsu(i) <= END_CONTEXT) ) {
iostore[i] = (struct spc1_io_s **)malloc(STREAMS * sizeof(struct spc_io_u *));
iocount[i] = (unsigned *)malloc(STREAMS * sizeof(unsigned));
for (j=0; j<STREAMS; j++) {
iostore[i][j] = (struct spc1_io_s *)malloc(AVG_IOPS*TOTAL_RUNTIME_SECS * sizeof(struct spc1_io_s));
iocount[i][j] = 0;
}
}
}
memset(status, 0, BSU * STREAMS* sizeof(int)); /* Flags to see if stream started/finished */
#ifdef _SPC1_DEBUG
printf("SPC-1 checking value of iocount, pid = %d\n", getpid());
for (i=0; i<BSU; i++)
if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
(spc1_context_from_bsu(i) <= END_CONTEXT) )
for (j=0; j<STREAMS; j++)
printf("iocount[%d][%d] = %d\n", i, j, iocount[i][j]);
fflush(stdout);
#endif
printf("Initialising SPC-1 benchmark generator...\n");
retcode = spc1_init(pname, // char *m Name of the program. Used for error messages.
BSU, // int b Number of BSUs.
GB_TO_BLOCK(ASU1_SIZE_GB), // unsigned int a1 Size of ASU 1 in 4K blocks.
GB_TO_BLOCK(ASU2_SIZE_GB), // unsigned int a2 Size of ASU 2 in 4K blocks.
GB_TO_BLOCK(ASU3_SIZE_GB), // unsigned int a3 Size of ASU 3 in 4K blocks.
numContexts, // int n_contexts The number of context blocks to allocate
vbuf, // version An output buffer where a version string may be written. If NULL, no version is written.
BUFLEN); // len The length of the output buffer.
if (retcode != SPC1_ENOERR) {
printf("Error initialising SPC-1 workload, errcode = %d\n", retcode);
return 1;
}
numStarted = numFinished = 0; /* No stream from any BSU has started/finished generating load */
printf("Initialisation complete!\n");
/* Generate SPC-1 IOs until the set time has elapsed */
printf("Creating SPC-1 workload...\n");
iops = 0;
do {
retcode = spc1_next_op_any(&nextio);
iops++;
if (retcode == SPC1_ENOERR) {
#ifdef _SPC1_DEBUG
if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
sprintf(message, "IOP #%d, context = %d", iops, spc1_context_from_bsu(nextio.bsu));
spc1_io_debug_info(message, &nextio);
}
#endif
if ( (spc1_context_from_bsu(nextio.bsu) < BEGIN_CONTEXT) ||
(spc1_context_from_bsu(nextio.bsu) > END_CONTEXT) )
continue; /* This bsu is outside the contexts considered */
bsu = nextio.bsu;
stream = nextio.stream;
whenMillis = nextio.when / 10;
if (status[bsu][stream] == -1)
continue; /* Nothing to do for this (bsu, stream), go to next IO */
else {
if (status[bsu][stream] == 0) { /* (bsu, stream) not started yet */
status[bsu][stream] = 1; /* Start (bsu, stream) */
numStarted++; /* One more (bsu, steam) started */
}
#ifdef _SPC1_DEBUG
if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
printf("current = %lld, max = %lld\n", whenMillis, finalMillis);
fflush(stdout);
}
#endif
if (whenMillis > finalMillis) { /* IO happens after time limit */
status[bsu][stream] = -1; /* Stop generating for this (bsu, stream) */
numFinished++; /* One more (bsu, steam) finished */
} else {
#ifdef _SPC1_DEBUG
if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
printf("Adding SPC-1 IO\n");
fflush(stdout);
}
#endif
iostore[bsu][stream][iocount[bsu][stream]] = nextio; /* Store IO for (bsu, stream) */
iocount[bsu][stream]++; /* One more IO for (bsu, stream) */
}
}
#ifdef _SPC1_DEBUG
if ( (iops % IOPS_INFO == 0) || (iops >= IOPS_ALL) ) {
printf("started = %d, finished = %d\n", numStarted, numFinished);
fflush(stdout);
}
#endif
} else {
printf("Problem creating workload.");
return 1;
}
} while ( (numStarted > numFinished) || (iops <= MIN_IOPS_GENERATED) );
totalio = 0;
for (i=0; i<BSU; i++)
if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
(spc1_context_from_bsu(i) <= END_CONTEXT) )
for (j=0; j<STREAMS;j++) {
totalio += iocount[i][j];
#ifdef _SPC1_DEBUG
printf("iocount[%d][%d] = %d\n", i, j, iocount[i][j]);
fflush(stdout);
#endif
}
printf("totalio = %d, totaltime = %d, average = %g\n",
totalio, TOTAL_RUNTIME_SECS, (double)totalio / TOTAL_RUNTIME_SECS / BSU);
printf("Workload created!\n");
return 0;
}
extern char* gen_spc1_file(char *string, unsigned int *global_addr,
unsigned int *line_addr, int *bsu_addr, int *str_addr) {
if (*global_addr) {
#ifdef _SPC1_DEBUG
printf("SPC-1, global section, line = %d\n", *line_addr);
#endif
switch (*line_addr) {
case 0:
sprintf(string, "[global]");
(*line_addr)++;
break;
case 1:
sprintf(string, "rw=randrw");
*global_addr = 0;
*line_addr = 0;
#ifdef SINGLE
*bsu_addr = BSU_ID_FOR_SINGLE;
#else
*bsu_addr = 0;
#endif
*str_addr = 0;
break;
}
} else {
if (*bsu_addr == BSU_ID_FOR_SINGLE) {
switch (*line_addr) {
case 0:
sprintf(string, "[spc_all]");
(*line_addr)++;
break;
case 1:
sprintf(string, "write_bw_log=spc_all");
(*line_addr)++;
break;
case 2:
sprintf(string, "write_lat_log=spc_all");
*str_addr = STREAMS;
*bsu_addr = BSU;
*line_addr = 0;
break;
}
} else if (*bsu_addr == BSU) {
#ifdef _SPC1_DEBUG
printf("SPC-1 generate null string\n");
#endif
return NULL;
} else if (iocount[*bsu_addr][*str_addr] > 0) {
#ifdef _SPC1_DEBUG
printf("SPC-1, bsu%d_str%d section, line = %d\n", *bsu_addr, *str_addr, *line_addr);
#endif
switch (*line_addr) {
case 0:
sprintf(string, "[bsu%d_str%d]", *bsu_addr, *str_addr);
(*line_addr)++;
break;
case 1:
sprintf(string, "write_bw_log=spc_bsu%d", *bsu_addr);
(*line_addr)++;
break;
case 2:
sprintf(string, "write_lat_log=spc_bsu%d", *bsu_addr);
(*str_addr)++;
if (*str_addr == STREAMS) {
(*bsu_addr)++;
*str_addr = 0;
}
*line_addr = 0;
break;
}
} else {
(*str_addr)++;
if (*str_addr == STREAMS) {
(*bsu_addr)++;
*str_addr = 0;
}
*line_addr = 0;
}
}
#ifdef _SPC1_DEBUG
printf("SPC-1 generate string %s\n", string);
#endif
return string;
}
int init_spc1_io(struct thread_data *td) {
int i, j, k;
int fileno;
if ( (td->bsu >= BSU) || (td->str >= STREAMS) ) return 1;
#ifdef SINGLE
#ifdef _SPC1_DEBUG
printf("Initialising max_bs...\n");
#endif
td->single_iopos = (unsigned **) malloc(BSU * sizeof(unsigned *));
for (i=0; i<BSU; i++)
if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
(spc1_context_from_bsu(i) <= END_CONTEXT) ) {
td->single_iopos[i] = (unsigned *) malloc(STREAMS * sizeof(unsigned));
for (j=0; j<STREAMS; j++) {
#ifdef _SPC1_DEBUG
printf("Initialising position, bsu = %d, str = %d\n", i, j);
fflush(stdout);
#endif
td->single_iopos[i][j] = 0;
if (td->single_iocount[i][j] < 0) return 1;
#ifdef _SPC1_DEBUG
printf("Checking max size, bsu = %d, str = %d\n", i, j);
fflush(stdout);
// printf("for %d positions\n", td->single_iocount[i][j]);
// fflush(stdout);
#endif
for (k=0; k<td->single_iocount[i][j]; k++) {
int rw = td->single_iostore[i][j][k].dir;
int bytes = BLOCK_TO_B(td->single_iostore[i][j][k].len);
#ifdef _SPC1_DEBUG
printf("IO from bsu = %d, str = %d, pos = %d has size %d\n", i, j, k, bytes);
#endif
if (bytes > td->o.max_bs[rw])
td->o.max_bs[rw] = bytes;
}
}
}
#ifdef _SPC1_DEBUG
printf("Adding ASU1 file...\n");
#endif
fileno = get_fileno(td, ASU1);
if (fileno == -1) {
td->o.nr_files++;
fileno = add_file(td, ASU1);
}
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("ASU1 file could not be added...\n");
#endif
return 1;
}
if (td_io_open_file(td, td->files[fileno])) {
#ifdef _SPC1_DEBUG
printf("ASU1 file could not be opened...\n");
#endif
return 1;
}
#ifdef _SPC1_DEBUG
printf("Adding ASU2 file...\n");
#endif
fileno = get_fileno(td, ASU2);
if (fileno == -1) {
td->o.nr_files++;
fileno = add_file(td, ASU2);
}
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("ASU2 file could not be added...\n");
#endif
return 1;
}
if (td_io_open_file(td, td->files[fileno])) {
#ifdef _SPC1_DEBUG
printf("ASU2 file could not be opened...\n");
#endif
return 1;
}
#ifdef _SPC1_DEBUG
printf("Adding ASU3 file...\n");
#endif
fileno = get_fileno(td, ASU3);
if (fileno == -1) {
td->o.nr_files++;
fileno = add_file(td, ASU3);
}
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("ASU3 file could not be added...\n");
#endif
return 1;
}
if (td_io_open_file(td, td->files[fileno])) {
#ifdef _SPC1_DEBUG
printf("ASU3 file could not be opened...\n");
#endif
return 1;
}
#else
if (td->iocount <= 0) return 1;
td->iopos = 0;
// Find the maximum block size for creating buffers
for (i=0; i<td->iocount; i++) {
int rw = td->iostore[i].dir;
int bytes = BLOCK_TO_B(td->iostore[i].len);
if (bytes > td->o.max_bs[rw])
td->o.max_bs[rw] = bytes;
}
#endif
#ifdef _SPC1_DEBUG
printf("Initialisation finished for bsu = %d, str = %d, pid = %d\n", td->bsu, td->str, getpid());
fflush(stdout);
#endif
return 0;
}
int fin_spc1_io(struct thread_data *td) {
int fileno;
if ( (td->bsu >= BSU) || (td->str >= STREAMS) ) return 1;
#ifdef SINGLE
fileno = get_fileno(td, ASU1);
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("ASU1 file not open when closing...\n");
#endif
return 1;
}
td_io_close_file(td, td->files[fileno]);
fileno = get_fileno(td, ASU2);
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("ASU2 file not open when closing...\n");
#endif
return 1;
}
td_io_close_file(td, td->files[fileno]);
fileno = get_fileno(td, ASU3);
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("ASU3 file not open when closing...\n");
#endif
return 1;
}
td_io_close_file(td, td->files[fileno]);
#else
if (td->iocount <= 0) return 1;
#endif
#ifdef _SPC1_DEBUG
printf("Finalisation finished for bsu = %d, str = %d, pid = %d\n", td->bsu, td->str, getpid());
fflush(stdout);
#endif
return 0;
}
int get_spc1_io(struct thread_data *td, struct io_u *io_u_addr) {
struct spc1_io_s spc1_io;
int i, j, bsu, str, pos, fileno;
unsigned long elapsed, whenMillis, least;
unsigned short found;
#ifdef _SPC1_DEBUG
printf("Reading from SPC-1 IOs, pid = %d\n", getpid());
fflush(stdout);
#endif
#ifdef SINGLE
least = TOTAL_RUNTIME_MILLIS;
found = 0;
for (i=0; i<BSU; i++)
if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
(spc1_context_from_bsu(i) <= END_CONTEXT) ) {
for (j=0; j<STREAMS; j++) {
#ifdef _SPC1_DEBUG
printf("single_iocount[%d][%d] = %d, single_iopos[%d][%d] = %d\n",
i, j, td->single_iocount[i][j],
i, j, td->single_iopos[i][j]
);
#endif
if (td->single_iopos[i][j] < td->single_iocount[i][j])
if (td->single_iostore[i][j][td->single_iopos[i][j]].when / 10 < least) {
bsu = i;
str = j;
pos = td->single_iopos[i][j];
#ifdef _SPC1_DEBUG
printf("Earlier SPC-1 IOs, bsu = %d, str = %d, pos = %d, pid = %d\n", bsu, str, pos, getpid());
fflush(stdout);
#endif
least = td->single_iostore[bsu][str][pos].when / 10;
found = 1;
}
}
}
if (!found) {
td->done = 1;
return 1;
}
spc1_io = td->single_iostore[bsu][str][pos];
#else
bsu = td->bsu;
str = td->str;
pos = td->iopos;
if (pos == td->iocount) {
printf("IOs exhausted... bsu = %d, str = %d, pid = %d\n", bsu, str, getpid());
/* No more IOs from this (bsu, stream) */
td->done = 1;
return 1;
}
#ifdef _SPC1_DEBUG
printf("IO # %d from bsu = %d, str = %d, pid = %d\n", pos, bsu, str, getpid());
#endif
fflush(stdout);
spc1_io = td->iostore[pos];
#endif
#ifdef _SPC1_DEBUG
spc1_io_debug_info("SPC-1 IOP", &spc1_io);
fflush(stdout);
#endif
elapsed = mtime_since_genesis();
whenMillis = spc1_io.when / 10;
#ifdef _SPC1_DEBUG
printf("SPC-1 IO: elapsed = %ld, when = %ld, pid = %d\n", elapsed, whenMillis, getpid());
fflush(stdout);
#endif
if (whenMillis > elapsed) {
usec_sleep(td, (whenMillis - elapsed) * 1000);
}
#ifdef _SPC1_DEBUG
printf("SPC-1 IO: wait over\n");
fflush(stdout);
#endif
io_u_addr->ddir = spc1_io.dir;
io_u_addr->offset = spc1_io.pos;
// io_u_addr->offset = BLOCK_TO_B(spc1_io.pos);
io_u_addr->buflen = BLOCK_TO_B(spc1_io.len);
#ifdef _SPC1_DEBUG
printf("IO # %d from bsu = %d, str = %d, pid = %d, identifying file...\n", pos, bsu, str, getpid());
fflush(stdout);
#endif
/* Get file to read from/write to */
switch (td->str) {
case 0: case 1: case 2: case 3:
fileno = get_fileno(td, ASU1);
if (fileno == -1) {
td->o.nr_files++;
fileno = add_file(td, ASU1);
}
break;
case 4: case 5: case 6:
fileno = get_fileno(td, ASU2);
if (fileno == -1) {
td->o.nr_files++;
fileno = add_file(td, ASU2);
}
break;
default: /* case 7: */
fileno = get_fileno(td, ASU3);
if (fileno == -1) {
td->o.nr_files++;
fileno = add_file(td, ASU3);
}
break;
}
if (fileno == -1) {
#ifdef _SPC1_DEBUG
printf("IO file has not been added... bsu = %d, str = %d, pid = %d\n", bsu, str, getpid());
#endif
return 1;
}
io_u_addr->file = td->files[fileno];
#ifndef SINGLE
if (pos == 0) {
#ifdef _SPC1_DEBUG
printf("IO # %d from bsu = %d, str = %d, pid = %d, opening file # %d...\n", pos, bsu, str, getpid(), fileno);
#endif
if (td_io_open_file(td, io_u_addr->file)) {
#ifdef _SPC1_DEBUG
printf("IO file could not be opened... bsu = %d, str = %d, pid = %d\n", bsu, str, getpid());
#endif
return 1;
}
}
if (pos == td->iocount) {
#ifdef _SPC1_DEBUG
printf("IO # %d from bsu = %d, str = %d, pid = %d, closing file # %d...\n", pos, bsu, str, getpid(), fileno);
#endif
td_io_close_file(td, io_u_addr->file);
}
#endif
#ifdef _SPC1_DEBUG
printf("IO # %d from bsu = %d, str = %d, pid = %d, getting file %d...\n", pos, bsu, str, getpid(), io_u_addr->file);
fflush(stdout);
#endif
get_file(io_u_addr->file);
#ifdef _SPC1_DEBUG
fio_io_debug_info("SPC-1 io_u", io_u_addr);
fflush(stdout);
#endif
#ifdef SINGLE
td->single_iopos[bsu][str]++;
#else
td->iopos++;
#endif
#ifdef _SPC1_DEBUG
printf("Finished reading IO # %d from bsu = %d, str = %d, pid = %d\n", pos, bsu, str, getpid());
fflush(stdout);
#endif
return 0;
}
unsigned short spc1_ios_left(struct thread_data *td) {
unsigned short left = 0;
int i, j;
for (i=0; i<BSU; i++)
if ( (spc1_context_from_bsu(i) >= BEGIN_CONTEXT) &&
(spc1_context_from_bsu(i) <= END_CONTEXT) ) {
for (j=0; j<STREAMS; j++)
if (td->single_iopos[i][j] < td->single_iocount[i][j]) {
left = 1;
break;
}
if (left) break;
}
return left;
}
#endif
