On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:
> Ævar Arnfjörð Bjarmason writes:
> > On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@xxxxxxxxx> wrote:
> >
> > > Currenly the survey has more than 5000 responses (in a not whole month)
> >
> > That seems somewhat low, but maybe people just aren't that interested
> > in taking surveys.
>
> Nice work on the survey! This is our best year by far. Some general
> observations:
>
> Interesting statistic: 24508 people viewed it, 7821 people completed
> it, but 0 people started filling out information and later decided not
> to submit it. It could mean that many people clicked through and found
> the survey, but probably left because it looked too long at a glance?
Without the knowledge how those two numbers are calculated we can only
speculate what do they mean.
I think that '0' in 'Incomplete' statistics is here because this survey
doesn't have compulsory questions: answering all questions are optional,
so it might mean that even if one question in the survey is answered,
then the survey is considered complete by Survs.com statistics.
I personally do not like the "wizard" formatting of surveys, i.e.
dividing survey into page so you are not presented with very long page,
but are presented withc chunks of survey at glance. Even if you see
how much survey did you fill in (how many pages there are in total),
and even if you can go back to previous page.
I'd prefer to create a better information about survey upfront. We say
that all questions are optional ("Note that you may skip questions as
you like"). It is also stated that you can fill only a part of survey,
and later go back to finish it... hmmm, I wonder if those cases where
one edited his/her survey responses multiple times are counted as one
finished survey, but multiple views.
We could also write how much time it takes on average to fil the survey.
> The average time spent on the survery is 34 minutes
It would be interesting to have more detailed statistics of time spent
on the survey that only average time, a single number. When one is
filling open-form essay-length question, it would obviously take much
more time than for one who doesn't.
But Survs.com currently doesn't provide it. I can try to ask for it,
though (via feedback).
> - I think we can bring that down to 10~15 minutes if we design
> questions to extract more information. Also, there's little incentive
> for taking the survey: while many companies actually give out
> discounts/ coupons for taking surveys, the least we can do is present
> real-time results in the most interesting manner possible ie. survey
> takers should see the "results so far" immediately after taking the
> survey; some visualizations such as pie charts?
What we can do is after finishing the survey to redirect to the
survey analysis page:
https://www.survs.com/results/33Q0OZZE/MV653KSPI2
http://tinyurl.com/GitSurvey2010Analysis
instead of IIRC currently used redirect to
https://git.wiki.kernel.org/index.php/GitSurvey2010
As far as I know Survs.com doesn't provide any API for extracting data
or survey statistics required for creating such visualization. Neither
we have a place where such app could be created, I think.
>
> In questions 5, 10, 12, 13, 16, cut down out the options that have
> very few respondents and let them all go into "other". It probably
> doesn't actually save the survey taker any time, but I think seeing a
> long page with many options can be scary.
The "5. Which Git version(s) are you using?" is not that long. We could
create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or
even earlier, we could remove alternate implementations answers
(git-bigfiles, JGit, other implementations), or even concatenate 'master',
'next', 'pu' into single response... but would it buy us much?
The other side of removing choices, relying instead on "other, please
specify" response is that it makes it harder to analyze results of survey:
different people use different words for the same thing (and there are
also spelling mistakes), and make results less reliable: people do not
fill "other" if there is at least partial match, or do not know how to
specify their version.
In "10. What Git interfaces, implementations and frontends do you use?"
we can remove tools marked as deprecated... if not for the fact that it
is actively interesting to know how many people use such deprecated
tools. Besides, the list of answers to this question is not overly long,
I don't think.
We can remove those choices in "12. What Git GUIs (graphical user
interfaces) do you use?" that got less than 1% rounded, or less than
10 responses. On the other hand some people stated earlier that the
list of possible choices in the survey (not necessarily about this
question in specific) serve as reminder / information about possible
choices.
The other side of removing options from "13. Which git hosting site(s)
do you use for your project(s)?" is that when sending requests to
announce the survey to those git hosting sites that are not on this
list, some of them requested to be added (which is impossible after
starting the survey; and before survey begins it is little sense to
send announcements).
Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
CipherHive) are also those that I didn't get response to request for
announcing Git User's Survey 2010...
We could make it more organized though, e.g. by sorting list of options
alphabetically, or something like that.
Removing options from "16. Which of the following features do you use?"
would make it harder to analyze and less reliable. Especially in this
question different people consider different features important enough
to mention, and describe feature in many different ways.
Besides, each option except of "git cvsserver" got more than 1% rounded,
and having "git cvsserver" is interesting on its own (perhaps in other
question?).
> 1. Country of residence: we can probably make this a nice click-on-map
> interface as opposed to freeform text. It'll be more useful to us,
> and more interesting to users when we advertise the results.
It would be nice to have click-on-map (Google Maps or Bing Maps based),
something like Ohloh provides, resulting in map of survey responders
similar to the map of git users and git contributors on Ohloh
http://www.ohloh.net/p/git/map
it isn't something that Survs.com offers currently. I can only ask for
it to be provided...
Another solution would be to have pre-filled combo box (<select> field)
with the list of countries to choose from, with GeoIP ised to pre-select
the country. I can generate list of all countries myself
$ perl -MLocale::Country \
-wle 'print join("\n", sort (all_country_names()))'
as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
to survey questions.
> 2. Age: Maybe we restrict the input to 2-digit integers and draw a
> graph with all these integers to show a mean, median etc?
Restricting input doesn't give us much.
There is nice histogram of responder's age for Git User's Survey 2009
https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F
and tabularization of responses. We can calculate mean, median, mode
(aka modal score, i.e. most common response), perhaps after eliminating
outliers, but would it give us much information?
> 11. Just change this to an optional sometimes/ often? Why should users
> spend time clicking on "never"?
"Never" is here because you can't un-click response in given row.
I'm also not sure how not answering row is represented in the export
of survey data (which we use for further analysis).
> 17, 18: Merge perhaps?
Those questions are split because of limitation of Survs.com; the
"other, please specify" results in limited width _text field_, while
it is much easier to write in-depth response in large _textarea_ field
that wuestion 18 provides.
> 24, 25: Merge into single question with options: "Yes, and my problems
> are solved more often than not", "Yes, but my problems often
> remain unresolved", "No, I don't go to others for help".
Good idea. Will do.
> 26, 27: Merge into "How do you talk to other people using Git, either
> for technical help or otherwise?"
I don't think it is good idea. Those two issues, namely requesting
help and being (perhaps silent) part of git community are two unrelated
issues.
> Ofcourse, I understand that there must be some technical constraints
> due to which some things are not implementable (eg. survs doesn't
> provide the feature?), but I've not taken that into consideration.
Note that as it currently stands we can use Survs.com account only for
2011 survey, provided that it is done earlier than this year (perhaps
1 June -- 31 July?), as our Premium account which we got thanks to
generosity of Survs.com admins (after Survs.com got out of beta) will
downgrade to the Free plan (which is offersn much too low limits)
on Sep 22, 2011.
--
Jakub Narebski
Poland
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html