Hi Jakub,
Thanks for taking the time to go through my long (somewhat
unstructured) note.
Jakub Narebski writes:
> On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:
> > Interesting statistic: 24508 people viewed it, 7821 people completed
> > it, but 0 people started filling out information and later decided not
> > to submit it. It could mean that many people clicked through and found
> > the survey, but probably left because it looked too long at a glance?
>
> Without the knowledge how those two numbers are calculated we can only
> speculate what do they mean.
>
> I think that '0' in 'Incomplete' statistics is here because this survey
> doesn't have compulsory questions: answering all questions are optional,
> so it might mean that even if one question in the survey is answered,
> then the survey is considered complete by Survs.com statistics.
I just opened up survs.com, signed up for a free account and created a
survey with two compulsory questions- I couldn't submit the results
without filling up both of them. Then I looked at the statistics and I
saw a '0' in incomplete- I'm confused. Perhaps we can email survs.com
and ask them?
> I personally do not like the "wizard" formatting of surveys, i.e.
> dividing survey into page so you are not presented with very long page,
> but are presented withc chunks of survey at glance. Even if you see
> how much survey did you fill in (how many pages there are in total),
> and even if you can go back to previous page.
Same. I also think "wizard" is a bad idea for this- it would make it
look like some sort of online test :p
> I'd prefer to create a better information about survey upfront. We say
> that all questions are optional ("Note that you may skip questions as
> you like"). It is also stated that you can fill only a part of survey,
> and later go back to finish it... hmmm, I wonder if those cases where
> one edited his/her survey responses multiple times are counted as one
> finished survey, but multiple views.
Sure. Again, we should perhaps email survs.com for such
clarifications.
> We could also write how much time it takes on average to fil the survey.
Sounds good.
> > The average time spent on the survery is 34 minutes
>
> It would be interesting to have more detailed statistics of time spent
> on the survey that only average time, a single number. When one is
> filling open-form essay-length question, it would obviously take much
> more time than for one who doesn't.
Hm, yes. I didn't think about this.
> > - I think we can bring that down to 10~15 minutes if we design
> > questions to extract more information. Also, there's little incentive
> > for taking the survey: while many companies actually give out
> > discounts/ coupons for taking surveys, the least we can do is present
> > real-time results in the most interesting manner possible ie. survey
> > takers should see the "results so far" immediately after taking the
> > survey; some visualizations such as pie charts?
>
> What we can do is after finishing the survey to redirect to the
> survey analysis page:
>
> https://www.survs.com/results/33Q0OZZE/MV653KSPI2
> http://tinyurl.com/GitSurvey2010Analysis
>
> instead of IIRC currently used redirect to
>
> https://git.wiki.kernel.org/index.php/GitSurvey2010
>
>
> As far as I know Survs.com doesn't provide any API for extracting data
> or survey statistics required for creating such visualization. Neither
> we have a place where such app could be created, I think.
Oh, ok- we should take too much trouble though; we should keep
cost-to-benefit ratio in mind.
> > In questions 5, 10, 12, 13, 16, cut down out the options that have
> > very few respondents and let them all go into "other". It probably
> > doesn't actually save the survey taker any time, but I think seeing a
> > long page with many options can be scary.
>
> The "5. Which Git version(s) are you using?" is not that long. We could
> create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or
> even earlier, we could remove alternate implementations answers
> (git-bigfiles, JGit, other implementations), or even concatenate 'master',
> 'next', 'pu' into single response... but would it buy us much?
>
> The other side of removing choices, relying instead on "other, please
> specify" response is that it makes it harder to analyze results of survey:
> different people use different words for the same thing (and there are
> also spelling mistakes), and make results less reliable: people do not
> fill "other" if there is at least partial match, or do not know how to
> specify their version.
Ofcourse. We should only remove options that don't necessarily tell us
very much.
> We can remove those choices in "12. What Git GUIs (graphical user
> interfaces) do you use?" that got less than 1% rounded, or less than
> 10 responses. On the other hand some people stated earlier that the
> list of possible choices in the survey (not necessarily about this
> question in specific) serve as reminder / information about possible
> choices.
Hm, I hadn't thought of that. I suppose it can be used to advertise
various applications.
> The other side of removing options from "13. Which git hosting site(s)
> do you use for your project(s)?" is that when sending requests to
> announce the survey to those git hosting sites that are not on this
> list, some of them requested to be added (which is impossible after
> starting the survey; and before survey begins it is little sense to
> send announcements).
I see. How are we going to tackle this in future?
> Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
> CipherHive) are also those that I didn't get response to request for
> announcing Git User's Survey 2010...
Interesting.
> We could make it more organized though, e.g. by sorting list of options
> alphabetically, or something like that.
Sounds good.
> > 1. Country of residence: we can probably make this a nice click-on-map
> > interface as opposed to freeform text. It'll be more useful to us,
> > and more interesting to users when we advertise the results.
>
> It would be nice to have click-on-map (Google Maps or Bing Maps based),
> something like Ohloh provides, resulting in map of survey responders
> similar to the map of git users and git contributors on Ohloh
>
> http://www.ohloh.net/p/git/map
>
> it isn't something that Survs.com offers currently. I can only ask for
> it to be provided...
>
>
> Another solution would be to have pre-filled combo box (<select> field)
> with the list of countries to choose from, with GeoIP ised to pre-select
> the country. I can generate list of all countries myself
>
> $ perl -MLocale::Country \
> -wle 'print join("\n", sort (all_country_names()))'
>
> as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
> to survey questions.
I suppose we could always work out a way to display the results from
the information Survs.com gives us.
> > 2. Age: Maybe we restrict the input to 2-digit integers and draw a
> > graph with all these integers to show a mean, median etc?
>
> Restricting input doesn't give us much.
I just meant it as a sanity check in case people enter "34 years old"
and the like.
> There is nice histogram of responder's age for Git User's Survey 2009
>
> https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F
>
> and tabularization of responses. We can calculate mean, median, mode
> (aka modal score, i.e. most common response), perhaps after eliminating
> outliers, but would it give us much information?
Nice histogram! How did we manage to do this in 2009? Did we use a
custom-made application to do the survey?
> > 11. Just change this to an optional sometimes/ often? Why should users
> > spend time clicking on "never"?
>
> "Never" is here because you can't un-click response in given row.
> I'm also not sure how not answering row is represented in the export
> of survey data (which we use for further analysis).
Oh, right. I didn't think about this.
> > 17, 18: Merge perhaps?
>
> Those questions are split because of limitation of Survs.com; the
> "other, please specify" results in limited width _text field_, while
> it is much easier to write in-depth response in large _textarea_ field
> that wuestion 18 provides.
Ah, I thought so.
> > Ofcourse, I understand that there must be some technical constraints
> > due to which some things are not implementable (eg. survs doesn't
> > provide the feature?), but I've not taken that into consideration.
>
> Note that as it currently stands we can use Survs.com account only for
> 2011 survey, provided that it is done earlier than this year (perhaps
> 1 June -- 31 July?), as our Premium account which we got thanks to
> generosity of Survs.com admins (after Survs.com got out of beta) will
> downgrade to the Free plan (which is offersn much too low limits)
> on Sep 22, 2011.
I see. Any thoughts on long-term plans? Do we pay for the premium
account or do we build a custom application?
-- Ram
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html