Tag Archives: semester report

The semester report for late 2015

Tree graphics from the second half of the yearThis site looks at locations’ TripAdvisor performances from time to time; it’s interesting to reflect on how far the industry has come. The Timeline shows that the number of known open exit games in the UK doubled (or, on every occasion but one, more than doubled) over the first half of 2013, the second half of 2013, the first half of 2014, the second half of 2014 and the first half of 2015. (Those numbers: 1 to 3 to 7 to 14 to 30 to 64.) Over the second half of 2015, the number did not double again but rose from 64 to 95. Maybe it’s only 94. It’s in the very high double figures, at least.

It’s worth occasionally looking at trends in popular reviews of exit games in the UK and Ireland, taken from TripAdvisor statistics. This site is using a little more reserve than once it did with regards to what it says, bearing TripAdvisor’s terms of use in mind; the aim is not to laud or criticise particular sites in this regard, more to look at the bigger picture. Besides, if you run a site and care about your performance in this regard, it’s probably not difficult to work out which site is which from context. As usual, there’s more than a hint of truth in xkcd’s snark about online star ratings; in this world, anything other than full marks (and, especially if you’re on eBay, several pluses and stars) is a “diss that don’t miss”. It’s not necessarily a healthy state of affairs for anyone who cares about subtlety, graduation and shades of light and dark – but, with this in mind, are five-star ratings quite as common as they used to be?

Here’s some raw data, aggregated over the universe of TripAdvisor reviews for exit games in the UK and Ireland that this site was able to find.

Time period Number of reviews Number of 5* reviews Proportion of 5% reviews
To end of June 2014 1665 1532 92%
Second half of 2014 2240 1998 89%
First half of 2015 4248 3900 92%
Second half of 2015 6697 6127 91%

From these figures, there has been no measured change in quality over the last six months; a z-test does not suggest any even slightly meaningful degree of significance in the change from the first half of 2015 to the second. These statistics make the considerable (and untestable) assumption that the standard required for a 5* review is the same as it ever was.

It may be closer to comparing like with like to only consider the 52 sites that have been open since before January 2015, where we have meaningful numbers of reviews (10+, and even that’s a stretch) for H1 2015 and for H2 2015.

Site location First half of 2014 Second half of 2015
  Reviews 5% reviews Prop’n 5% Reviews 5% reviews Prop’n 5%
Scotland 66 52 79% 115 95 83%
South 79 70 89% 75 65 87%
Ireland 24 18 75% 20 16 80%
Midlands 11 1 9% 15 0 0%
Midlands 109 108 99% 113 107 95%
North 84 82 98% 69 64 93%
South 22 21 95% 23 15 65%
South 64 60 94% 92 88 96%
Scotland 203 167 82% 103 82 80%
Scotland 103 98 95% 98 96 98%
Scotland 119 113 95% 163 151 93%
Scotland 66 64 97% 148 142 96%
Scotland 122 109 89% 106 90 85%
Scotland 14 12 86% 35 30 86%
Scotland 41 41 100% 111 109 98%
North 36 35 97% 23 23 100%
Scotland 11 11 100% 48 45 94%
North 200 194 97% 176 163 93%
North 156 150 96% 93 86 92%
North 80 80 100% 95 94 99%
North 72 66 92% 104 97 93%
North 28 27 96% 23 19 83%
North 40 37 93% 147 132 90%
North 21 18 86% 89 77 87%
London 189 178 94% 177 163 92%
London 220 207 94% 235 213 91%
London 71 54 76% 62 47 76%
London 23 22 96% 23 23 100%
London 20 19 95% 35 33 94%
London 59 50 85% 115 97 84%
London 31 29 94% 128 114 89%
London 62 60 97% 181 178 98%
London 75 71 95% 202 191 95%
North 31 30 97% 32 32 100%
North 172 155 90% 211 184 87%
North 76 69 91% 258 216 84%
North 190 169 89% 254 195 77%
Midlands 13 13 100% 15 15 100%
North 41 37 90% 37 31 84%
North 65 62 95% 38 29 76%
North 41 41 100% 59 57 97%
Midlands 82 76 93% 148 137 93%
Midlands 46 46 100% 162 158 98%
Midlands 27 20 74% 46 38 83%
South 121 112 93% 104 91 88%
South 71 70 99% 74 74 100%
North 141 134 95% 185 161 87%
North 39 37 95% 34 33 97%
North 210 203 97% 198 190 96%
North 95 90 95% 197 186 94%
Ireland 36 36 100% 85 85 100%
Ireland 22 20 91% 53 48 91%

The first column is classified as Scotland, Ireland, London, and (referring to different parts of provincial England) North, Midlands and South. Be aware that Ireland refers to both Northern Ireland and Ireland as such; this is poor practice that the site does not usually follow, but the alternative would be to let at least one site be individually identifiable, which this site considers to be an even worse alternative. There is further ordering in the table which this site chooses not to make explicit but is not hard to deduce. (If you run a site and can’t work out which site you are, you could always ask.)

So, only among these 52 popular and well-established sites:

Time period Number of reviews Number of 5* reviews Proportion of 5% reviews
First half of 2015 4040 3744 93%
Second half of 2015 5432 4905 90%

That is a statistically highly significant difference over the whole of the population and some individual sites have similarly statistically significant differences as well. Does this mean that those sites have got worse over time? Not necessarily; it may just mean that people are holding well-established sites to an even higher standard still to the one that once they did. Run your own tests!

There’s one other comparison worth running:

Time period Number of reviews Number of 5* reviews Proportion of 5% reviews
52 well-established sites 5432 4905 90%
The other 40+ sites 1265 1222 97%

That too is a statistically highly significant difference. Does this mean that new sites are better than well-established sites? That could be one conclusion. Does this mean that people are holding these less well-established sites to a less exacting standard? That could be another. Certainly some new sites have got off to remarkably accomplished starts. This is excellent news and this site looks forward to seeing whether they can maintain their remarkably high ratings over time. Every site was a less well-established site before it became a well-established one.

There is one very important assumption that this analysis makes: that the reviews that people leave are a genuinely representative sample of participants. Different sites seem to perform more or less effectively at converting participants into reviewers and it is not clear why. Looking at the geographic locations of reviewers, it’s also sometimes possible that more than one member of the same team might choose to leave a review for some games, though there’s not necessarily anything wrong with that; it’s conceivable that different members of one group might leave – say – both a ***** review and a *** review, rather than the group leaving a single **** review. It’s not unknown, in the wider world at large, for there to be such things as fake reviews; this site isn’t aware of it having happened in the exit games it covers, but this site has chosen not to look too hard.

In conclusion: in aggregate, the statistics suggest that exit games were awesome up until the first half of 2015, and have been just as awesome in the second half of 2015 as well.

The semester report for early 2015

A book displaying "Semester Readings"The single months’ worth of TripAdvisor that are tracked in the League Table feature are only really meaningful as snapshots in time. However, with sufficiently many of them, it is possible to draw slightly more meaningful trends – or, at least, to reflect on how far the industry has come. The Timeline shows that the number of known open exit games in the UK more than doubled over the first half of 2013, more than doubled again over the second half of 2013, doubled over the first half of 2014, more than doubled once more over the second half of 2014 and more than doubled once more still over the first half of 2015. (Those numbers: 1 to 3 to 7 to 14 to 30 to 64.) Past performance is not an indicator of future results, as you may have previously been told, which is just as well or the metaphorical king’s chessboard will become swamped with rice. Already the latest square is looking alarmingly full.

It’s worth occasionally looking at trends in popular reviews of exit games in the UK and Ireland, taken from TripAdvisor statistics. This site is using a little more reserve than once it did with regards to what it says, bearing TripAdvisor’s terms of use in mind; the aim is not to laud or criticise particular sites in this regard, more to look at the bigger picture. Besides, if you run a site and care about your performance in this regard, it’s probably not difficult to work out which site is which from context. As usual, there’s more than a hint of truth in xkcd’s snark about online star ratings; in this world, anything other than full marks (and, especially if you’re on eBay, several pluses and stars) is a “diss that don’t miss”. It’s not necessarily a healthy state of affairs for anyone who cares about subtlety, graduation and shades of light and dark – but, with this in mind, are five-star ratings quite as common as they used to be?

Here’s some raw data, aggregated over the universe of TripAdvisor reviews for exit games in the UK and Ireland that this site was able to find.

Time period Number of reviews Number of 5* reviews Proportion of 5% reviews
To end of June 2014 1665 1532 92%
Second half of 2014 2240 1998 89%
First half of 2015 4248 3900 92%

From these figures, it would be tempting to suggest that the market has increased in quality over the last six months. Indeed, using a z-test, it is statistically significant at a highly meaningful (0.01) level that the proportion of 5* reviews has increased from H2 2014 to H1 2015, but there is no statistical significance in the changes between reviews from “H1 2014 and before” and reviews from H1 2015. These are not quite comparable statistics to the ones that this site presented six months ago; these reflect reviews of all exit games open at that time, rather than comparing like with like directly. They also make the considerable (and untestable) assumption that the standard required for a 5* review is the same as it ever was. An explanation may be as simple as people enjoy playing (mostly indoor) exit games more when they do so in the relatively cold first half of the year than when they do so in the relatively warm second half of the year.

It may be closer to comparing like with like to only consider the 22 sites that have been open since before July 2014, where we have meaningful numbers of reviews (10+, and even that’s a stretch) for H2 2014 and for H1 2015.

Site location Second half of 2014 First half of 2015
  Reviews 5% reviews Prop’n 5% Reviews 5% reviews Prop’n 5%
South 49 41

84% 79 70 89%
N.I. 28 16

57% 24 18 75%
N.I. 28 24

86% 15 13 87%
Midlands 16 5

31% 11 1 9%
South 53 50

94% 64 60 94%
Scotland 226 204

90% 203 167 82%
Scotland 36 35

97% 103 98 95%
Scotland 21 21

100% 119 113 95%
Scotland 68 66

97% 122 109 89%
North 221 210

95% 200 194 97%
North 66 63

95% 156 150 96%
London 292 270

92% 189 178 94%
London 381 348

91% 220 207 94%
London 80 68

85% 37 30 81%
London 92 70

76% 71 54 76%
London 20 19

95% 23 22 96%
London 61 48

79% 104 73 70%
North 14 14

100% 31 30 97%
North 150 133

89% 172 155 90%
Midlands 32 21

66% 27 20 74%
North 232 217

93% 210 203 97%
North 16 16

100% 95 90 95%

The first column is classified as Scotland, Northern Ireland, London, and provincial England is split roughly into North, Midlands and South. There is further ordering in the table which this site chooses not to make explicit but is not hard to deduce. (If you run a site and can’t work out which site you are, you could always ask.)

So, only among these 22 popular and well-established sites:

Time period Number of reviews Number of 5* reviews Proportion of 5% reviews
Second half of 2014 2182 1959 90%
First half of 2015 2275 2055 90%

Practically unchanged – though, beyond the decimal point, a rise from “just under 90%” to “just over 90%”. If one particular site had been getting 100 reviews with 90% 5/5s previous to one half-year and then another 100 reviews with only 75% 5/5s in a half-year then there might be cause for alarm, but the sample sizes here are generally so small that there are only one or two cases in which the observed lowering of the percentage for a particular site is at all meaningfully significant. Run your own tests!

There is one very important assumption that this analysis makes, that the reviews that people leave are a genuinely representative sample of participants. Different sites seem to perform more or less effectively at converting participants into reviewers and it is not clear why. Looking at the geographic locations of reviewers, it’s also sometimes possible that more than one member of the same team might choose to leave a review for some games, though there’s not necessarily anything wrong with that; it’s conceivable that different members of one group might leave – say – both a ***** review and a *** review, rather than the group leaving a single **** review. It’s not unknown, in the wider world at large, for there to be such things as fake reviews; this site isn’t aware of it having happened in the exit games it covers, but it’s not as if it has performed meaningful investigative journalism in this regard.

In conclusion: exit games were awesome up until the second half of 2014, and have been just as awesome in the first half of 2015 as well.

The semester report for late 2014

Calendar for second half of 2014This site was looking for an image of half a calendar, covering only the last six months, and thus searched for JASOND. Perhaps it shouldn’t have come as a surprise to get pictures of a R&B singer-songwriter…

The single months’ worth of TripAdvisor that are tracked in the League Table feature are only really meaningful as snapshots in time. However, with sufficiently many of them, it is possible to draw slightly more meaningful trends – or, at least, to reflect on how far the industry has come. The Timeline shows that the number of open exit games in the UK more than doubled over the first half of 2013, more than doubled again over the second half of 2013, doubled over the first half of 2014 and more than doubled once more over the second half of 2014. (Those numbers: 1 to 3 to 7 to 14 to 30.) Past performance is not an indicator of future results, as you may have previously been told, which is just as well or the metaphorical king’s chessboard will become swamped with rice.

So, for the sites that have stood the test of time, it’s worth looking back to where they were six months ago and compare that to how they are doing now. Here’s the situation from six months ago:

Site name Number of exit rooms Number of different games Number of TripAdvisor reviews Number of 5/5 TripAdvisor reviews Proportion of 5/5 ratings
Breakout Manchester 2 2 30 28 97%
Cipher 1 1 – – –
Clue HQ 1 1 1 1 (*)
clueQuest 3 2 493 472 96%
Cryptopia 1 1 25 22 88%
Escape 2 2 29 29 100%
Escape Land 1 1 1 1 (*)
ESCAP3D 1 1 81 68 84%
Ex(c)iting Game 2 2 21 15 71%
HintHunt 5 2 846 783 93%
Keyhunter 3 3 39 25 64%
Make A Break 1 1 25 16 64%
Puzzlair 2 2 45 44 98%
Tick Tock Unlock 1 1 25 25 100%
XIT 4 4 4 3 (*)

Across the population, that’s 1,532 5/5 reviews out of 1,665, a proportion of 92%. The (*) symbol represents a population of fewer than ten reviews, where a proportion is not particularly meaningful. Now let’s consider the same sites again, but only those reviews posted in the second half of 2014:

Site name Number of exit rooms Number of different games Number of recent TripAdvisor reviews Number of recent 5/5 TripAdvisor reviews Proportion of recent 5/5 ratings
Breakout Manchester 4 4 150 133 89%
Cipher 1 1 – – –
Clue HQ 2 2 232 217 94%
clueQuest 5 2 381 348 91%
Cryptopia 1 1 1 1 (*)
Escape (Edinburgh) 3 2 226 204 90%
Escape Land 1 1 80 68 85%
ESCAP3D (Belfast) 1 1 28 16 57%
Ex(c)iting Game 2 2 32 21 66%
HintHunt 5 2 292 270 92%
Keyhunter 3 3 16 5 31%
Make A Break 1 1 25 16 64%
Puzzlair 2 2 53 50 94%
Tick Tock Unlock 1 1 221 210 95%
XIT 4 4 4 2 (*)

Across the population, that’s 1,578 5/5 reviews out of 1,741, a proportion of 90½%. Again, the (*) symbol represents a population of fewer than ten reviews, where a proportion is not particularly meaningful.

There is nothing to suggest that the market has decreased in quality over the last six months; surely the opposite is true. It might be tempting to suggest that the novelty of exit games among the reviewing has faded, resulting in a fall of 5/5 scores across the population from 92% to 90½%, but that difference is so small that it might well arise through the noise of random chance and is not statistically significant to a particularly meaningful level. (OK, a χ2 test suggests it might be approaching significance, but if you accept that level of significance then you’re going to go on a lot of wild goose chases.)

If one particular site had been getting 100 reviews with 90% 5/5s previous to one half-year and then another 100 reviews with only 75% 5/5s in a half-year then there might be cause for alarm, but the sample sizes here are generally so small that there are only a very few cases in which the observed lowering of the percentage for a particular site is at all meaningfully significant. Run your own tests!

In conclusion: exit games were awesome up until the first half of 2014, and have been just as awesome in the second half of 2014 as well.