Friday, 23 August 2013

Death Statistics in England and Wales - Postscript

In the last post about death statistics I put in an animated GIF that I had created using a combination of Excel and GIMP.

Quite why I did it this way I don't know. It took hours (probably) to adjust the chart, copy it to paint, save it and then copy it as a layer to GIMP.

I've since discovered a few websites detailing how to create exactly the same thing using R. Googling 'animated gif R' gives, almost, thousands of examples. I now realise quite how stupid I was.

Below is the script that I used:

-------------------------------------------------------------------------------
death <- read.table("death.txt",header=TRUE)
#summary(death)

library(ggplot2)
library(Cairo)

for(i in 2:49){
  dp <- ggplot(data=death,aes(x=Age,y=death[,i])) + geom_line() + ylim(0,0.05)
  
  if(i<10){
    ttle <- paste('plot0',i,'.png',sep="")
  } else {
    ttle <- paste('plot',i,'.png',sep="")
  }

  CairoPNG(filename=ttle,width=480,height=480,dpi=300)
  #png(filename=ttle)
  print(dp)
  dev.off()
}

shell('"C:/Program Files/GraphicsMagick/gm.exe" convert -delay 15 -loop 0 plot*.png death.gif')
shell('"C:/Program Files/FFMpeg/bin/ffmpeg.exe" -start_number 02 -i plot%02d.png -vcodec libx264 death.mp4')
-------------------------------------------------------------------------------

The script loads in the data which is arranged as a table with age as the rows, year as the columns and deaths as a proportion of the total.

It then loads the required libraries.

Lines 7-20 do the work of creating the plots. These are saved as PNG files. Two things to note:
  1. I've set the y axis scale to run from 0->0.05. This is to keep the axes on a consistent scale.
  2. To make sure that the PNG files are ordered correctly I've had to zero pad the numbers for 2-9.
The work of creating the animated GIF is done by GraphicMagick. You can also use ImageMagick.

I won't go into any more detail as other sites give much better information. There's no point copying it.

The last line uses ffmeg to create a movie from the plots. This was inspired by www.animatedgraphs.co.uk.

Below is the resultant GIF:

The whole process takes seconds - once the script is written.

Thursday, 8 August 2013

An Excel macro to simplistically seasonally adjust time series data

Introduction

There are many ways to deseasonalise seasonal time series data.

I'm going to write about one of the easier methods. I can't remember what the official name for the method is but all we're doing is dividing the data point by the average for each seasonal period.

That definition probably needs explaining. So, say, for example we have quarterly data for 3 years. We take the average for each quarter over the years i.e. average Q1 for years 1, 2 and 3. Then do the same for Q2, Q3 and Q4. After this you divide the Q1 points by the Q1 average, the Q2 points by the Q2 average, etc.

There are some things to be careful of when using this method:

  1. The more data you have the better. You'll probably need more than 3 years worth of data for this to work well.
  2. You need to be careful of outliers affecting the seasonal averages by too much when there is very little data.
  3. The data series needs to be stationary. Or at least, mostly stationary. If it's not, you tend to get step changes in the resulting seasonally adjusted data. By stationary I mean that there is no long term trend in the data.

The Macro

Here is the listing of the macro.

--------------------------------------------------------------------------
Function SeasonAverage(labelRange As range, dataRange As range) As Variant

    Dim labelSum As Dictionary
    Dim labelCount As Dictionary
    Dim labelAvg As Dictionary
    Dim totalSum As Double
    Dim totalCount As Integer
    Dim OutputRows As Long
    Dim OutputCols As Long
    Dim output() As Double
    Dim i As Integer
    
    totalSum = 0
    totalCount = 0
    
    Set labelSum = New Dictionary
    Set labelCount = New Dictionary
    Set labelAvg = New Dictionary
    
    With Application.Caller
        OutputRows = .Rows.Count
        OutputCols = .Columns.Count
    End With
    ReDim output(1 To OutputRows, 1 To OutputCols)
    
    'Is dataRange, labelRange and outputRange the same size and shape
    If dataRange.Rows.Count = labelRange.Rows.Count & dataRange.Columns.Count = labelRange.Columns.Count & dataRange.Rows.Count = OutputRows & dataRange.Columns.Count = OutputCols Then
        Exit Function
    End If
    
    If labelRange.Rows.Count < labelRange.Columns.Count Then
        For i = 1 To labelRange.Columns.Count
            If labelSum.Exists(labelRange(1, i).Value) Then
                labelSum.Item(labelRange(1, i).Value) = labelSum.Item(labelRange(1, i).Value) + dataRange(1, i).Value
            Else
                labelSum.Add Key:=labelRange(1, i).Value, Item:=dataRange(1, i).Value
            End If
            If labelCount.Exists(labelRange(1, i).Value) Then
                labelCount.Item(labelRange(1, i).Value) = labelCount.Item(labelRange(1, i).Value) + 1
            Else
                labelCount.Add Key:=labelRange(1, i).Value, Item:=1
            End If
            totalSum = totalSum + dataRange(1, i).Value
            totalCount = totalCount + 1
        Next
    Else
        For i = 1 To labelRange.Rows.Count
            If labelSum.Exists(labelRange(i, 1).Value) Then
                labelSum.Item(labelRange(i, 1).Value) = labelSum.Item(labelRange(i, 1).Value) + dataRange(i, 1).Value
            Else
                labelSum.Add Key:=labelRange(i, 1).Value, Item:=dataRange(i, 1).Value
            End If
            If labelCount.Exists(labelRange(i, 1).Value) Then
                labelCount.Item(labelRange(i, 1).Value) = labelCount.Item(labelRange(i, 1).Value) + 1
            Else
                labelCount.Add Key:=labelRange(i, 1).Value, Item:=1
            End If
            totalSum = totalSum + dataRange(i, 1).Value
            totalCount = totalCount + 1
        Next
    End If
    
    'Calculate average
    'Dim ts As String
    Dim rKey
    For Each rKey In labelSum
        labelAvg.Add Key:=rKey, Item:=((labelSum.Item(rKey) * totalCount) / (labelCount.Item(rKey) * totalSum))
    Next
    
    'Output data
    If dataRange.Rows.Count < dataRange.Columns.Count Then
        For i = 1 To dataRange.Columns.Count
            output(1, i) = dataRange(1, i).Value / labelAvg.Item(labelRange(1, i).Value)
        Next
    Else
        For i = 1 To dataRange.Rows.Count
            output(i, 1) = dataRange(i, 1).Value / labelAvg.Item(labelRange(i, 1).Value)
        Next
    End If
    
    Set labelSum = Nothing
    Set labelCount = Nothing
    Set labelAvg = Nothing
    
    SeasonAverage = output
    
End Function
--------------------------------------------------------------------------


The function takes two arguments. The first is the list of labels. These need to be the list of the seasons (Q1, Q2, etc. or January, February, etc.). The second argument is the data. The output is a variant and hence the function needs to be used as an array function.

The labels are placed in to dictionary objects and averages calculated for each label. Then the data is divided by these averages.

Example

I'm going to use an artificial set of data for my example. The data has been generated by adding a constant line of 100 to a seasonal trend (ranging from -10 to +40) and then some noise (normal distribution with a mean of zero and standard deviation of 4). This is basically the definition of a seasonal data series.

The first chart shows the raw generated data:


The next chart shows the data once the seasonal component has been removed.

And the third chart compares the generated noise with the derived noise from applying the macro.

As expected the noises compare quite well. As they should, given the mathematics involved.

Conclusion

This is a very simplistic way of seasonally adjusting time series data. It does not take into account trend breaks, outliers (although if there is enough data, this method can be used to spot them), holiday movements, trading days, variable days in the month etc.

It is quite a good first stab at adjusting data and seeing long term trends though. Also, because of the simplicity of the method you can be fairly sure that not too many artifacts are being introduced into the data.

As always, comments, potential improvements etc are welcome.

Thursday, 1 August 2013

Excel macro to detect outliers

Introduction

Automatically detecting outliers in a set of data is generally fairly difficult. There are various papers for example from the University of York or the University of Pittsburgh summarising the various methods. What follows is a description of one the most simple. It is useful for straight line data (univariate) or any data that can be made linear.

The macro just looks at the correlation of the whole data series and calculates, for each data point, the difference of the correlation from the total without that point.

The point with the highest difference is the point that affects the correlation the most and is the most likely to be the outlier.

The Macro

As per usual most of the macro relates to arranging the data and checking the inputs.
--------------------------------------------------------------------------------
Function OutlierCorrelation(dataRange1 As range, dataRange2 As range) As Variant

    Dim OutputRows As Long
    Dim OutputCols As Long
    Dim output() As Double
    Dim vert As Boolean
    Dim ma As Double
    Dim i, j As Integer
    Dim totalCor As Double
    Dim cor() As Double
    Dim x() As Double
    Dim y() As Double
    
    With Application.Caller
        OutputRows = .Rows.Count
        OutputCols = .Columns.Count
    End With
    ReDim output(1 To OutputRows, 1 To OutputCols)
    
    'Check that dataRange1, dataRange2 and outputRange are the same size
    If dataRange1.Rows.Count <> dataRange2.Rows.Count Then Exit Function
    If dataRange1.Columns.Count <> dataRange2.Columns.Count Then Exit Function
    If dataRange1.Rows.Count <> OutputRows Then Exit Function
    If dataRange1.Columns.Count <> OutputCols Then Exit Function
    
    vert = False
    If dataRange1.Rows.Count > dataRange1.Columns.Count Then
        vert = True
        ReDim x(1 To dataRange1.Rows.Count - 1)
        ReDim y(1 To dataRange1.Rows.Count - 1)
        ReDim cor(1 To dataRange1.Rows.Count)
    Else
        ReDim x(1 To dataRange1.Columns.Count - 1)
        ReDim y(1 To dataRange1.Columns.Count - 1)
        ReDim cor(1 To dataRange1.Columns.Count)
    End If
    
    'Populate output with zeroes
    For i = 1 To OutputRows
        For j = 1 To OutputCols
            output(i, j) = 0
        Next
    Next
    
    'Calculate total correlation
    totalCor = Application.WorksheetFunction.Correl(dataRange1, dataRange2)
    
    If vert = True Then
        For i = 1 To dataRange1.Rows.Count
            For j = 1 To dataRange1.Rows.Count
                If i = j Then
                    
                Else
                    If j > i Then
                        x(j - 1) = dataRange1(j, 1).Value
                        y(j - 1) = dataRange2(j, 1).Value
                    Else
                        x(j) = dataRange1(j, 1).Value
                        y(j) = dataRange2(j, 1).Value
                    End If
                End If
            Next
            cor(i) = Correlation(x, y)
            output(i, 1) = Abs(cor(i) - totalCor)
        Next
    Else
        For i = 1 To dataRange1.Columns.Count
            For j = 1 To dataRange1.Columns.Count
                If i = j Then
                    
                Else
                    If j > i Then
                        x(j - 1) = dataRange1(1, j).Value
                        y(j - 1) = dataRange2(1, j).Value
                    Else
                        x(j) = dataRange1(1, j).Value
                        y(j) = dataRange2(1, j).Value
                    End If
                End If
            Next
            cor(i) = Correlation(x, y)
            output(1, i) = Abs(cor(i) - totalCor)
        Next
    End If
    
    OutlierCorrelation = output
    
End Function
--------------------------------------------------------------------------------


The macro also needs another function to calculate the correlation:
--------------------------------------------------------------------------------
Function Correlation(x() As Double, y() As Double) As Double

    Dim sx As Double
    Dim sy As Double
    Dim s1 As Double
    Dim s2 As Double
    Dim s3 As Double
    Dim k As Integer

    sx = 0
    sy = 0
    s1 = 0
    s2 = 0
    s3 = 0
    For k = 1 To UBound(x)
        sx = x(k) + sx
        sy = y(k) + sy
    Next
    sx = sx / UBound(x)
    sy = sy / UBound(x)
        
    For k = 1 To UBound(x)
        s1 = s1 + (x(k) - sx) * (y(k) - sy)
        s2 = s2 + (x(k) - sx) ^ 2
        s3 = s3 + (y(k) - sy) ^ 2
    Next
    Correlation = s1 / Sqr(s2 * s3)

End Function
--------------------------------------------------------------------------------

Example

Using the following data, which admittedly is very false:

The line was produced using y=2x+3 then adding uniform errors (8*rand()-4). I could have used normal errors using the norminv() function but it was easier using the rand() function on its own. It doesn't make too much difference for the purposes. I then changed the error on one of the points to be much higher. No prizes for spotting it and indeed a quick calculation of point-fitted line would quickly find it.

Using the macro gives:

The outlier is now very obvious. The correlation difference for the outlier is about 20 times higher than for the other points. For comparison the difference from the best fit line for the outlier is 6.5 time the average difference of the other points.

So this example is rather artificial but does mark less obvious outliers with real data.


Monday, 29 July 2013

ONS Interactive Content

Just a quick post this time. Came across the interactive data sets that the ONS seem to be pushing quite a lot on their home page.

Some of them are excellent and very interesting. I particularly liked their flow maps. There are two that are quite interesting:

  1. Internal migration - where do people move to within the England and Wales.
  2. UK trade in services - how much is being exported and where to (within the EU).
I did something similar in Java recently whilst working on some Spanish data. What I did only shows distribution of data within the regions of Spain so not quite as flashy. I'll write it up soon, although it may take a while.

The interactive content on the ONS website is split into several sections including the recent census, economy, the labour market, people and places, population and migration and education.

All very exciting.

Wednesday, 3 July 2013

European GDP Statistics

Much is written about GDP in Europe these days. So I thought I'd have a look and see what the actual figures are and what has changed in them over the last few years.

I obtained the figures from Eurostat. The figures in question were the GDP per capita figures and the GDP figures were from the 'nama_gdp_c' data set.

The first chart below shows the average relative GDP for separate European countries from 2000 to 2014 (forecasted) indexed on Germany.


I then looked at the growth in GDP for each of the top 5 countries by indexing the GDP for each year on the GDP for 2000.



As you can see, the UK has shown the largest relative drop, although it seems to be growing fairly well since 2009. It's still not back at the levels it was at in 2007 though. The other 4 countries show a small decline in 2009.

The third chart shows the GDP per capita in PPS. To quote Eurostat:


Gross domestic product (GDP) is a measure for the economic activity. It is defined as the value of all goods and services produced less the value of any goods or services used in their creation. The volume index of GDP per capita in Purchasing Power Standards (PPS) is expressed in relation to the European Union (EU27) average set to equal 100. If the index of a country is higher than 100, this country's level of GDP per head is higher than the EU average and vice versa. Basic figures are expressed in PPS, i.e. a common currency that eliminates the differences in price levels between countries allowing meaningful volume comparisons of GDP between countries. Please note that the index, calculated from PPS figures and expressed with respect to EU27 = 100, is intended for cross-country comparisons rather than for temporal comparisons.

Therefore it shows the GDP per capita in PPS relative to the EU as a whole.



On this measure Italy has dropped the most, closely followed by the UK. Germany has actually grown. But remember these figures are relative to the EU. Therefore even if Germany looks like it's growing, on this measure, it's only against the rest of Europe.

Anyway, of the 5 big EU countries, the UK seems to be doing quite badly.

Saturday, 29 June 2013

Deaths statistics in England and Wales

I found some interesting data on the ONS website recently. It related to the number of registered deaths by year by age of the person in England and Wales. A bit morbid maybe but interesting none the less.



Above is an animated gif of the number of deaths by year of age as a proportion of the total number of deaths in that year. The gif was created in GIMP, following their tutorial, although I did find it a lot easier to just paste the charts as new layers. I've decided that I don't really like animated gifs. You can't stop them, and when you want to study a particular point in detail it's gone before you can.

Anyway, there are 3 interesting points to this:

  1. Average age - The movement of the peak to the right.
  2. Infant mortality - The drop in the percentage of the number of deaths of people aged zero.
  3. The war - The dip in the percentage of the number of deaths of people aged 80 around the year 2000.


Wednesday, 26 June 2013

An Excel macro to concatenate values in ranges of cells

Introduction

The concatenate function in Excel seems very limited. Maybe I'm not using it correctly or there's a clever way of concatenating ranges of cells that I don't know but having to click on every cell that you want to concatenate just seems a waste of time.

I therefore thought that a better solution would be to add a function that automatically concatenates ranges of cells. Better yet, you can specify a separator for the cells if, for example, you wanted to output a range of semi-colon delimited data rather than just comma or tab delimited.

Macro

Anyway here's the macro:

-----------------------------------------------------------------------------
Function ConcatenateRange(d As Range, Optional separator As String) As String

    Dim s As String
    Dim i, j As Integer
    
    s = ""

    For i = 1 To d.Columns.Count
        For j = 1 To d.Rows.Count
            s = s + d.Cells(j, i).Text + separator
        Next
    Next
    
    s = Left(s, Len(s) - Len(separator))
    
    ConcatenateRange = s

End Function
-----------------------------------------------------------------------------

Example

Say you had a list of data spread over six cells from A1 to F1:


The function will join the cells, separated by the semi-colon.