Calculating Average Consumption From One Week of Purchases
[This article was first published on EconBS, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
A number of large surveys have attempted to quantify consumer consumption from a limited period
of time observed. This task can be fairly complex as it is fraught with potentially large difficulties directly observing who is consuming what. Rather than this expensive method some researchers have attempted to substitute more easily observed purchase patterns inferring that in general house holds are going to consume what they purchase.
In order to aid in this analysis researchers collect data on both what is purchased and over what period of time it is to be consumed, for instance today (1) or over the next week (7).
Yet purchase patterns can be difficult to work with. Typically household consumption does not map perfectly to household consumption. For one, households can consume stocks from previous weeks. Likewise, households can purchase food to be held in stock for future weeks.
In order to adjust for missing consumption levels we want to adjust consumption to account for both the food items that will not all be consumed the week of observations
(1)
$$ C_{current.purchases} = C_{purchase} frac{days.remaining.in.observation.period}{days.expected.to.consume}$$
as well as the food items that were purchased the previous week and consumed this week. We can calculate the probabilities of observing an individual outcome in the following way:
(2)
$$ P_{observing.purchase} = frac{observation.period}{days.expected.to.consume}$$
We can note that the probability of observing a particular purchase if greater than 1 need only be set to one since if this is the case it is likely that this particular purchase will appear one or more times in our data.
Now we can combine (1) and (2) by dividing the current purchases by the likelihood of observing those purchases.
(3)
$$ E(C_{current.purchases}) = C_{purchase} frac{days.remaining.in.observation.period}{days.expected.to.consume}/frac{observation.period}{days.expected.to.consume}$$
$$=C_{purchase} frac{days.remaining.in.observation.period}{observation.period}$$
If the probability is less than 1 otherwise we can use equation (1).
Finally in order to calculate average consumption we take the daily average for our estimated expected consumption levels? Right?
Not even close. This only begins to capture the problem as we have multiple purchases often on different days consumed in different patterns throughout the week.
In order to get us closer to the appropriate level of estimated consumption we need to both infer the missing consumption as well as spread out the observed consumption so that when we look at daily averages good A purchased on day 1 with an expected consumption period of 1 week will also be included with good B purchased on day 7.
In order to explore how to estimate consumption from only observing a limited period of time I have written a simulation testing four methods of estimation. The true consumption level for any individual is 1 unit. If there are multiple goods consumed than that 1 unit of consumption is spread across all goods so that every day only one unit is consumed.
Using only one good we get the following results. M1 is just taking the mean consumption if we divide quantity of goods purchased by number of days expected to consume. M2 is adjusting consumption by the inverse of the likelihood of observing that consumption. M3 is spreading consumption across all of the days of the week observed. M4 is both adjusting by likelihood of observations and spreading consumption across days of the week observed.
Table 1:Sim # is the simulation number while # Items is the number of different food items purchased while C Spread is the number of days consumption of that item is spread over. All values are simulated 250 times.
Notice that with only 1 item consumed M1 and M3 are equivalent and M2 and M4 are equivalent. We can see that expected consumption for M2 and M4 provide much better estimates than for M1 and M3 when the consumption is spread out for goods for more than the observation period of one week on average.
Things get much more difficult when we include other goods in our calculation.
Table 2: Equivalent to Table 1 except now multiple items are being purchased at different periods (identified as # Items). In this the C Spread only refers to the first item. The remaining items are drawn randomly from the possible consumption spreads with much greater weight applied to lower consumption levels.
When consuming multiple items simultaneously, the importance of spreading consumption out across all days observed becomes increasingly important. This is because daily consumption need be calculated as the sum of goods consumed that each day averaged across the number of days observed. Thus we see that while in Table 1 M2 does very well. In Table 2 M3 and M4 do much better than either M1 or M2 and M4 does slightly better than any other method at approximating total consumption.
It is worth noting that all of the methods underestimate total consumption though M4 does the best at adjusting for the missing data problem.
There are some things to consider when estimating consumption data in this way. One important thing is that if consumption tends to be for goods consumed over a long period of time then using anything but directly dividing by the period of time expected to be consumed over is going to give some pretty lumpy values.
For instance, imagine someone buys four liters of oil which they expect to consume over the next 30 days. Sure on average in order to account for the oil not observed for the many other similar people who bought their oil on previous periods you may want to divide the oil not by the thirty days (4L/30days) but by the probably adjustment value equation (3).
Thus you get (4L/7days). Averaging across four similar people who did not happen to purchase oil you approximate the population consumption level. (1L/7days*1/4=1/28). Thus on average for the population estimate, you are pretty close. However for that one guy in your data you now have one person who looks like they are consuming 4/7 of a liter of oil per day.
When screening your data for outliers this oil consumption positively pops out of the page at you. So you figure it is some kind of recording error and replace it using population estimates.
But the problem here is entirely created by the method used to infer consumption levels. If instead you had taken the consumer at his/her reported level and said that average consumption for that individual is 4L/30days or 2/15 liters per day then you would never need to substitute out this particular outlier because it would not exist in the data in the first place.
If you would like to review the R simulation used to generate these results you can find it here.
In order to aid in this analysis researchers collect data on both what is purchased and over what period of time it is to be consumed, for instance today (1) or over the next week (7).
Yet purchase patterns can be difficult to work with. Typically household consumption does not map perfectly to household consumption. For one, households can consume stocks from previous weeks. Likewise, households can purchase food to be held in stock for future weeks.
In order to adjust for missing consumption levels we want to adjust consumption to account for both the food items that will not all be consumed the week of observations
(1)
$$ C_{current.purchases} = C_{purchase} frac{days.remaining.in.observation.period}{days.expected.to.consume}$$
as well as the food items that were purchased the previous week and consumed this week. We can calculate the probabilities of observing an individual outcome in the following way:
(2)
$$ P_{observing.purchase} = frac{observation.period}{days.expected.to.consume}$$
We can note that the probability of observing a particular purchase if greater than 1 need only be set to one since if this is the case it is likely that this particular purchase will appear one or more times in our data.
Now we can combine (1) and (2) by dividing the current purchases by the likelihood of observing those purchases.
(3)
$$ E(C_{current.purchases}) = C_{purchase} frac{days.remaining.in.observation.period}{days.expected.to.consume}/frac{observation.period}{days.expected.to.consume}$$
$$=C_{purchase} frac{days.remaining.in.observation.period}{observation.period}$$
If the probability is less than 1 otherwise we can use equation (1).
Finally in order to calculate average consumption we take the daily average for our estimated expected consumption levels? Right?
Not even close. This only begins to capture the problem as we have multiple purchases often on different days consumed in different patterns throughout the week.
In order to get us closer to the appropriate level of estimated consumption we need to both infer the missing consumption as well as spread out the observed consumption so that when we look at daily averages good A purchased on day 1 with an expected consumption period of 1 week will also be included with good B purchased on day 7.
In order to explore how to estimate consumption from only observing a limited period of time I have written a simulation testing four methods of estimation. The true consumption level for any individual is 1 unit. If there are multiple goods consumed than that 1 unit of consumption is spread across all goods so that every day only one unit is consumed.
Using only one good we get the following results. M1 is just taking the mean consumption if we divide quantity of goods purchased by number of days expected to consume. M2 is adjusting consumption by the inverse of the likelihood of observing that consumption. M3 is spreading consumption across all of the days of the week observed. M4 is both adjusting by likelihood of observations and spreading consumption across days of the week observed.
Table 1:Sim # is the simulation number while # Items is the number of different food items purchased while C Spread is the number of days consumption of that item is spread over. All values are simulated 250 times.
| Sim | # Items | C Spread | M1 | M2 | M3 | M4 |
|---|---|---|---|---|---|---|
| 1 | 1 | 1 | 1.00 | 1.00 | 1.00 | 1.00 |
| 2 | 1 | 2 | 1.00 | 1.00 | 1.00 | 1.00 |
| 3 | 1 | 3 | 1.00 | 1.00 | 1.00 | 1.00 |
| 4 | 1 | 5 | 1.00 | 1.00 | 1.00 | 1.00 |
| 5 | 1 | 6 | 1.00 | 1.00 | 1.00 | 1.00 |
| 6 | 1 | 7 | 1.00 | 1.00 | 1.00 | 1.00 |
| 7 | 1 | 8 | 0.88 | 1.01 | 0.88 | 1.01 |
| 8 | 1 | 9 | 0.74 | 0.95 | 0.74 | 0.95 |
| 9 | 1 | 10 | 0.70 | 1.00 | 0.70 | 1.00 |
| 10 | 1 | 15 | 0.42 | 0.89 | 0.42 | 0.89 |
| 11 | 1 | 20 | 0.35 | 1.01 | 0.35 | 1.01 |
Notice that with only 1 item consumed M1 and M3 are equivalent and M2 and M4 are equivalent. We can see that expected consumption for M2 and M4 provide much better estimates than for M1 and M3 when the consumption is spread out for goods for more than the observation period of one week on average.
Things get much more difficult when we include other goods in our calculation.
Table 2: Equivalent to Table 1 except now multiple items are being purchased at different periods (identified as # Items). In this the C Spread only refers to the first item. The remaining items are drawn randomly from the possible consumption spreads with much greater weight applied to lower consumption levels.
| Sim | # Items | C Spread | M1 | M2 | M3 | M4 |
|---|---|---|---|---|---|---|
| 12 | 2 | 1 | 0.72 | 0.72 | 0.90 | 0.91 |
| 13 | 2 | 2 | 0.71 | 0.71 | 0.90 | 0.91 |
| 14 | 2 | 3 | 0.71 | 0.71 | 0.91 | 0.92 |
| 15 | 2 | 5 | 0.64 | 0.65 | 0.87 | 0.89 |
| 16 | 2 | 6 | 0.66 | 0.67 | 0.83 | 0.84 |
| 17 | 2 | 7 | 0.60 | 0.60 | 0.80 | 0.81 |
| 18 | 2 | 8 | 0.59 | 0.62 | 0.78 | 0.83 |
| 19 | 2 | 9 | 0.59 | 0.65 | 0.74 | 0.83 |
| 20 | 2 | 10 | 0.58 | 0.65 | 0.70 | 0.81 |
| 21 | 2 | 15 | 0.58 | 0.71 | 0.64 | 0.83 |
| 22 | 2 | 20 | 0.54 | 0.69 | 0.59 | 0.83 |
| 23 | 3 | 1 | 0.62 | 0.62 | 0.86 | 0.88 |
| 24 | 3 | 2 | 0.61 | 0.62 | 0.88 | 0.89 |
| 25 | 3 | 3 | 0.58 | 0.59 | 0.87 | 0.88 |
| 26 | 3 | 5 | 0.53 | 0.53 | 0.83 | 0.84 |
| 27 | 3 | 6 | 0.55 | 0.55 | 0.81 | 0.82 |
| 28 | 3 | 7 | 0.50 | 0.51 | 0.78 | 0.79 |
| 29 | 3 | 8 | 0.52 | 0.53 | 0.78 | 0.82 |
| 30 | 3 | 9 | 0.52 | 0.54 | 0.75 | 0.81 |
| 31 | 3 | 10 | 0.50 | 0.54 | 0.72 | 0.80 |
| 32 | 3 | 15 | 0.49 | 0.55 | 0.68 | 0.82 |
| 33 | 3 | 20 | 0.47 | 0.53 | 0.63 | 0.76 |
| 34 | 4 | 1 | 0.58 | 0.58 | 0.84 | 0.85 |
| 35 | 4 | 2 | 0.55 | 0.55 | 0.85 | 0.86 |
| 36 | 4 | 3 | 0.52 | 0.52 | 0.84 | 0.85 |
| 37 | 4 | 5 | 0.48 | 0.48 | 0.81 | 0.82 |
| 38 | 4 | 6 | 0.49 | 0.49 | 0.80 | 0.82 |
| 39 | 4 | 7 | 0.46 | 0.46 | 0.77 | 0.78 |
| 40 | 4 | 8 | 0.46 | 0.48 | 0.76 | 0.79 |
| 41 | 4 | 9 | 0.48 | 0.49 | 0.76 | 0.81 |
| 42 | 4 | 10 | 0.48 | 0.50 | 0.73 | 0.79 |
| 43 | 4 | 15 | 0.45 | 0.49 | 0.70 | 0.79 |
| 44 | 4 | 20 | 0.44 | 0.47 | 0.68 | 0.78 |
| 45 | 5 | 1 | 0.56 | 0.57 | 0.85 | 0.86 |
| 46 | 5 | 2 | 0.52 | 0.52 | 0.84 | 0.85 |
| 47 | 5 | 3 | 0.49 | 0.49 | 0.83 | 0.84 |
| 48 | 5 | 5 | 0.45 | 0.45 | 0.79 | 0.80 |
| 49 | 5 | 6 | 0.45 | 0.46 | 0.78 | 0.79 |
| 50 | 5 | 7 | 0.45 | 0.45 | 0.78 | 0.79 |
| 51 | 5 | 8 | 0.44 | 0.45 | 0.76 | 0.78 |
| 52 | 5 | 9 | 0.44 | 0.45 | 0.76 | 0.79 |
| 53 | 5 | 10 | 0.44 | 0.45 | 0.74 | 0.79 |
| 54 | 5 | 15 | 0.44 | 0.46 | 0.71 | 0.77 |
| 55 | 5 | 20 | 0.42 | 0.45 | 0.69 | 0.76 |
| 56 | 6 | 1 | 0.52 | 0.53 | 0.83 | 0.85 |
| 57 | 6 | 2 | 0.49 | 0.49 | 0.83 | 0.85 |
| 58 | 6 | 3 | 0.47 | 0.48 | 0.82 | 0.83 |
| 59 | 6 | 5 | 0.44 | 0.45 | 0.79 | 0.81 |
| 60 | 6 | 6 | 0.44 | 0.45 | 0.80 | 0.81 |
| 61 | 6 | 7 | 0.43 | 0.44 | 0.78 | 0.79 |
| 62 | 6 | 8 | 0.45 | 0.46 | 0.76 | 0.79 |
| 63 | 6 | 9 | 0.44 | 0.45 | 0.77 | 0.80 |
| 64 | 6 | 10 | 0.42 | 0.44 | 0.73 | 0.78 |
| 65 | 6 | 15 | 0.43 | 0.45 | 0.74 | 0.82 |
| 66 | 6 | 20 | 0.41 | 0.43 | 0.70 | 0.76 |
When consuming multiple items simultaneously, the importance of spreading consumption out across all days observed becomes increasingly important. This is because daily consumption need be calculated as the sum of goods consumed that each day averaged across the number of days observed. Thus we see that while in Table 1 M2 does very well. In Table 2 M3 and M4 do much better than either M1 or M2 and M4 does slightly better than any other method at approximating total consumption.
|
| Figure 1: Estimator performance given different item consumption spreads. The above values are for the estimator value averaged across between 1 and 6 items consumed with only the first item being at that particular spread value. M is method 1 through 4 described above. |
There are some things to consider when estimating consumption data in this way. One important thing is that if consumption tends to be for goods consumed over a long period of time then using anything but directly dividing by the period of time expected to be consumed over is going to give some pretty lumpy values.
For instance, imagine someone buys four liters of oil which they expect to consume over the next 30 days. Sure on average in order to account for the oil not observed for the many other similar people who bought their oil on previous periods you may want to divide the oil not by the thirty days (4L/30days) but by the probably adjustment value equation (3).
Thus you get (4L/7days). Averaging across four similar people who did not happen to purchase oil you approximate the population consumption level. (1L/7days*1/4=1/28). Thus on average for the population estimate, you are pretty close. However for that one guy in your data you now have one person who looks like they are consuming 4/7 of a liter of oil per day.
When screening your data for outliers this oil consumption positively pops out of the page at you. So you figure it is some kind of recording error and replace it using population estimates.
But the problem here is entirely created by the method used to infer consumption levels. If instead you had taken the consumer at his/her reported level and said that average consumption for that individual is 4L/30days or 2/15 liters per day then you would never need to substitute out this particular outlier because it would not exist in the data in the first place.
If you would like to review the R simulation used to generate these results you can find it here.
To leave a comment for the author, please follow the link and comment on their blog: EconBS.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.