Getting Started

To start with we need to ensure that RStudio and R for your operating system are installed by following the instructions on the RStudio website here. Do not forget to follow the link to also install R for your operating system.

When working in RStudio it is good practice to work with projects. This will keep all of the files for a project in the same folder. To create a project go to File, New Project and create a project. Call the project something like infographic. Any file you create and save for the project will now be listed under the Files tab in RStudio.

R works using packages (libraries) and there are around 7,490 of them for a whole range of purposes. We will use just a few of them. To install a package we use the following. Copy and paste the code into the Console and press enter.

install.packages("tidyverse")  # the group of packages you will need

Packages can also be installed by selecting the Packages tab and typing the name of the package.

To load the packages (libraries) use the following or check the tick box in the Packages pane.

library(readr)
library(dplyr)
library(tidyr)
library(stringr)
library(ggplot2)

We are now ready to go.

Load a .csv file using readr

We will work with the pizza_medium_clean dataset in the online Github Manual repository. If manually downloading a file remember to click on the file name and select Raw to download the actual file.

We can use the easy to use read_csv() function from the readr package to quickly read in our pizza data directly from the Github repository. Note the raw at the beginning of the filename.

library(readr)
pizza <- read_csv("https://github.com/wipo-analytics/opensource-patent-analytics/blob/master/2_datasets/pizza_medium_clean/pizza.csv?raw=true")

readr will display a warning for the file arising from its efforts to parse publication dates on import. We will ignore this as we will not be using this field.

As an alternative to importing directly from Github download the file and in RStudio use File > Import Dataset > From .csv. If you experience problems with direct import of a file the File > Import Dataset approach will give you a range of easy to use controls for figuring this out (e.g. where .csv is actually a tab separated file).

Viewing Data

We can view data in a variety of ways.

1. In the console:

pizza
## # A tibble: 9,996 x 31
##    applicants_cleaned    applicants_clean… applicants_orga… applicants_original
##    <chr>                 <chr>             <chr>            <chr>              
##  1 <NA>                  People            <NA>             <NA>               
##  2 Ventimeglia Jamie Jo… People            <NA>             Ventimeglia Jamie …
##  3 Cordova Robert; Mart… People            <NA>             Cordova Robert;Mar…
##  4 Lazarillo De Tormes … Corporate         Lazarillo De To… LAZARILLO DE TORME…
##  5 <NA>                  People            <NA>             <NA>               
##  6 Depoortere, Thomas    People            <NA>             DEPOORTERE, Thomas 
##  7 Frisco Findus Ag      Corporate         Frisco Findus Ag FRISCO-FINDUS AG   
##  8 Bicycle Tools Incorp… Corporate         Bicycle Tools I… Bicycle Tools Inco…
##  9 Castiglioni, Carlo    People            <NA>             CASTIGLIONI, CARLO 
## 10 <NA>                  People            <NA>             <NA>               
## # ... with 9,986 more rows, and 27 more variables: inventors_cleaned <chr>,
## #   inventors_original <chr>, ipc_class <chr>, ipc_codes <chr>,
## #   ipc_names <chr>, ipc_original <chr>, ipc_subclass_codes <chr>,
## #   ipc_subclass_detail <chr>, ipc_subclass_names <chr>,
## #   priority_country_code <chr>, priority_country_code_names <chr>,
## #   priority_data_original <chr>, priority_date <chr>,
## #   publication_country_code <chr>, publication_country_name <chr>,
## #   publication_date <chr>, publication_date_original <chr>,
## #   publication_day <int>, publication_month <int>, publication_number <chr>,
## #   publication_number_espacenet_links <chr>, publication_year <int>,
## #   title_cleaned <chr>, title_nlp_cleaned <chr>,
## #   title_nlp_multiword_phrases <chr>, title_nlp_raw <chr>,
## #   title_original <chr>

2. In Environment click on the blue arrow to see in the environment. Keep clicking to open a new window with the data.

3. Use the View() command (for data.frames and tables)

View(pizza)

If possible use the View() command or environment. The difficulty with the console is that large amounts of data will simply stream past.

Identifying Types of Object

We often want to know what type of object we are working with and more details about the object so we know what to do later. Here are some of the most common commands for obtaining information about objects.

class(pizza)  ## type of object
names(pizza)  ## names of variables
str(pizza)  ## structure of object
dim(pizza)  ## dimensions of the object

The most useful command in this list is str() because this allows us to access the structure of the object and see its type.

str(pizza, max.level = 1)

str() is particularly useful because we can see the names of the fields (vectors) and their type. Most patent data is a character vector with dates forming integers.

Working with Data

We will often want to select aspects of our data to focus on a specific set of columns or to create a graph. We might also want to add information, notably numeric counts.

The dplyr package provides a set of very handy functions for selecting, adding and counting data. The tidyr and stringr packages are sister packages that contain a range of other useful functions for working with our data. We have covered some of these in other chapters on graphing using R but will go through them quickly and then pull them together into a function that we can use across our dataset.

library(dplyr)
pizza_number <- select(pizza, publication_number, publication_year)

library(dplyr)
pizza_number <- mutate(pizza_number, n = sum(publication_number = 1))

library(dplyr)
pizza_total <- count(pizza_number, publication_year, wt = n)
pizza_total
## # A tibble: 58 x 2
##    publication_year    nn
##               <int> <dbl>
##  1             1940    1.
##  2             1954    1.
##  3             1956    1.
##  4             1957    1.
##  5             1959    1.
##  6             1962    1.
##  7             1964    2.
##  8             1966    1.
##  9             1967    1.
## 10             1968    8.
## # ... with 48 more rows

library(dplyr)
sum(pizza_total$nn)
## [1] 9996

library(dplyr)
pizza_total <- rename(pizza_total, pubyear = publication_year, publications = nn) %>% 
    print()
## # A tibble: 58 x 2
##    pubyear publications
##      <int>        <dbl>
##  1    1940           1.
##  2    1954           1.
##  3    1956           1.
##  4    1957           1.
##  5    1959           1.
##  6    1962           1.
##  7    1964           2.
##  8    1966           1.
##  9    1967           1.
## 10    1968           8.
## # ... with 48 more rows

library(ggplot2)
qplot(x = pubyear, y = publications, data = pizza_total, geom = "line")

library(dplyr)
library(readr)
pizza_total <- filter(pizza_total, pubyear >= 1990, pubyear <= 2012)
write_csv(pizza_total, "pizza_total_1990_2012.csv")
pizza_total
## # A tibble: 23 x 2
##    pubyear publications
##      <int>        <dbl>
##  1    1990         139.
##  2    1991         154.
##  3    1992         212.
##  4    1993         201.
##  5    1994         162.
##  6    1995         173.
##  7    1996         180.
##  8    1997         186.
##  9    1998         212.
## 10    1999         290.
## # ... with 13 more rows

Simplify code with pipes %>%

So far we have handled the code one line at a time. But, one of the great strengths of using a programming language is that we can run multiple lines of code together. There are two basic ways that we can do this.

We will create a new temporary object df to demonstrate this.

1. The standard way

library(dplyr)
library(ggplot2)
df <- select(pizza, publication_number, publication_year)
df <- mutate(df, n = sum(publication_number = 1))
df <- count(df, publication_year, wt = n)
df <- rename(df, pubyear = publication_year, publications = nn)
df <- filter(df, pubyear >= 1990, pubyear <= 2012)
qplot(x = pubyear, y = publications, data = df, geom = "line")

The code we have just created is six lines long. If we select all of this code and run it in one go it will produce our graph.

One feature of this code is that each time we run a function on the object total we name it at the start of each function (e.g. mutate(df…)) and then we overwrite the object.

We can save quite a lot of typing and reduce the complexity of the code using the pipe operator introduced by the the magrittr package and then adopted in Hadley Wickham’s data wrangling and tidying packages.

2. Using pipes %>%

Pipes are now a very popular way of writing R code because they simplify writing R code and speed it up. The most popular pipe is %>% which means “this” then “that”. In this case we are going to create a new temporary object df1 by first applying select to pizza, then mutate, count, rename and filter. Note that we only name our dataset once (in select()) and we do not need to keep overwriting the object.

library(dplyr)
library(ggplot2)
df1 <- select(pizza, publication_number, publication_year) %>% mutate(n = sum(publication_number = 1)) %>% 
    count(publication_year, wt = n) %>% rename(pubyear = publication_year, publications = nn) %>% 
    filter(pubyear >= 1990, pubyear <= 2012) %>% qplot(x = pubyear, y = publications, 
    data = ., geom = "line") %>% print()

In the standard code we typed df nine times to arrive at the same result. Using pipes we typed df1 once. Of greater importance is that the use of pipes simplifies the structure of R code by introducing a basic “this” then “that” logic which makes it easier to understand.

One point to note about this code is that qplot() requires us to name our data (in this case df1). However, df1 is actually the final output of the code and does not exist as an input object before the final line is run. So, if we attempt to use data = df1 in qplot() we will receive an error message. The way around this is to use . in place of our data object. That way qplot() will know we want to graph the outputs of the earlier code. Finally, we need to add an explicit call to print() to display the graph (without this the code will work but we will not see the graph).

If we now inspect the structure of the df1 object (using str(df1)) in the console, it will be a list. The reason for this is that it is an object with mixed components, including a data.frame with our data plus additional data setting out the contents of the plot. As there is no direct link between R and our infographics software this will create problems for us later because the infographics software won’t know how to interpret the list object. So, it is generally a good idea to use a straight data.frame by excluding the call to qplot and adding it later when needed as follows.

library(dplyr)
library(ggplot2)
df2 <- select(pizza, publication_number, publication_year) %>% mutate(n = sum(publication_number = 1)) %>% 
    count(publication_year, wt = n) %>% rename(pubyear = publication_year, publications = nn) %>% 
    filter(pubyear >= 1990, pubyear <= 2012) %>% print()
## # A tibble: 23 x 2
##    pubyear publications
##      <int>        <dbl>
##  1    1990         139.
##  2    1991         154.
##  3    1992         212.
##  4    1993         201.
##  5    1994         162.
##  6    1995         173.
##  7    1996         180.
##  8    1997         186.
##  9    1998         212.
## 10    1999         290.
## # ... with 13 more rows

Note that in this case the only change is that we need to explicitly include the reference to the df2 data frame as the data argument in the call to qplot().

library(ggplot2)
qplot(x = pubyear, y = publications, data = df2, geom = "line")

Harmonising data

One challenge with creating multiple tables from a baseline dataset is keeping track of subdatasets. At the moment we have two basic objects we will be working with:

1. pizza - our raw dataset
2. pizza_total - created via pizza_number limited to 1990_2012.

In the remainder of the chapter we will want to create some additional datasets from our pizza dataset. These are:

1. Country trends
2. Applicants
3. International Patent Classification (IPC) Class
4. Phrases
5. Google
6. Google IPC
7. Google phrases

We need to make sure that any data that we generate from our raw dataset matches the period for the pizza_total dataset. If we do not do this there is a risk that we will generate subdatasets with counts for the raw pizza dataset.

To handle this we will use filter() to create a new baseline dataset with an unambiguous name.

library(dplyr)
pizza_1990_2012 <- rename(pizza, pubyear = publication_year) %>% filter(pubyear >= 
    1990, pubyear <= 2012)
pizza_1990_2012
## # A tibble: 8,262 x 31
##    applicants_cleaned  applicants_clean… applicants_organ… applicants_original 
##    <chr>               <chr>             <chr>             <chr>               
##  1 <NA>                People            <NA>              <NA>                
##  2 Lazarillo De Torme… Corporate         Lazarillo De Tor… LAZARILLO DE TORMES…
##  3 <NA>                People            <NA>              <NA>                
##  4 Depoortere, Thomas  People            <NA>              DEPOORTERE, Thomas  
##  5 Frisco Findus Ag    Corporate         Frisco Findus Ag  FRISCO-FINDUS AG    
##  6 Bicycle Tools Inco… Corporate         Bicycle Tools In… Bicycle Tools Incor…
##  7 Castiglioni, Carlo  People            <NA>              CASTIGLIONI, CARLO  
##  8 <NA>                People            <NA>              <NA>                
##  9 Bujalski, Wlodzimi… People            <NA>              BUJALSKI, WLODZIMIE…
## 10 Ehrno Flexible A/S… Corporate; People Ehrno Flexible A… "EHRNO FLEXIBLE A/S…
## # ... with 8,252 more rows, and 27 more variables: inventors_cleaned <chr>,
## #   inventors_original <chr>, ipc_class <chr>, ipc_codes <chr>,
## #   ipc_names <chr>, ipc_original <chr>, ipc_subclass_codes <chr>,
## #   ipc_subclass_detail <chr>, ipc_subclass_names <chr>,
## #   priority_country_code <chr>, priority_country_code_names <chr>,
## #   priority_data_original <chr>, priority_date <chr>,
## #   publication_country_code <chr>, publication_country_name <chr>,
## #   publication_date <chr>, publication_date_original <chr>,
## #   publication_day <int>, publication_month <int>, publication_number <chr>,
## #   publication_number_espacenet_links <chr>, pubyear <int>,
## #   title_cleaned <chr>, title_nlp_cleaned <chr>,
## #   title_nlp_multiword_phrases <chr>, title_nlp_raw <chr>,
## #   title_original <chr>

In this case we start with a call to rename() to make this consistent with our pizza_total table and then use a pipe to filter the data on the year. Note here that when filtering raw data on a set of values it is important to inspect it first to check that the field is clean (e.g. not concatenated). If for some reason your data is concatenated (which happens quite a lot with patent data) then lookup ?tidyr::separate_rows.

We are now in a position to create our country trends table.

Country Trends using spread()

There are two basic data formats: long and wide. Our pizza dataset is in long format because each column is a variable (e.g. publication_country) and each row in publication_country contains a country name. This is the most common and useful data format.

However, in some cases, such as infogr.am our visualisation software will expect the data to be in wide format. In this case each country name would become a variable (column name) with the years forming the rows and the number of records per year the observations. The key to this is the tidyr() function spread().

As above we will start off by using select() to create a table with the fields that we want. We will then use mutate() to add a numeric field and then count up that data. To illustrate the process run this code (we will not create an object).

library(dplyr)
select(pizza_1990_2012, publication_country_name, publication_number, pubyear) %>% 
    mutate(n = sum(publication_number = 1)) %>% count(publication_country_name, pubyear, 
    wt = n) %>% print()
## # A tibble: 223 x 3
##    publication_country_name pubyear    nn
##    <chr>                      <int> <dbl>
##  1 Canada                      1990   19.
##  2 Canada                      1991   49.
##  3 Canada                      1992   66.
##  4 Canada                      1993   59.
##  5 Canada                      1994   50.
##  6 Canada                      1995   39.
##  7 Canada                      1996   36.
##  8 Canada                      1997   45.
##  9 Canada                      1998   46.
## 10 Canada                      1999   47.
## # ... with 213 more rows

When we run this code we will see the results in long format. We now want to take our publication_country_name column and spread it to form columns with nn as the values.

In using spread note that it takes a data argument (pizza_1990_2012), a key (publication_country_name), and value column (nn) (created from count()). We are using pipes so the data only needs to be named in the first line. For additional arguments see ?spread().

library(dplyr)
library(tidyr)
country_totals <- select(pizza_1990_2012, publication_country_name, publication_number, 
    pubyear) %>% mutate(n = sum(publication_number = 1)) %>% count(publication_country_name, 
    pubyear, wt = n) %>% spread(publication_country_name, nn)
country_totals
## # A tibble: 23 x 17
##    pubyear Canada China `Eurasian Patent… `European Paten… Germany Israel Japan
##      <int>  <dbl> <dbl>             <dbl>            <dbl>   <dbl>  <dbl> <dbl>
##  1    1990    19.   NA                NA               22.      2.    NA    NA 
##  2    1991    49.   NA                NA               29.      2.    NA    NA 
##  3    1992    66.   NA                NA               36.      2.     1.   NA 
##  4    1993    59.   NA                NA               29.      2.    NA    NA 
##  5    1994    50.   NA                NA               26.      5.    NA    NA 
##  6    1995    39.   NA                NA               29.      2.     1.   NA 
##  7    1996    36.    1.               NA               27.      1.     1.   NA 
##  8    1997    45.   NA                NA               34.      1.    NA    NA 
##  9    1998    46.   NA                NA               36.      1.    NA    17.
## 10    1999    47.    2.                2.              60.      4.    NA    26.
## # ... with 13 more rows, and 9 more variables: `Korea, Republic of` <dbl>,
## #   Mexico <dbl>, `Patent Co-operation Treaty` <dbl>, Portugal <dbl>, `Russian
## #   Federation` <dbl>, Singapore <dbl>, `South Africa` <dbl>, Spain <dbl>,
## #   `United States of America` <dbl>

We now have data in wide format.

In some cases, such as infogr.am, visualisation software may expect the country names to be the name of rows and the column names to be years . We can modify our call to spread() by replacing the publication_country_name with pubyear. Then we will write the data to disk for use in our infographic.

library(dplyr)
library(readr)
country_totals <- select(pizza_1990_2012, publication_country_name, publication_number, pubyear) %>%
  mutate(n = sum(publication_number = 1)) %>% 
  count(publication_country_name, pubyear, wt = n) %>% # note n
  spread(pubyear, nn) # note nn
country_totals
## # A tibble: 16 x 24
##    publication_country… `1990` `1991` `1992` `1993` `1994` `1995` `1996` `1997`
##    <chr>                 <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>
##  1 Canada                  19.    49.    66.    59.    50.    39.    36.    45.
##  2 China                   NA     NA     NA     NA     NA     NA      1.    NA 
##  3 Eurasian Patent Org…    NA     NA     NA     NA     NA     NA     NA     NA 
##  4 European Patent Off…    22.    29.    36.    29.    26.    29.    27.    34.
##  5 Germany                  2.     2.     2.     2.     5.     2.     1.     1.
##  6 Israel                  NA     NA      1.    NA     NA      1.     1.    NA 
##  7 Japan                   NA     NA     NA     NA     NA     NA     NA     NA 
##  8 Korea, Republic of      NA     NA     NA      1.    NA     NA      1.     1.
##  9 Mexico                  NA     NA     NA     NA     NA     NA     NA     NA 
## 10 Patent Co-operation…     8.    13.    31.    16.    20.    22.    23.    26.
## 11 Portugal                NA     NA     NA     NA     NA     NA     NA     NA 
## 12 Russian Federation      NA     NA     NA     NA     NA     NA     NA      5.
## 13 Singapore               NA     NA     NA     NA     NA     NA     NA     NA 
## 14 South Africa             2.     3.     3.     3.     3.     1.     9.     7.
## 15 Spain                   NA     NA     NA     NA     NA     NA     NA     NA 
## 16 United States of Am…    86.    58.    73.    91.    58.    79.    81.    67.
## # ... with 15 more variables: `1998` <dbl>, `1999` <dbl>, `2000` <dbl>,
## #   `2001` <dbl>, `2002` <dbl>, `2003` <dbl>, `2004` <dbl>, `2005` <dbl>,
## #   `2006` <dbl>, `2007` <dbl>, `2008` <dbl>, `2009` <dbl>, `2010` <dbl>,
## #   `2011` <dbl>, `2012` <dbl>
write_csv(country_totals, "pizza_country_1990_2012.csv")

To restore the data to long format we would need to use gather() as the counterpart to spread(). gather() takes a dataset, a key for the name of the column we want to gather the countries into, a value for the numeric count (in this case n), and finally the positions of the columns to gather in. Note here that we need to look up the column positions in country_totals (e.g. using View()) or count the columns using ncol(country_totals).

library(dplyr)
gather(country_totals, year, n, 2:24) %>% print()
## # A tibble: 368 x 3
##    publication_country_name     year      n
##    <chr>                        <chr> <dbl>
##  1 Canada                       1990    19.
##  2 China                        1990    NA 
##  3 Eurasian Patent Organization 1990    NA 
##  4 European Patent Office       1990    22.
##  5 Germany                      1990     2.
##  6 Israel                       1990    NA 
##  7 Japan                        1990    NA 
##  8 Korea, Republic of           1990    NA 
##  9 Mexico                       1990    NA 
## 10 Patent Co-operation Treaty   1990     8.
## # ... with 358 more rows

The combination of spread and gather work really well to prepare data in formats that are expected by other software. However, one of the main issues we encounter with patent data is that our data is not tidy because various fields are concatenated.

Tidying data - Separating and Gathering

In patent data we often see concatenated fields with a separator (normally a ;). These are typically applicant names, inventor names, International Patent Classification (IPC) codes, or document numbers (priority numbers, family numbers). We need to tidy this data prior to data cleaning (such as cleaning names) or to prepare for analysis and visualisation. For more on the concept of tidy data read Hadley Wickham’s Tidy Data article. The new R for Data Science book by Garrett Grolemund and Hadley Wickham (see Chapter 12) is also strongly recommended.

To tidy patent data we will typically need to do two things.

1. Separate the data so that each cell contains a unique data point (e.g. a name, code or publication number). This normally involves separating data into columns.

2. Gathering the data back in. This involves transforming the data in the columns we have created into rows.

Separating data into columns is very easy in tools such as Excel. However, gathering the data back into separate rows is remarkably difficult. Happily, this is very easy to do in R with the tidyr package.

The tidyr package contains three functions that are very useful when working with patent data. When dealing with concatenated fields in columns the key function is separate_rows.

Here we will work with the applicants_cleaned field in the pizza dataset. This field contains concatenated names with a ; as the separator. For example, on lines 1_9 there are single applicant names or NA values. However, on lines 10 and line 59 we see:

Ehrno Flexible A/S; Stergaard, Ole Farrell Brian; Mcnulty John; Vishoot Lisa

The problem here is that when we are dealing with thousands of lines of applicant names we don’t know how many names might be concatenated into each cell as a basis for separating the data into columns. Once we had split the columns (for example using Text to Columns in Excel) we would then need to work out how to gather the columns into rows. The separate_rows() function from tidyr makes light work of this problem. To use the function we name the dataset, the column we want to separate into rows and the separator (sep).

library(dplyr)
library(tidyr)
pizza1 <- separate_rows(pizza_1990_2012, applicants_cleaned, sep = ";")
pizza1
## # A tibble: 12,729 x 31
##    applicants_cleaned applicants_cleane… applicants_organ… applicants_original 
##    <chr>              <chr>              <chr>             <chr>               
##  1 <NA>               People             <NA>              <NA>                
##  2 Lazarillo De Torm… Corporate          Lazarillo De Tor… LAZARILLO DE TORMES…
##  3 <NA>               People             <NA>              <NA>                
##  4 Depoortere, Thomas People             <NA>              DEPOORTERE, Thomas  
##  5 Frisco Findus Ag   Corporate          Frisco Findus Ag  FRISCO-FINDUS AG    
##  6 Bicycle Tools Inc… Corporate          Bicycle Tools In… Bicycle Tools Incor…
##  7 Castiglioni, Carlo People             <NA>              CASTIGLIONI, CARLO  
##  8 <NA>               People             <NA>              <NA>                
##  9 Bujalski, Wlodzim… People             <NA>              BUJALSKI, WLODZIMIE…
## 10 Ehrno Flexible A/S Corporate; People  Ehrno Flexible A… "EHRNO FLEXIBLE A/S…
## # ... with 12,719 more rows, and 27 more variables: inventors_cleaned <chr>,
## #   inventors_original <chr>, ipc_class <chr>, ipc_codes <chr>,
## #   ipc_names <chr>, ipc_original <chr>, ipc_subclass_codes <chr>,
## #   ipc_subclass_detail <chr>, ipc_subclass_names <chr>,
## #   priority_country_code <chr>, priority_country_code_names <chr>,
## #   priority_data_original <chr>, priority_date <chr>,
## #   publication_country_code <chr>, publication_country_name <chr>,
## #   publication_date <chr>, publication_date_original <chr>,
## #   publication_day <int>, publication_month <int>, publication_number <chr>,
## #   publication_number_espacenet_links <chr>, pubyear <int>,
## #   title_cleaned <chr>, title_nlp_cleaned <chr>,
## #   title_nlp_multiword_phrases <chr>, title_nlp_raw <chr>,
## #   title_original <chr>

Our original dataset contained 8,262 rows. Our new dataset split on applicant names contains 12,729 rows. The function has moved our target column from column 1 to column 31 in the data frame. We can easily move it back to inspect.

library(dplyr)
pizza1 <- select(pizza1, 31, 1:30)

separate_rows() has done a great job but one of the problems with concatenated names is extra white space around the separator. We will deal with this next.

tail(pizza1$applicants_cleaned, 20)
##  [1] "Yahoo! Inc"                        "Clarcor Inc"                      
##  [3] "Holden Jeffrey A"                  " Vengroff Darren E"               
##  [5] "Casper Jeffrey L"                  " Erickson Braden J"               
##  [7] " Oppenheimer Alan A"               " Ray Madonna M"                   
##  [9] " Weber Jean L"                     "Pandey Neena"                     
## [11] " Sharma Sudhanshu"                 " Verizon Patent And Licensing Inc"
## [13] "Pandey Neena"                      " Sharma Sudhanshu"                
## [15] "Brown Michael"                     " Urban Scott"                     
## [17] "Brown Michael"                     " Urban Scott"                     
## [19] "Cole Lorin R"                      " Middleton Scott W"

tmp <- separate_rows(pizza_1990_2012, applicants_cleaned, sep = "; ")
tail(tmp$applicants_cleaned, 20)
##  [1] "Yahoo! Inc"                       "Clarcor Inc"                     
##  [3] "Holden Jeffrey A"                 "Vengroff Darren E"               
##  [5] "Casper Jeffrey L"                 "Erickson Braden J"               
##  [7] "Oppenheimer Alan A"               "Ray Madonna M"                   
##  [9] "Weber Jean L"                     "Pandey Neena"                    
## [11] "Sharma Sudhanshu"                 "Verizon Patent And Licensing Inc"
## [13] "Pandey Neena"                     "Sharma Sudhanshu"                
## [15] "Brown Michael"                    "Urban Scott"                     
## [17] "Brown Michael"                    "Urban Scott"                     
## [19] "Cole Lorin R"                     "Middleton Scott W"

library(stringr)
pizza1$applicants_cleaned <- str_trim(pizza1$applicants_cleaned, side = "both")
tail(pizza1$applicants_cleaned, 20)
##  [1] "Yahoo! Inc"                       "Clarcor Inc"                     
##  [3] "Holden Jeffrey A"                 "Vengroff Darren E"               
##  [5] "Casper Jeffrey L"                 "Erickson Braden J"               
##  [7] "Oppenheimer Alan A"               "Ray Madonna M"                   
##  [9] "Weber Jean L"                     "Pandey Neena"                    
## [11] "Sharma Sudhanshu"                 "Verizon Patent And Licensing Inc"
## [13] "Pandey Neena"                     "Sharma Sudhanshu"                
## [15] "Brown Michael"                    "Urban Scott"                     
## [17] "Brown Michael"                    "Urban Scott"                     
## [19] "Cole Lorin R"                     "Middleton Scott W"

library(dplyr)
library(tidyr)
library(stringr)
applicants <- rename(pizza, pubyear = publication_year) %>% filter(pubyear >= 1990, 
    pubyear <= 2012) %>% separate_rows(applicants_cleaned, sep = "; ") %>% rename(applicants = applicants_cleaned) %>% 
    select(31, 1:30)  # moves separated column to the beginning
applicants
## # A tibble: 12,729 x 31
##    title_original applicants applicants_clea… applicants_orga… applicants_orig…
##    <chr>          <chr>      <chr>            <chr>            <chr>           
##  1 PIZZA          <NA>       People           <NA>             <NA>            
##  2 IMPROVED PIZZA Lazarillo… Corporate        Lazarillo De To… LAZARILLO DE TO…
##  3 Pizza separat… <NA>       People           <NA>             <NA>            
##  4 Pizza separat… Depoorter… People           <NA>             DEPOORTERE, Tho…
##  5 PIZZA PREPARA… Frisco Fi… Corporate        Frisco Findus Ag FRISCO-FINDUS AG
##  6 Pizza Cutter   Bicycle T… Corporate        Bicycle Tools I… Bicycle Tools I…
##  7 PIZZA BOX      Castiglio… People           <NA>             CASTIGLIONI, CA…
##  8 PIZZA BOX      <NA>       People           <NA>             <NA>            
##  9 MORE ATTRACTI… Bujalski,… People           <NA>             BUJALSKI, WLODZ…
## 10 PIZZA PACKAGI… Ehrno Fle… Corporate; Peop… Ehrno Flexible … "EHRNO FLEXIBLE…
## # ... with 12,719 more rows, and 26 more variables: inventors_cleaned <chr>,
## #   inventors_original <chr>, ipc_class <chr>, ipc_codes <chr>,
## #   ipc_names <chr>, ipc_original <chr>, ipc_subclass_codes <chr>,
## #   ipc_subclass_detail <chr>, ipc_subclass_names <chr>,
## #   priority_country_code <chr>, priority_country_code_names <chr>,
## #   priority_data_original <chr>, priority_date <chr>,
## #   publication_country_code <chr>, publication_country_name <chr>,
## #   publication_date <chr>, publication_date_original <chr>,
## #   publication_day <int>, publication_month <int>, publication_number <chr>,
## #   publication_number_espacenet_links <chr>, pubyear <int>,
## #   title_cleaned <chr>, title_nlp_cleaned <chr>,
## #   title_nlp_multiword_phrases <chr>, title_nlp_raw <chr>

library(tidyr)
library(dplyr)
applicant_count <- select(applicants, applicants, publication_number) %>% mutate(n = sum(publication_number = 1)) %>% 
    count(applicants, wt = n) %>% arrange(desc(nn)) %>% filter(nn >= 1) %>% na.omit()
applicant_count
## # A tibble: 6,178 x 2
##    applicants                              nn
##    <chr>                                <dbl>
##  1 Graphic Packaging International, Inc  154.
##  2 Kraft Foods Holdings, Inc             132.
##  3 Google Inc                            123.
##  4 Microsoft Corporation                  88.
##  5 The Pillsbury Company                  83.
##  6 General Mills, Inc                     77.
##  7 Nestec                                 77.
##  8 The Procter & Gamble Company           59.
##  9 Pizza Hut, Inc                         57.
## 10 Yahoo! Inc                             54.
## # ... with 6,168 more rows

library(dplyr)
library(tidyr)
library(readr)
applicant_count <- select(applicants, applicants, publication_number) %>% mutate(n = sum(publication_number = 1)) %>% 
    count(applicants, wt = n) %>% arrange(desc(nn)) %>% filter(nn >= 54) %>% na.omit()
applicant_count
## # A tibble: 10 x 2
##    applicants                              nn
##    <chr>                                <dbl>
##  1 Graphic Packaging International, Inc  154.
##  2 Kraft Foods Holdings, Inc             132.
##  3 Google Inc                            123.
##  4 Microsoft Corporation                  88.
##  5 The Pillsbury Company                  83.
##  6 General Mills, Inc                     77.
##  7 Nestec                                 77.
##  8 The Procter & Gamble Company           59.
##  9 Pizza Hut, Inc                         57.
## 10 Yahoo! Inc                             54.
write_csv(applicant_count, "pizza_applicants_1990_2012.csv")

Selecting applicants using filter()

As we saw above, while select() functions with columns, filter() from dplyr works with rows. Here we will filter the data to select the rows in the applicants column that contain Google Inc. and then write that to a .csv for use in our infographic. Note the use of double == and the quotes around “Google Inc”.

library(dplyr)
library(readr)
google <- filter(applicants, applicants == "Google Inc")
google
## # A tibble: 123 x 31
##    title_original applicants applicants_clea… applicants_orga… applicants_orig…
##    <chr>          <chr>      <chr>            <chr>            <chr>           
##  1 Location base… Google Inc Corporate; Peop… Google Inc       Hafsteinsson Gu…
##  2 AUTHORITATIVE… Google Inc Corporate        Google Inc       GOOGLE INC.     
##  3 Location-Base… Google Inc Corporate; Peop… Google Inc       GOOGLE INC.;HAF…
##  4 Controlling t… Google Inc Corporate; Peop… Google Inc       GOOGLE, INC.;BE…
##  5 METHOD AND SY… Google Inc Corporate        Google Inc       GOOGLE INC.     
##  6 Routing queri… Google Inc Corporate        Google Inc       GOOGLE INC.     
##  7 METHODS AND S… Google Inc Corporate        Google Inc       GOOGLE INC.     
##  8 Aspect-based … Google Inc Corporate; Peop… Google Inc       Reis George;Goo…
##  9 Interpreting … Google Inc Corporate        Google Inc       GOOGLE INC.     
## 10 Interpreting … Google Inc Corporate        Google Inc       GOOGLE INC.     
## # ... with 113 more rows, and 26 more variables: inventors_cleaned <chr>,
## #   inventors_original <chr>, ipc_class <chr>, ipc_codes <chr>,
## #   ipc_names <chr>, ipc_original <chr>, ipc_subclass_codes <chr>,
## #   ipc_subclass_detail <chr>, ipc_subclass_names <chr>,
## #   priority_country_code <chr>, priority_country_code_names <chr>,
## #   priority_data_original <chr>, priority_date <chr>,
## #   publication_country_code <chr>, publication_country_name <chr>,
## #   publication_date <chr>, publication_date_original <chr>,
## #   publication_day <int>, publication_month <int>, publication_number <chr>,
## #   publication_number_espacenet_links <chr>, pubyear <int>,
## #   title_cleaned <chr>, title_nlp_cleaned <chr>,
## #   title_nlp_multiword_phrases <chr>, title_nlp_raw <chr>
write_csv(google, "google_1990_2012.csv")

Note that the correct result for the period 1990 to 2012 for Google is 123 records from 191 records across the whole pizza dataset. The correct result will be achieved only where you use the filtered, separated and trimmed data we created in the applicants data frame.

Generating IPC Tables

In the next step we will want to generate two tables containing International Patent Classification (IPC) data. IPC codes and the Cooperative Patent Classification (CPC, not present in this dataset) provide information on the technologies involved in a patent document. The IPC is hierarchical and proceeds from the general class level to the detailed group and subgroup level. Experience reveals that the majority of patent documents receive more than one IPC code to more fully describe the technological aspects of patent documents.

The pizza dataset contains IPC codes on the class and the subclass level in concatenated fields. One important consideration in using IPC data is that the descriptions are long and can be difficult for non-specialists to grasp. This can make visualising the data difficult and often requires manual efforts to edit labels for display.

We now want to generate three IPC tables.

1. A general IPC table for the pizza dataset
2. A general IPC table for the Google dataset
3. A more detailed IPC subclass table for the Google dataset

For ease of presentation in an infographic we will use the ipc_class field. For many patent analytics purposes this will be too general. However it has the advantage of being easy to visualise.

To generate the table we can use a generic function based on the code developed for dealing with the applicants data. We will call the function patent_count().

patent_count <- function(data, col, count_col, sep, n_results) {
    p_count <- dplyr::select(data, !!col, !!count_col) %>% tidyr::separate_rows(col, 
        sep = sep) %>% dplyr::mutate(`:=`(!!col, stringr::str_trim(.[[col]], side = "both"))) %>% 
        dplyr::mutate(n = sum(count_col = 1)) %>% dplyr::group_by(`:=`(!!col, .[[col]])) %>% 
        dplyr::tally(sort = TRUE) %>% dplyr::rename(records = nn) %>% na.omit() %>% 
        head(n_results)
}

The patent_count() function is based on the the code we developed for applicants. It contains variations to make it work as a function. The function takes four arguments:

1. col = the concatenated column that we want to split and gather back in
2. col_count = a column for generating counts (in this dataset the publication_number)
3. n_results = the number of results we want to see in the new table (typically 10 or 20 for visualisation). This is equivalent to the number of rows that you want to see.
4. sep = the separator to use to separate the data in col. With patent data this is almost always “;” (as ;space.

To generate the ipc_class data we can do the following and then write the file to .csv. Note that we have set the number of results n_results to 10.

pizza_ipc_class <- patent_count(data = pizza_1990_2012, col = "ipc_class", count_col = "publication_number", 
    n_results = 10, sep = ";")
pizza_ipc_class
## # A tibble: 10 x 2
##    ipc_class                                           records
##    <chr>                                                 <dbl>
##  1 A21: Baking                                           2233.
##  2 A23: Foods Or Foodstuffs                              1843.
##  3 B65: Conveying                                        1383.
##  4 G06: Computing                                        1326.
##  5 A47: Furniture                                         932.
##  6 H04: Electric Communication Technique                  747.
##  7 H05: Electric Techniques Not Otherwise Provided For    613.
##  8 F24: Heating                                           512.
##  9 A61: Medical Or Veterinary Science                     318.
## 10 G07: Checking                                          226.
write_csv(pizza_ipc_class, "pizza_ipcclass_1990_2012.csv")

Note that this dataset is based on the main pizza_1990_2012 dataset (including cases where no applicant name is available). The reason we have not used the applicants dataset is because that dataset will duplicate the IPC field for each split of an applicant name. As a result it will over count the IPCs by the number of applicants on a document name. As this suggests, it is important to be careful when working with data that has been tidied because of the impact on other counts.

This problem does not apply in the case of our Google data because the only applicant listed in that data is Google (excluding co-applicants). We can therefore safely use the Google dataset to identify the IPC codes.

google_ipc_class <- patent_count(data = google, col = "ipc_class", count_col = "publication_number", 
    n_results = 10, sep = ";")
google_ipc_class
## # A tibble: 7 x 2
##   ipc_class                             records
##   <chr>                                   <dbl>
## 1 G06: Computing                           105.
## 2 H04: Electric Communication Technique     17.
## 3 G01: Measuring                            14.
## 4 G09: Educating                            11.
## 5 G10: Musical Instruments                   7.
## 6 A63: Sports                                1.
## 7 G08: Signalling                            1.
write_csv(google_ipc_class, "google_ipcclass_1990_2012.csv")

There are only 7 classes and as we might expect they are dominated by computing. We might want to dig into this in a little more detail and so let’s also create an IPC subclass field.

google_ipc_subclass <- patent_count(data = google, col = "ipc_subclass_detail", count_col = "publication_number", 
    n_results = 10, sep = ";")
google_ipc_subclass
## # A tibble: 10 x 2
##    ipc_subclass_detail                                                  records
##    <chr>                                                                  <dbl>
##  1 G06F: Electric Digital Data Processing                                   89.
##  2 G06Q: Data Processing Systems Or Methods, Specially Adapted For Adm…     24.
##  3 G01C: Measuring Distances, Levels Or Bearings                            14.
##  4 G09B: Educational Or Demonstration Appliances                             9.
##  5 G10L: Speech Analysis Or Synthesis                                        7.
##  6 H04W: Wireless Communication Networks                                     7.
##  7 G09G: Arrangements Or Circuits For Control Of Indicating Devices Us…      5.
##  8 H04B: Transmission                                                        4.
##  9 H04L: Transmission Of Digital Information, E.G. Telegraphic Communi…      4.
## 10 H04M: Telephonic Communication                                            4.
write_csv(google_ipc_subclass, "google_ipcsubclass_1990_2012.csv")

We now have the data on technology areas that we need to understand our data. The next and final step is to generate data from the text fields.

pizza_phrases <- patent_count(data = pizza_1990_2012, col = "title_nlp_multiword_phrases", 
    count_col = "publication_number", n_results = 15, sep = ";")
pizza_phrases
## # A tibble: 15 x 2
##    title_nlp_multiword_phrases records
##    <chr>                         <dbl>
##  1 Food Product                   179.
##  2 Microwave Ovens                137.
##  3 Making Same                     48.
##  4 conveyor Oven                   46.
##  5 Crust Pizza                     44.
##  6 microwave Heating               41.
##  7 Bakery Product                  40.
##  8 pizza Box                       40.
##  9 Microwave Cooking               39.
## 10 Pizza Oven                      37.
## 11 pizza Dough                     35.
## 12 Cook Food                       34.
## 13 Baked Product                   33.
## 14 Related Method                  32.
## 15 Food Item                       29.
write_csv(pizza_phrases, "pizza_phrases_1990_2012.csv")

google_phrases <- patent_count(data = google, col = "title_nlp_multiword_phrases", 
    count_col = "publication_number", n_results = 15, sep = ";")
google_phrases
## # A tibble: 15 x 2
##    title_nlp_multiword_phrases           records
##    <chr>                                   <dbl>
##  1 Digital Map System                        10.
##  2 conversion Path Performance Measures       9.
##  3 Mobile Device                              8.
##  4 Search Results                             8.
##  5 Geographical Relevance                     4.
##  6 Local Search Results                       4.
##  7 Location Prominence                        4.
##  8 Network Speech Recognizers                 4.
##  9 Processing Queries                         4.
## 10 Search Query                               4.
## 11 aspect-Based Sentiment Summarization       3.
## 12 authoritative Document Identification      3.
## 13 Business Listings Search                   3.
## 14 Content Providers                          3.
## 15 indexing Documents                         3.
write_csv(google_phrases, "google_phrases_1990_2012.csv")

Creating an infographic in infogr.am

If you are starting this chapter here then download the datasets we will be using as a single zip file from the Manual repository here and then unzip the file.

We first need to sign up for a free account with infogr.am

We will then see a page with some sample infographics to provide ideas to get you started.

Click on one of the infograms with a graph such as Trends in Something and then click inside the graph box itself and select the edit button in the top right.

This will open up a data panel with the toy data displayed.

We want to replace this data by choosing the upload button and selecting our pizza_country_1990_2012.csv file.

We now have a decent looking graph for our country trends data where we can see the number of records per country and year by hovering over the relevant data points. While some of the countries with low frequency data are crunched at the bottom (and would be better displayed in a separate graph), hovering over the data or over a country name will display the relevant country activity. We will therefore live with this.

We now want to start adding story elements by clicking on the edit button in the title. Next we can start adding new boxes using the menu icons on the right. Here we have changed the title, added a simple body text for the data credit and then a quote from someone describing themselves as the Head of Pizza Analytics.

Next we need to start digging into the data using our IPC, applicants and phrases data.

To work with our IPC class data we will add a bar chart and load the data. To do this select the graph icon in the right and then Bar. Once again we will choose edit and then load our pizza_ipcclass_1990_2012 dataset. Then we can add a descriptive text box. We can then continue to add elements as follows:

1. applicants bar chart
2. pizza phrases by selecting graph and word cloud
3. Google ipc-subclass
4. Google word cloud.

One useful approach to developing an infographic is to start by adding the images and then add titles and text boxes to raise key points. In infogram new text boxes appear below existing boxes but can be repositioned by dragging and dropping boxes onto each other.

One nice feature of infogram is that it is easy to share the infographic with others through a url, an embed code or on facebook or via twitter.

At the end of the infographic it is a good idea to provide a link where the reader can obtain more information, such as the full report or the underlying data. In this case we will add a link to the Tableau workbook on pizza patent activity that we developed in an earlier chapter.

Our final infographic should look something like this.