Counting Patent First Filings the Tidy Way with R

Dealing with duplicates

We will deal with the question of duplicate data first. This is extremely common with patent data. Duplication arises because a single patent application may be published and republished multiple times (as an application, grant or correction). Duplication is inherent to the system as a global system. Duplication is also prominent when working with patent data because the most common way of retrieving data from a patent database is through publication numbers. Put simply, we can’t read a document that hasn’t been published and so when querying databases it is publications that we see. It is also publications that are downloaded from databases. For some databases, such as Derwent Innovation, there does not appear to be a way to deduplicate the data prior to export and so this has to be handled after export. In other cases, such as the free Lens patent databases or other commercial databases, it is possible to reduce the data onto a single filing. However, the criteria that are applied when deduplicating at source are often unclear – and may vary between databases – so this can impact on your ability to understand the data. If in doubt choose the rawest form and work from there.

Let’s look at the data again to gain an understanding of the duplication issue. We will arrange the data by the application number for reasons that will become clear in a moment. If you are following this in R then note that arrange() puts the application number in alphabetical order. select() using - drops the columns we don’t want to see right now. Because the duplicates can be difficult to spot I have selected a few rows to make this clear.

numbers %>% 
  arrange(application_number) %>% 
  select(-family_first, -family_number, -publication_number) %>% 
  .[53:60,]
## # A tibble: 8 x 2
##   priority_number                                      application_number 
##   <chr>                                                <chr>              
## 1 EP1980400905A 1980-06-19; FR197916840A 1979-06-29    EP1980400905A 1980…
## 2 EP1980400905A 1980-06-19; FR197916840A 1979-06-29    EP1980400905A 1980…
## 3 SE19799920A 1979-11-30                               EP1980850181A 1980…
## 4 SE19799920A 1979-11-30                               EP1980850181A 1980…
## 5 JP1979145870A 1979-11-09; JP1979145871A 1979-11-09;… EP1980902127A 1980…
## 6 JP1979145870A 1979-11-09; JP1979145871A 1979-11-09;… EP1980902127A 1980…
## 7 US1979105606A 1979-12-20                             EP1981900248A 1980…
## 8 US1979105606A 1979-12-20                             EP1981900248A 1980…

There are two things we need to note in this view.

First, some of our data is concatenated (joined) with ; as the separator. Second, and more importantly for the moment, we can see that we seem to have duplicate application numbers e.g. EP1980400905A in the first and second rows and then onwards.

The reason that we have duplicates in the data is that a patent application may be published multiple times (for example as an application and as a grant or with corrections etc.). So, in the data above we can see that EP1980400905A 1980-06-19 has been published as EP22391A1 and EP22391B1 with the two letter kind codes at the end of the publication number representing the first publication of the application (A1) and kind code B1 representing the first publication of a patent grant.⁴

numbers %>% 
  arrange(application_number) %>% 
  select(-family_first, -family_number, - priority_number) %>% 
  .[53:60,]
## # A tibble: 8 x 2
##   application_number       publication_number
##   <chr>                    <chr>             
## 1 EP1980400905A 1980-06-19 EP22391B1         
## 2 EP1980400905A 1980-06-19 EP22391A1         
## 3 EP1980850181A 1980-11-28 EP30219B1         
## 4 EP1980850181A 1980-11-28 EP30219A1         
## 5 EP1980902127A 1980-11-06 EP39740B1         
## 6 EP1980902127A 1980-11-06 EP39740A1         
## 7 EP1981900248A 1980-12-17 EP42004B1         
## 8 EP1981900248A 1980-12-17 EP42004A1

Where a document is republished the application number will be duplicated. This means that we will also end up with duplicated priority numbers and we will over count. Removing duplicate records is the key requirement for accurate counts of patent data.

So, lets remove the duplicate application numbers first. To do that we will use a simple piece of R code from the dplyr package (you loaded it with the tidyverse) to add a new column that identifies the duplicated records. We will create a new column called duplicated using mutate() which adds columns. We will add the duplicated column by applying the R function duplicated() to the application_number column. What this does is to loop over the column and identifies the first instance of the application_number and then duplicates of the application_number. The first instance of the application_number will be marked as FALSE (not duplicated) and the others as TRUE (duplicated). We will then use the filter() function from dplyr to limit the data to our non-duplicated numbers == FALSE.

A couple of other things to note is that we will put this in a new table called numbers unique by using the assignment <- operator. We also use the pipe %>% operator which takes what it finds on the left hand side and passes it into the right hand side. So, we see that numbers %>% mutate() passes the numbers table or data.frame from the left hand side into mutate() to create a new column based on the contents of the call to mutate. It’s simple and logical when you become familiar with it. At the end of this chunk of code we will limit the data to just the priority_number and the application_number which we will be using as a key.⁵ The select() function from dplyr will only select those columns that we name inside it and will drop the others. These three functions: select() for columns, filter() for rows, and mutate() to add new values connected with the pipe %>% represent the building blocks for almost everything you need to do with patent data in R. The rest such as duplicated() help you to perform particular operations and we will go into more detail with that below.

numbers_unique <- numbers %>%
  mutate(duplicated = duplicated(application_number)) %>% 
  filter(duplicated == "FALSE") %>% 
  select(priority_number, application_number, publication_number)

nrow(numbers_unique) # count the rows
## [1] 15776

This reduces our original 18,970 records to 15,776 records. We now want to take a look at our data to check for missingness.

Missing Data

We will be counting and graphing the priority numbers. So we will want to check that all of our records have a priority number. We will also be using dates to graph the data and it will be a very good idea to check the dates at this stage. The reason for this is that strange things can happen with patent dates and this is often linked to missingness in the data as we will see in a moment.

In R missing data is represented by NA for Not Available. Working with NA data can be awkward and a source of considerable frustration because NA is not a value, it is the absence of a value. We can solve this by adding a column using mutate() that will test the priority number field for NA values is.na(). We will then apply a filter to allow us to see the top results where the value for is.na() is TRUE. To see all the data add %>% View() to the end.

numbers_unique %>% 
  mutate(missing_priority = is.na(priority_number)) %>% 
  filter(missing_priority == "TRUE")
## # A tibble: 96 x 4
##    priority_number application_number    publication_num… missing_priority
##    <chr>           <chr>                 <chr>            <lgl>           
##  1 <NA>            USD502486A 0001-01-01 US502486A        TRUE            
##  2 <NA>            USD500197A 0001-01-01 US500197A        TRUE            
##  3 <NA>            USD499490A 0001-01-01 US499490A        TRUE            
##  4 <NA>            USD497518A 0001-01-01 US497518A        TRUE            
##  5 <NA>            USD565353A 0001-01-01 US565353A        TRUE            
##  6 <NA>            USD474115A 0001-01-01 US474115A        TRUE            
##  7 <NA>            USD459287A 0001-01-01 US459287A        TRUE            
##  8 <NA>            USD540479A 0001-01-01 US540479A        TRUE            
##  9 <NA>            USD522772A 0001-01-01 US522772A        TRUE            
## 10 <NA>            USD593712A 0001-01-01 US593712A        TRUE            
## # ... with 86 more rows

The first thing we notice about this data is that the dates for the records are 0001-01-01. This type of device (along with 999999) is often used to denote the absence of a date. In this case if we look up some of these cases we will discover that they are very old records. For example US322982A dates to 1885. The Paris Convention did not enter into force in the United States until May 1887 and it is unclear when exactly the USPTO started using the system, so it is not surprising that these documents lack priority numbers.⁶ If we continue to keep these records we will see an artificial spike of activity somewhere at the start of our graph. In this case we can safely drop these records using the handy drop_na() function from dplyr. We will simply overwrite the existing table and specify the priority number as the column where we will drop the rows with NA values.

numbers_unique <- numbers_unique %>% 
  drop_na(priority_number) %>% 
  select(-publication_number)

nrow(numbers_unique)
## [1] 15680

We have now reduced our dataset to 15,680 unique application numbers. Note that you may want to also use this type of test, investigate, decide approach with other fields but it is always a good idea to note down the decisions that you make when doing so, otherwise what Hadley Wickham has called “future you” will have no idea and your audience will also have no clue.

When working with this kind of data it is useful to create a reference number or even a full table that allows you to work out whether any operations you run afterwards are working correctly. In this case we now know that we have 15,680 application numbers. In the next section we will be working out the earliest priority dates for each of these documents. We therefore need to ensure that we end up with 15,680 application numbers. We will create a reference number called target from the number of rows nrow() in the dataset before we go any further. This can help us work out what is going wrong if we end up with different numbers at the end. For more complex cases try creating a copy of the full table that you can use to work out what is getting lost or not counting correctly.

target <- nrow(numbers_unique)
target
## [1] 15680

We now know that we have no missing priority dates so we can proceed to wrangling or processing the priority numbers.

Wrangling the Priority Numbers

We can’t count concatenated data properly, so our next step is to separate out the concatenated priority numbers into individual rows. We will also want to extract the dates from the priority numbers that we will use later on to create a graph. We will do this in one go. In the first step we will use separate_rows() from tidyr to break the priority numbers onto individual rows using ; as the separator. We then use separate() to separate out the priority number and the date component. This will create two new columns called priority and priority_date. We will then apply two functions to these columns using mutate(). The first will convert the priority date to date format in R. The second will trim any white space that appears at the front or end of the priority number field from the earlier separation. Trimming white space is an extremely important step. For example US1234 and the same number with a white space at the front or rear _US1234, where _ stands for the space, will be treated as a distinct number and will not count correctly. Trimming white space is a fundamental task when counting patent data and the single most common reason that your counts will not be correct at the end of all your hard work!

As a final step in data preparation we will add some additional features. We will identify the US provisional applications and we will count the number of priorities associated with an application. We will also extract the two letter country codes at the beginning of the priority and application number fields as they may assist us later and will be used in counts. Note that the count of priority numbers in n reveals the total number of priorities associated with an application number. We will not use all of these fields for this type of count but they are useful to assist with understanding the data as we move along.

numbers_unique <- numbers_unique %>% 
  separate_rows(priority_number, sep = ";") %>% 
  mutate(priority_number = str_trim(priority_number, side = "both")) %>%
  separate(priority_number, into = c("priority", "priority_date"), sep = " ", remove = FALSE) %>% 
  mutate(priority_date = lubridate::ymd(priority_date)) %>% 
  mutate(priority = str_trim(priority, side = "both")) %>%
  mutate(priority_number = str_trim(priority_number, side = "both")) %>% 
  mutate(provisional = str_detect(.$priority_number, "[[:digit:]]P ")) %>%
  group_by(application_number) %>%
  mutate(priority_count = seq_along(1)) %>%
  add_tally(wt = priority_count) %>% 
  ungroup() %>% 
  mutate(priority_country = str_sub(.$priority_number, 1,2)) %>% 
  mutate(application_country = str_sub(.$application_number, 1,2)) %>%
  select(-priority_count, -priority) # drop temporary count and unused column

numbers_unique
## # A tibble: 68,361 x 7
##    priority_number     priority_date application_number  provisional     n
##    <chr>               <date>        <chr>               <lgl>       <int>
##  1 US2016578323F 2016… 2016-09-20    US2016578323F 2016… FALSE           1
##  2 US14954632A 2015-1… 2015-11-30    US14954632A 2015-1… FALSE           1
##  3 US15360203A 2016-1… 2016-11-23    US15360203A 2016-1… FALSE           1
##  4 US62203383P 2015-0… 2015-08-10    US15454805A 2017-0… TRUE            2
##  5 US62314047P 2016-0… 2016-03-28    US15454805A 2017-0… TRUE            2
##  6 US62200764P 2015-0… 2015-08-04    US15263985A 2016-0… TRUE            2
##  7 US62314042P 2016-0… 2016-03-28    US15263985A 2016-0… TRUE            2
##  8 KR201528901A 2015-… 2015-03-02    US15057264A 2016-0… FALSE           1
##  9 US15217944A 2016-0… 2016-07-22    US15217944A 2016-0… FALSE           3
## 10 US2015196885P 2015… 2015-07-24    US15217944A 2016-0… TRUE            3
## # ... with 68,351 more rows, and 2 more variables: priority_country <chr>,
## #   application_country <chr>

Let’s quickly review what we just did in plain language. We separated each priority number onto its own row using the semicolon as the separator then we split off the priority number and the date and reformatted the date before finally trimming the white space around the numbers in the priority column. A couple of points to note here: in the call to separate we specified the separator or sep as a space, we opted to keep the original column with remove = FALSE (the default is true and removes the column). We then added a simple count because we know the number of application numbers are duplicated to the number of priorities and then grouped the applications to add a count of the total priorities per application with add_tally. We ungrouped the table and then extracted the priority country and application country. Ungrouping is important, but hard to remember, because if we do not ungroup the data then any calculation we apply will be applied by group. This will normally cause unexpected results or the calculation simply won’t work.

When preparing data in this way one of the signs that there are unresolved issues with your data is that you will receive warnings about missing or extra pieces of data when you use separate(). If you see these messages go back and inspect your data. It can mean that there are NA values in the column you are separating or it can mean that you have extra spaces (so there will be too many pieces) or something else is present in the data. Issues with white space are common culprits with patent data (following separation) and this is one of the reasons that there are two calls to trim white space with str_trim() as a security blanket to avoid later problems.

To make this clearer lets just try and run separate on our original concatenated data using the space as the separator.

numbers %>% 
  separate(priority_number, into = c("one", "two"), sep = " ")
## Warning: Expected 2 pieces. Additional pieces discarded in 10937 rows
## [4, 5, 7, 8, 11, 21, 22, 32, 33, 34, 37, 39, 40, 41, 42, 43, 44, 45, 48,
## 49, ...].
## # A tibble: 18,970 x 6
##    one         two        application_number    family_first family_number
##    <chr>       <chr>      <chr>                 <chr>        <chr>        
##  1 US20165783… 2016-09-20 US2016578323F 2016-0… <NA>         <NA>         
##  2 US14954632A 2015-11-30 US14954632A 2015-11-… <NA>         <NA>         
##  3 US15360203A 2016-11-23 US15360203A 2016-11-… <NA>         <NA>         
##  4 US62203383P 2015-08-1… US15454805A 2017-03-… <NA>         <NA>         
##  5 US62200764P 2015-08-0… US15263985A 2016-09-… <NA>         <NA>         
##  6 KR20152890… 2015-03-02 US15057264A 2016-03-… <NA>         <NA>         
##  7 US15217944A 2016-07-2… US15217944A 2016-07-… <NA>         <NA>         
##  8 US20081007… 2008-09-2… US14808174A 2015-07-… <NA>         <NA>         
##  9 FR20142036A 2014-09-12 US14848061A 2015-09-… <NA>         <NA>         
## 10 US14970643A 2015-12-16 US14970643A 2015-12-… <NA>         <NA>         
## # ... with 18,960 more rows, and 1 more variable: publication_number <chr>

We immediately get a warning about extra pieces in over 10,000 rows signifying that we need to go back and pay more attention to our data. In other cases you will not always be concerned about this, although it is an extremely good idea to be clear about why you are not concerned, and you can deal with extra data by specifying extra = "merge". For fun let’s try that.

numbers %>% 
  separate(priority_number, into = c("one", "two"), sep = " ", extra = "merge") %>% 
  select(one, two)
## # A tibble: 18,970 x 2
##    one           two                                                      
##    <chr>         <chr>                                                    
##  1 US2016578323F 2016-09-20                                               
##  2 US14954632A   2015-11-30                                               
##  3 US15360203A   2016-11-23                                               
##  4 US62203383P   2015-08-10; US62314047P 2016-03-28                       
##  5 US62200764P   2015-08-04; US62314042P 2016-03-28                       
##  6 KR201528901A  2015-03-02                                               
##  7 US15217944A   2016-07-22; US2015196885P 2015-07-24; US62196885P 2015-0…
##  8 US2008100721P 2008-09-27; US2008108743P 2008-10-27; US2008121159P 2008…
##  9 FR20142036A   2014-09-12                                               
## 10 US14970643A   2015-12-16                                               
## # ... with 18,960 more rows

As we would expect from using the space as a separator, the function is showing us the number in the first column and the date in the second but is then tacking on the rest of the data in cases with multiple priority numbers. Warnings and arguments such as extra = "merge" can help you get to grips with the issues in your data.

Identifying the earliest priority date

In the discussion of the options identified above we noted that we could:

1. Identify the earliest priority document
2. Identify the priority that is closest to the specific invention

Here we will focus on simply identifying the earliest priority. We can do this in a straight forward way by grouping our application numbers and then using the rank() function inside a call to mutate() to rank the dates from 1 to x. A key point here is that the default ranking method for the rank function is actually average. We therefore need to specify ties.method = "first" to get what we want. We then ungroup our table and filter to the earliest priority date using filing_order == 1.

earliest <- numbers_unique %>% 
  group_by(application_number) %>% 
  mutate(filing_order = rank(priority_date, ties.method = "first")) %>%
  ungroup() %>% 
  filter(filing_order == 1)

earliest
## # A tibble: 15,680 x 8
##    priority_number     priority_date application_number  provisional     n
##    <chr>               <date>        <chr>               <lgl>       <int>
##  1 US2016578323F 2016… 2016-09-20    US2016578323F 2016… FALSE           1
##  2 US14954632A 2015-1… 2015-11-30    US14954632A 2015-1… FALSE           1
##  3 US15360203A 2016-1… 2016-11-23    US15360203A 2016-1… FALSE           1
##  4 US62203383P 2015-0… 2015-08-10    US15454805A 2017-0… TRUE            2
##  5 US62200764P 2015-0… 2015-08-04    US15263985A 2016-0… TRUE            2
##  6 KR201528901A 2015-… 2015-03-02    US15057264A 2016-0… FALSE           1
##  7 US2015196885P 2015… 2015-07-24    US15217944A 2016-0… TRUE            3
##  8 US2008100721P 2008… 2008-09-27    US14808174A 2015-0… TRUE           22
##  9 FR20142036A 2014-0… 2014-09-12    US14848061A 2015-0… FALSE           1
## 10 US14970643A 2015-1… 2015-12-16    US14970643A 2015-1… FALSE           1
## # ... with 15,670 more rows, and 3 more variables: priority_country <chr>,
## #   application_country <chr>, filing_order <int>

We now have a data frame that identifies the earliest priority numbers in a set. The 15,680 records corresponds match our target of 15,680 application numbers and so all is good.

The final step with this data is to remember that this dataset is based on unique application numbers and not unique priority numbers. In practice, some of the application numbers in our set will share priority numbers with other applications and will be follow on filings. We therefore need to identify duplicates in the priority numbers and deduplicate to unique priority numbers.

earliest_unique <- earliest %>% 
  mutate(duplicate_priority = duplicated(.$priority_number)) %>% 
  filter(duplicate_priority == "FALSE")

This reduces our dataset to a total of 9,366 priority numbers. That is, these priority numbers are the earliest filings giving rise to the 15,680 applications in the drones dataset.

By pursuing this option we have arrived at the absolute earliest dateline in this dataset on drones through a process of deduplication. However, as the numbers suggest we have also taken out a lot of potentially useful information. At this point it is important to bear in mind that this type of calculation can only be used to graph baseline first filings. We will look at this in further depth in a follow on article.

Let’s quickly graph this data. Here we are using the popular R graphing package ggplot2 to draw quick graphs of the data. To learn more about using ggplot2 try the excellent R Graphics Cookbook by Winston Chang which is available in open access form online. A step by step walk through on using ggplot2 to visualise patent data is available in this article. If you prefer using Excel or Tableau then write the file to a .csv and then open it in your tool of choice. You can do this simply with the following line of code.

readr::write_csv(earliest_unique, "earliest_unique.csv")

ggplot2 is quite a lot more involved than working with Tableau, Excel or other tools but provides a powerful way to control graphing. Let’s take a quick look at the data.

Note that graphs of priority data display a characteristic data cliff the closer that we move towards the present. This reflects the fact that patent applications are normally published at least 24 months after they were originally filed. This data cliff can easily mislead an audience into believing that interest in a technology has suddenly collapsed when in reality we are missing or only have partial data for the period. It is therefore important to pull the year range back to accommodate this. Depending on your data it is sensible to pull back the year by at least two years and possibly three years.

earliest_unique %>% 
  select(-n) %>% 
  mutate(year = lubridate::year(priority_date)) %>% 
  filter(year >= 1990 & year <= 2015) %>% 
  group_by(year) %>%
  tally() %>%  
  ggplot(., aes(x = year, y = n)) +
  geom_line() +
  labs(title = "Trends in First Filings of Patent Applications for Drone Technology", x = "priority year", y = "first filings")

Note the speed bump in the data around 2008 that is likely to reflect the impact of the financial crisis with filings relating to drone technology before accelerating rapidly in recent years.⁷

In this dataset we can also gain an insight into the countries driving this trend by ranking them in a bar graph for the same period.

library(ggthemes)
earliest_unique %>%
  select(-n) %>% 
  filter(priority_date >= "1990-01-01" & priority_date <= "2017-12-01") %>% 
  group_by(priority_country) %>% 
  tally(sort = TRUE) %>% 
  filter(n > 100) %>% 
  ggplot(aes(x = reorder(priority_country, n), y = n, fill = priority_country)) +
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = "First Filings by Priority Country", x = "Priority Country", y = "First Filings") +
  geom_text(aes(y = n, label = n), size = 3, hjust = -0.1) +
  theme_igray() +
  scale_fill_tableau("tableau20") +
  theme(legend.position = "none")

Note here that the United States emerges first in the top 5 followed by Japan, France, Korea and the Patent Cooperation Treaty (WO). It is important to bear in mind here that WO records will typically be filed through national offices, although no priority number will be present, as we can see in the references to country codes inside the WO priority documents.

earliest_unique %>% 
  filter(priority_country == "WO") %>% 
  select(priority_number)
## # A tibble: 286 x 1
##    priority_number          
##    <chr>                    
##  1 WO2016US65141A 2016-12-06
##  2 WO2015CN79094A 2015-05-15
##  3 WO2014CN86739A 2014-09-17
##  4 WO2015EP76803A 2015-11-17
##  5 WO2014EP72175A 2014-10-16
##  6 WO2013US65291A 2013-10-16
##  7 WO2014PL50044A 2014-07-24
##  8 WO2014US21626A 2014-03-07
##  9 WO2013US46840A 2013-06-20
## 10 WO2012US69292A 2012-12-12
## # ... with 276 more rows

We can summarise this data by extracting the country codes in the middle of the WO numbers. This suggests that the country where the WO application was submitted was the US followed by China (CN) and so on. The reference to the IB in these numbers are for so called PCT direct filings that are filed directly with WIPO as the International Bureau (IB) for the Patent Cooperation Treaty.

earliest_unique %>% 
  filter(priority_country == "WO") %>% 
  select(priority_number) %>% 
  mutate(wo_source = str_sub(.$priority_number, 7,8)) %>% 
  count(wo_source, sort = TRUE)
## # A tibble: 25 x 2
##    wo_source     n
##    <chr>     <int>
##  1 US           71
##  2 CN           46
##  3 EP           43
##  4 JP           43
##  5 IB           19
##  6 KR           14
##  7 SE           13
##  8 RU            7
##  9 FR            5
## 10 PL            5
## # ... with 15 more rows

As such, for a fuller count we might consider reallocating these priority numbers to their respective national country offices.

Note that we cannot go much further with this data to chart application countries accurately because we have deduplicated the priority numbers that would provide access to the application country data. A superior approach would be to create a temporary field for the unique priorities that allows the linked application countries to be viewed.

Bringing together the code

To finish off this discussion let’s briefly summarise the code required to reduce the dataset to the early priority filings. Here we will present the code in one go following some pruning to remove extra elements that we did not use to generate this calculation.

earliest_priority <- numbers %>%
  mutate(duplicated = duplicated(application_number)) %>% 
  filter(duplicated == "FALSE") %>% 
  select(priority_number, application_number, publication_number) %>% 
  drop_na(priority_number) %>% 
  separate_rows(priority_number, sep = ";") %>% 
  mutate(priority_number = str_trim(priority_number, side = "both")) %>%
  separate(priority_number, into = c("priority", "priority_date"), sep = " ", remove = FALSE) %>% 
  mutate(priority_date = lubridate::ymd(priority_date)) %>% 
  mutate(priority_number = str_trim(priority_number, side = "both")) %>% 
  mutate(priority_country = str_sub(.$priority_number, 1,2)) %>% 
  group_by(application_number) %>% 
  mutate(filing_order = rank(priority_date, ties.method = "first")) %>% 
  ungroup() %>% 
  filter(filing_order == 1) %>% 
  mutate(duplicate_priority = duplicated(.$priority_number)) %>% 
  filter(duplicate_priority == "FALSE") %>% 
  select(-priority)

Calculating the earliest priority from the moment of import involved 18 lines of code focusing on using mutate(), filter(), select(), drop_na(), separate_rows(), group_by(), and ungroup(). Inside the mutate() function we created new columns to test for duplicates using duplicated(), we trimmed white space with str_trim(), extracted data with str_sub() and ranked data with rank(). As this makes clear R functions from the tidyverse provide building blocks that can be chained together in an easy to read way to transform data into a desired result. For this reason we advocate a tidy approach to patent analytics with R.

One of the most important features of R as a functional programming language is that we can wrap this code (a collection of instructions to functions) into a single function. We will call it extract priority. This code basically reproduces that above but with some additional decoration to address something called tidy evaluation in R. Tidy evaluation is intellectually challenging and will not be addressed here.⁸.

extract_priority <- function(data = NULL, priority_number = NULL, key = NULL){
  x <- data %>%
    select(!!priority_number, !!key) %>%
    mutate(duplicated = duplicated(.[[!!key]])) %>%
    filter(duplicated == FALSE) %>%
    drop_na(!!priority_number) %>%
    separate_rows(!!priority_number, sep = ";") %>%
    mutate(!!priority_number := str_trim(.[[!!priority_number]], side = "both")) %>%
    separate(!!priority_number, into = c("priority", "priority_date"), sep = " ", remove = FALSE) %>%
    mutate(priority_date = lubridate::ymd(priority_date)) %>%
    mutate(!!key := str_trim(.[[!!key]], side = "both")) %>%
    mutate(priority_country = str_sub(.[[!!priority_number]], 1,2)) %>% 
    group_by(!!!rlang::syms(key)) %>% 
    mutate(filing_order = rank(priority_date, ties.method = "first")) %>% 
    ungroup() %>% 
    filter(filing_order == 1) %>%
    mutate(duplicate_priority = duplicated(.[[!!priority_number]])) %>% 
    filter(duplicate_priority == "FALSE") %>% 
    select(-priority, -duplicated, -filing_order, -duplicate_priority)
}

This function takes three arguments. Data is a dataset, the priority number is the field that contains the raw priority number data and the key is the field that is used for grouping (assumed to be the application number).

We can test this as follows:

results <- extract_priority(data = numbers, priority_number = "priority_number", key = "application_number")
results
## # A tibble: 9,366 x 4
##    priority_number      priority_date application_number  priority_country
##    <chr>                <date>        <chr>               <chr>           
##  1 US2016578323F 2016-… 2016-09-20    US2016578323F 2016… US              
##  2 US14954632A 2015-11… 2015-11-30    US14954632A 2015-1… US              
##  3 US15360203A 2016-11… 2016-11-23    US15360203A 2016-1… US              
##  4 US62203383P 2015-08… 2015-08-10    US15454805A 2017-0… US              
##  5 US62200764P 2015-08… 2015-08-04    US15263985A 2016-0… US              
##  6 KR201528901A 2015-0… 2015-03-02    US15057264A 2016-0… KR              
##  7 US2015196885P 2015-… 2015-07-24    US15217944A 2016-0… US              
##  8 US2008100721P 2008-… 2008-09-27    US14808174A 2015-0… US              
##  9 FR20142036A 2014-09… 2014-09-12    US14848061A 2015-0… FR              
## 10 US14970643A 2015-12… 2015-12-16    US14970643A 2015-1… US              
## # ... with 9,356 more rows

What this means is that where we have a dataset with a priority number field and the application number field as a key we do not need to write all the code again by hand. We may have to adjust the code… for example if the numbers contain different separators (such as ;; in the case of the Lens database) or junk such as “[” is found in a data field. However, the ability to turn code into a reusable function is the most powerful feature of programming languages such as R and a powerful reason to engage with R when working with patent data.

Wrap Up

In this article we have taken a deep dive into the exploration of how to count the first filings of patent applications using information in the priority number field. We have focused on reducing a set of 18,970 patent applications to the earliest filings and arrived at 9,366 results.

The key take home messages from this article are that to identify the earliest priority filing we have to do the following

1. Deduplicate our data on application numbers
2. Separate the individual priority numbers onto their own row
3. Make sure we trim white space
4. Group the data on application numbers and then identify the earliest priority date for each application
5. Filter the data to the earliest priority date per application
6. Identify and remove duplicate priority numbers

As discussed above, this approach focuses on a straightforward method for identifying the earliest priority filing. A more sophisticated approach would break the dataset down to identify the cases where the priority number is identical to an application number and then work through the data focusing on provisional applications. The outcome of such an exercise will not be radically different, however it would arguably be more accurate in terms of identifying the priority date closest to the date of a specific invention and working through the filing route issues. For today however this is more than enough for a first deep dive into counting patent filings by priority. If you have survived this far congratulations. You now know more than most people alive about how to count priority filings. Yay!