Scraping some French medical school rankings
[This article was first published on blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
In this post, I will analyze the results of the “épreuves classantes nationales (ECN)”, which is a competitive examination at the end of the 6th year of medical school in France. First ones get to choose first where they want to continue their medical training.
A very clean dataset
The data is in a PDF there. I’m not an expert in scraping and parsing data but this was actually very simple due to a well-formatted dataset.
If you see that I’m doing something too complicated or could do it cleaner or faster, please comment this post and help me learn some scraping/parsing jedi techniques.
Scraping
I’ll use package pdftools to get the text from this PDF.
head(txt <- pdftools::pdf_text("https://goo.gl/wUXvjk"), n = 1)
## [1] " Paris, le 28 juin 2017\n Liste des étudiants et des internes de médecine, classés par ordre de\n mérite, ayant satisfait aux épreuves classantes nationales anonymes\n donnant accès au troisième cycle des études médicales, organisées au\n titre de l’année universitaire 2017-2018.\n Nota : Il est demandé aux étudiants de vérifier les informations d’état civil les\n concernant et d’envoyer à la gestionnaire des ECN la copie d’une pièce d’identité pour\n prise en compte des modifications. Ces corrections sont importantes avant parution de\n cette liste au Journal officiel de la République française.\n0001 Mme Beaumont (Anne-Lise), née le 1 septembre 1993.\n0002 M. Petitdemange (Arthur, Paul, Joseph), né le 15 septembre 1993.\n0003 M. Bacon (Seraphin, Charly, Philippe), né le 29 janvier 1993.\n0004 M. Ditac (Geoffroy, Arnaud, André), né le 23 juin 1994.\n0005 M. Faure (Guillaume, Thomas), né le 7 août 1992.\n0006 Mme Weil (Amandine), née le 4 mars 1992.\n0007 M. Ezzouhairi (Nacim), né le 1 juin 1993.\n0008 M. Coulon (Antoine), né le 11 juin 1992.\n0009 Mme Le Gaudu (Violette, Luce, Catherine), née le 3 juin 1994.\n0010 M. Boyer (Jeremy), né le 14 mai 1993.\n0011 M. Villemaire (Axel, Michaël, Pierre), né le 24 juillet 1991.\n0012 M. Azoulay (Levi-Dan), né le 13 décembre 1993.\n0013 M. Assouline (Allan), né le 14 janvier 1994.\n0014 M. Rouchaud (Aymeric), né le 16 mai 1993.\n0015 M. Padden (Michael, James), né le 6 février 1993.\n0016 M. Gavoille (Antoine, Paul, Jean), né le 12 juillet 1994.\n0017 M. Marie (Benjamin, Pierre, Alexandre), né le 14 mars 1994.\n0018 Mlle Boulle (Charlotte, Marie, Cécile), née le 3 septembre 1988.\n0019 Mme Chatelain (Juliette), née le 23 juillet 1993.\n0020 Mme Laporte (Amandine, Capucine), née le 9 octobre 1993.\n0021 M. D'izarny Gargas (Thibaut, François, Arnaud), né le 2 mai 1993.\n0022 Mme Boccon Gibod (Clémentine, Raphaëlle), née le 24 avril 1993.\n0023 Mme Chan (Camille, Marie), née le 11 juillet 1990.\n0024 M. Lemasle (Aymeric), né le 7 avril 1994.\n0025 M. Sulman (David), né le 9 novembre 1992.\n0026 Mlle Fresnel (Clémentine), née le 5 mars 1992.\n0027 M. Dumortier (Pierre, Antoine, Frédéric), né le 19 juillet 1993.\n0028 Mme Torres-Villaros (Héloïse, Laure, Barbara), née le 18 septembre 1992.\n0029 M. Memmi (Benjamin), né le 3 décembre 1993.\n0030 Mme Kherabi (Yousra), née le 23 décembre 1993.\n 1\n"
Parsing
I’ll use the little I know about regular expressions to parse this data.
pat <- "([0-9]{4} [M\\.|Mme|Mlle]{1}.*?, [né|née]{1}.*?)\\."
data <- unlist(gsubfn::strapply(txt, pattern = pat))
head(data)
## [1] "0001 Mme Beaumont (Anne-Lise), née le 1 septembre 1993"
## [2] "0002 M. Petitdemange (Arthur, Paul, Joseph), né le 15 septembre 1993"
## [3] "0003 M. Bacon (Seraphin, Charly, Philippe), né le 29 janvier 1993"
## [4] "0004 M. Ditac (Geoffroy, Arnaud, André), né le 23 juin 1994"
## [5] "0005 M. Faure (Guillaume, Thomas), né le 7 août 1992"
## [6] "0006 Mme Weil (Amandine), née le 4 mars 1992"
library(stringr)
data_parsed <- matrix(NA_character_, length(data), 7)
data_words <- str_extract_all(data, boundary("word"))
data_parsed[, 1:4] <- t(sapply(data_words, head, n = 4))
data_parsed[, 5:7] <- t(sapply(data_words, tail, n = 3))
head(data_parsed)
## [,1] [,2] [,3] [,4] [,5] [,6] [,7]
## [1,] "0001" "Mme" "Beaumont" "Anne" "1" "septembre" "1993"
## [2,] "0002" "M" "Petitdemange" "Arthur" "15" "septembre" "1993"
## [3,] "0003" "M" "Bacon" "Seraphin" "29" "janvier" "1993"
## [4,] "0004" "M" "Ditac" "Geoffroy" "23" "juin" "1994"
## [5,] "0005" "M" "Faure" "Guillaume" "7" "août" "1992"
## [6,] "0006" "Mme" "Weil" "Amandine" "4" "mars" "1992"
suppressMessages(library(tidyverse))
data_parsed2 <- as_tibble(data_parsed) %>%
transmute(
ranking = as.integer(V1),
is_male = (V2 == "M"),
family_name = V3,
first_name = V4,
birth_date = pmap(list(V5, V6, V7), function(d, m, y) {
paste(d, m, y, collapse = " ")
}) %>% lubridate::dmy()
)
data_parsed2
## # A tibble: 8,370 x 5
## ranking is_male family_name first_name birth_date
## <int> <lgl> <chr> <chr> <date>
## 1 1 FALSE Beaumont Anne 1993-09-01
## 2 2 TRUE Petitdemange Arthur 1993-09-15
## 3 3 TRUE Bacon Seraphin 1993-01-29
## 4 4 TRUE Ditac Geoffroy 1994-06-23
## 5 5 TRUE Faure Guillaume 1992-08-07
## 6 6 FALSE Weil Amandine 1992-03-04
## 7 7 TRUE Ezzouhairi Nacim 1993-06-01
## 8 8 TRUE Coulon Antoine 1992-06-11
## 9 9 FALSE Le Gaudu 1994-06-03
## 10 10 TRUE Boyer Jeremy 1993-05-14
## # ... with 8,360 more rows
Note: there is a problem with people who have a family name composed of multiple words.
Analysis
Proportion male/female
mean(data_parsed2$is_male) ## [1] 0.4345281
I’m still a bit surprised there are only 43% of males in French medical schools.
How old are they?
myggplot <- function(...) bigstatsr:::MY_THEME(ggplot(...)) myggplot(data_parsed2) + geom_histogram(aes(x = birth_date), bins = 100)
If one pass without repeating any year, they would be born in 1993, like me. There are a lot of people who repeat the first year because it is a very selective competitive examination, so who were born in 1992. Yet, there are quite a lot of older people and even some very young ones (we’ll see better in another plot).
How males compare to females when it comes to ranking?
myggplot(mutate(data_parsed2, prop_male = cummean(data_parsed2$is_male))) + geom_hline(yintercept = mean(data_parsed2$is_male), col = "red") + geom_line(aes(x = ranking, y = prop_male))
Even though the first one is a female, among the best ranked people, there is a majority of males.
Ranking versus Age
(myggplot(data_parsed2) + geom_point(aes(ranking, birth_date, color = is_male)) + aes(text = bigstatsr::asPlotlyText(data_parsed2))) %>% plotly::ggplotly(tooltip = "text")
To leave a comment for the author, please follow the link and comment on their blog: blog.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.