Data for language research

Language research can include data from a variety of sources, linguistic and non-linguistic, that record observations about the world. A typical type of data used in quantitative language research is a corpus. In short, a corpus is set of machine-readable texts that have been compiled with an eye towards linguistic research. All corpora are not created equal, however, in content and/or format. A corpus may aim to represent a wide swath of language behavior or very specific aspects. It can be language specific (English or French), target a particular modality (spoken or written), or approximate domains of language use (medicine, business, etc.). A corpus that aims to represent a language (including modalities, registers, and sub-domains), for example, is known as a generalized corpus. Corpora that aim to capture a snapshot of a particular modality or sub-domain of language use are known as specialized corpora. Each corpus will have an underlying target population and the sampling process will reflect the authors’ best attempt (given the conditions at the point the corpus was compiled) at representing the stated target population. Whether a corpus is generalized or specialized can become difficult to nail down between the extremes. As such, it is key to be clear about the scope of a particular corpus to be able to ascertain its potential applications and gauge the extent to which these applications entail the research goals of your particular project.

A corpus will often include various types of non-linguistic attributes, or meta-data, as well. Ideally this will include information regarding the source(s) of the data, dates when it was acquired or published, and other author or speaker information. It may also include any number of other attributes that were identified as potentially important in order to appropriately document the target population. Again, it is key to match the available meta-data with the goals of your research. In some cases a corpus may be ideal in some aspects but not contain all the key information to address your research question. This may mean you will need to compile your own corpus if there are fundamental attributes missing. Before you consider compiling your own corpus, however, it is worth investigating the possibility of augmenting an available corpus to bring it inline with your particular goals. This may include adding new language sources, harnessing software for linguistic annotation (part-of-speech, syntactic structure, named entities, etc.), or linking available corpus meta-data to other resources, linguistic or non-linguistic.

Corpora come in various formats, the main three being: running text, structured documents, and databases. The format of a corpus is often influenced by characteristics of the data but may also reflect an author’s individual preferences as well. It is typical for corpora with few meta-data characteristics to take the form of running text. In corpora with more meta-data a header may be appended to the top of each running text document or the meta-data may be contained in a separate file with appropriate coding to coordinate meta-data attributes with each text in the corpus. When meta-data increases in complexity it is common to structure each corpus document more explicitly with a markup language such as XML (Extensible Markup Language) or organize relationships between language and meta-data attributes in a database. Although there has been a push towards standardization of corpus formats, most available resources display some degree of idiosyncrasy. Being able to parse the structure of a corpus is a skill that will develop with time. With more experience working with corpora you will become more adept at identifying how the data is stored and whether its content and format will serve the needs of your analysis.

Sources of language data

The most common source of data used in contemporary quantitative research is the internet. On the web an investigator can access corpora published for research purposes and language used in natural settings that can be coerced by the investigator into a corpus. Many organizations exist around the globe that provide access to corpora in browsable catalogs, or repositories. There are repositories dedicated to language research, in general, such as the Language Data Consortium or for specific language domains, such as the language acquisition repository TalkBank. It is always advisable to start looking for the available language data in a repository. The advantage of beginning your data search in repositories is that a repository, especially those geared towards the linguistic community, will make identifying language corpora faster than through a general web search. Furthermore, repositories often require certain standards for corpus format and documentation for publication. A standardized resource many times will be easier to interpret and evaluate for its appropriateness for a particular research project.

In the table below I’ve compiled a list of some corpus repositories to help you get started.

Repositories are by no means the only source of corpora on the web. Researchers from around the world provide access to corpora and other data sources on their own sites or through data sharing platforms. Corpora of various sizes and scopes will often be accessible on a dedicated homepage or appear on the homepage of a sponsoring institution. Finding these resources is a matter of doing a web search with the word ‘corpus’ and a list of desired attributes, including language, modality, register, etc. As part of a general movement towards reproducible more corpora are available on the web than ever before. Therefore data sharing platforms supporting reproducible research, such as GitHub, Zenodo, Re3data, etc., are a good place to look as well, if searching repositories and targeted web searches do not yield results.

In the table below you will find a list of corpus resources and datasets.

It is important to note that there can be access and use restrictions for data from particular sources. Compiling, hosting, and maintaining corpus resources can be costly. To gain full access to data, some repositories and homepages of larger corpora require a fee to offset these costs. In other cases, resources may require individual license agreements to ensure that the data is not being used in ways it was not intended or to ensure potentially sensitive participant information will be treated appropriately. You can take a look at a license agreement for the BYU Corpora as an example. If you are a member of an academic institution and aim to conduct research for scholarly purposes licensing is often easily obtained. Fees, on the other hand, may present a more challenging obstacle. If you are an affiliate of an academic institution it is worth checking with your library to see if there are funds for acquiring licensing for you as an individual, a research group or lab or, for the institution.

If your corpus search ends in a dead-end, either because a suitable resource does not appear to exist or an existing resource is unattainable given licensing restrictions or fees, it may be time to compile your own corpus. Turning to machine readable texts on the internet is usually the logical first step to access language for a new corpus. Language texts may be found on sites as uploaded files, such as pdf or doc (Word) documents, or found displayed as the primary text of a site. Given the wide variety of documents uploaded and language behavior recorded daily on social media, news sites, blogs and the like, compiling a corpus has never been easier. Having said that, how the data is structured and how much data needs to be retrieved can pose practical obstacles to collecting data from the web, particularly if the approach is to acquire the data by hand instead of automating the task. Our approach here, however, will be to automate the process as much as possible whether that means leveraging R package interfaces to language data, converting hundreds of pdf documents to plain text, or scraping content from web documents.

The table below lists some R packages that serve to interface language data directly through R.

Data for language research is not limited to (primary) text sources. Other sources may include processed data from previous research; word lists, linguistic features, etc.. Alone or in combination with text sources this data can be a rich and viable source of data for a research project.

Below I’ve included some processed language resources.

The list of data available for language research is constantly growing. I’ve document very few of the wide variety of resources. Below I’ve included attempts by others to provide a summary of the corpus data and language resources available.

Round up

In this post we have covered some of the basics on data for language research including types of data and sources to get you started on the path of identifying a viable source for your data analysis project. In the next post we will begin working directly with R code to access and acquire data through R. Along the way I will introduce fundamental programming concepts of the language you will use throughout your project.

References