How divided is the Senate?

I very seldom pay attention to politics directly, because politics have always seemed a bit circular and cyclical to me. Most of the political news that I take in ends up worming its way into the news sources that I do consume, like the excellent longform.org. Even given my limited intake of political news, one trend that I have noticed lately is the increasing number of references to the Senate as “polarized” or “divided.” Here is a link to an interesting series of charts on polarization. Is it possible to quantify this polarization? Can quantifying the polarization enable us to draw interesting conclusions?

As I started to walk down this road, I figured that it would be tough to find the data that I needed. My time in the US foreign service showed me just how slow the government can be at effectively publishing and using data. Imagine my surprise when I found that the senate website has a very convenient listing of all of the votes from the 101st congress to the 113th (current) congress. This data tells us, for each vote, whether each senator voted yes, no, or abstained.

From the vote data, we can generate plots showing how polarized the Senate is. We will assume that two people are not polarized if they have similar voting patterns. If we take only this vote, we would assume that Senator Ayotte and Senator Alexander, who both voted no, are not polarized, as they share the same opinion. This is well and good, but one bill isn’t really reflective of the voting records of the two Senators. If we really want to figure out where they stand, we would need to perform the analysis across all votes. I will describe the process further down, but for now, let’s jump to a polarization chart:

The above chart has a dot for each Senator, although only some senators are labelled due to space constraints. The further apart the dots are, the more the views of the two senators contrast. Dots are shaded by political affiliation. How can we generate this chart? Keep reading to find out.

Getting senate data

The first thing that we need to do is get the Senate data. We can start on this page. We see a roll call table in the bottom right. Each roll call table has a listing of each vote in a given congress:

If we click on the vote, we can see the results of the vote.

We can easily write a web scraper, such as the one i wrote to grab this data and store it. This will give us a file of all the voting data.

We can read the file into R (see this script), which gives us a list of lists containing all the vote information. Here is an excerpt:

“`r

senate[[2]] $number [1] “00187”

$session [1] “1”

$data

  Coons (D-DE),       McCain (R-AZ),    Chambliss (R-GA),      Franken (D-MN),       Inhofe (R-OK),      Johnson (D-SD),       Tester (D-MT),       Carper (D-DE),
           "Yea"                "Nay"                "Nay"                "Yea"                "Nay"                "Yea"                "Yea"                "Yea"
Schumer (D-NY),   Gillibrand (D-NY),      Fischer (R-NE),       Bennet (D-CO),     Barrasso (R-WY),      Shaheen (D-NH),      Boozman (R-AR),         Kirk (R-IL),
           "Yea"                "Yea"                "Nay"                "Yea"                "Nay"                "Yea"                "Nay"                "Nay"

“`

We can reformat this into a dataframe that has a record for each vote by each senator, and the congress/session/vote metadata:

This is just a short excerpt from the actual dataframe, which has 798,835 individual vote records.

Generating a vote matrix

Now we have a long list of all the votes by the senators, but we really want something like this:

“`

           Vote1      Vote2      Vote3

Senator1 (Yes/No) (Yes/No) (Yes/No) Senator2 (Yes/No) (Yes/No) (Yes/No) Senator3 (Yes/No) (Yes/No) (Yes/No) “`

To accomplish this, we will reformat our data by looping through each congress, then looping through each session in the congress, then looping through each vote in the session, and extracting the vote information.

We end up with this:

The above is an excerpt, so we are missing a lot of columns. As you can see, the leading column names are in the format congress.session.vote.

Decomposing vote matrix

Once we have a vote matrix, we can use singular value decomposition to reduce the vote matrix to two dimensions so that we can plot points for each senator. SVD works by trying to combine information (variance) from the multiple columns into less columns.

We end up with this for the 113th Congress:

x and y are our two dimensional singular values that represent our vote matrices. label is the party of the senator. label_code is the numeric representation of the party (1 is Democrat, 3 is Republican, 2 is Independent). `state is the state the senator is from.

Once we have these singular values, we can use them to plot our original chart:

Interesting observations

• From the chart, we can see that there is significant polarization in the Senate. In fact, there is a dividing line between the two parties.
• Both independents seem to vote solidly democrat.
• Massachussetts has some really out there senators (full disclosure: I live in MA right now)
• So does New Jersey
• Collins (R-ME), Murkowski (R-AK), Chiesa (R-NJ), Machin (D-WV), and Pryor (D-AR), are the closest things to centrists in the Senate.
• There are solid voting clusters around the party leaderships of both parties.
• The party line seems to come before all else, judging by how closely voting aligns by party.

There are other interesting things in this chart. Feel free to let me know if you notice anything good.

But wait, there’s more!

Now that we have these vote matrices, we can do all manner of cool things. One of the cool things we can do is calculate the euclidean distance between the votes of each Senator and the average votes on all issues. The greater the distance, the more “radical”, or extreme in their views, a senator is.

Here are all the senators, this time sorted by their distances:

We can also make a graphic of the most “extreme” senators given this distance:

Note that the minority party is more likely to be extreme by this distance metric, because the typical view is titled towards the party with more votes.

Using historical data

We can also calculate how polarized the parties have been by calculating how “extreme” the average member in each party was at any given time.

We can see how polarization has changed over time, and the average distance of each member to the typical voting pattern has shifted.

Thoughts

This analysis was interesting to do, and I hope to do more in the future. You can find all of my code here.

Some cautions:

• I would hesitate to make any sweeping generalizations from this that are not supported by the data.
• What you see is pretty much what you get. All of this data is publicly available, and I highly encourage you to look at it if you are interested.

Ideas:

• It could be interesting to analyze voting patterns as compared to the text of bills. (Does senator X always vote for bills with the phrase “increase defense spending” in them?)
• Voting patterns vs demographic shifts in the US.
• Linking voting patterns and the rise and fall of political parties.
• Predicting state senate votes.
• Doing similar analysis for the House.