Getting help on the internet

As mentioned earlier: this chapter will not cover it all. You will - at some point - need to search for help. First of all: a lot of what you will need is covered in the book “R for data science”. Try that first.

A second very important point is that R has a very active online community. You will find solutions to your problems in a range of online discussion forums, blogs, various cours pages, online books, tutorials and so forth. A major advantage of R is exactly this: there is so much free online resources and a very helful and friendly online community! But it can be overwhelming at first, and it can be hard to navigate. (And as in all other online forums, you should search the forum for existing solutions before opening a new topic that might already be solved).

The various flavors of R-coding

You remember we have used the package “tidyverse” a lot? This is a collection of packages which provides an entire set of functions to handle data in a logical and consistent way. It is almost like a programming language in its own, inside R.

That implies there are other ways of doing the same things. Some of the suggested solutions you find online will not be the tidyverse-way, but “base”, “data.table” or some other approach. In forums such as Stackoverflow, someone might have a problem, someone else present a solution that works well, and then: others add futher solutions often starting by saying “For completeness…”. And the do the same thing just doing it differently. You will often find solutions in both “base”, “tidyverse” og “data.table” - and some more.

If you want to avoid being too confused, look for the tidyverse-solutions as we have used in this course. Sometimes, it is referred to as “tidy” or “dplyr” as well. The package “dplyr” is the main package in the “tidyverse” for datamanagement. The “tidy” refers to tidy-principles for data handling which the tidyverse is built upon. Also: any solutions using the pipe-symbol, %>%, is probably tidyverse, as this is a tidyverse-invention. (Other apporaches have pipes as well, but written differently).

This applies to books as well! Introductory textbooks in R are quite often written in base-R. While this is fine, I do not think is a particularly good idea if one should also prepare for a modern approach to practical data analyses. At some point in professional life, you might need to learn base-R, but hey: you can wait!

So, before you buy an expensive book, be aware that it might use a different approach than this one. As long as it works, it might not matter, but on the other hand: it is a lot easier to work consistently in one approach.

A lot is free

• You can learn a lot from the tidyverse homepage.
• Check out the online version of R for data science
• The company Rstudio provides some very good cheat sheets using tidy-approach
• Check out free online books written in R at bookdown.org

Why not just make life easier for everyone?

My preferred choice for all permanent storeage of data is plain text, where the variables are delimited using either tab, comma or semi-colon. If so, any software can open and use it. It also means you will never have problems with compatibility across platforms, reading older, deprecated formats and so on.

Those used to work in e.g. SPSS or Stata might say that if you do that, all the formatting would be lost. In these softwares one would typically use “labeles” which links the values of the variable to a text-string indicating what it means. Such formats is not supported by plain text format. I would say that is not really a limitation of the format, but of the person or organization storing the data. There are ways of handling this.

It might be more convenient using such formats supporting labels for some uses, but all permanent storing should be based on textfiles. Both the values and labels should be stored. Labels can be e.g. be stored either as separate variables in the dataset or as separate textfiles with coding lists designed for easy reading for recoding in any software. (No, a pdf documentation is not the same thing).

If you ever get a job where you need to store data permanently, you really need to find out how to do it this way.

Importing common data file formats

All imports of data in R implies reading data and put it into an object. We have covered some basics in the earlier section of R basics on reading data. You need to use a different function for each type of format. (There exists multiple functions for each format, but we only cover some basics here).

We rely on the haven-package, which is a part of the tidyverse, to read Stata, SPSS and SAS files. These have very similar names and syntax, where you specify the filepath and filename with appropriate ending.

• SPSS: read_sav("filepath/filename.sav")
• Stata: read_dta("filepath/filename.dta")
• SAS: read_sas("filepath/filename.sas7bdat")

As encountered earlier, for csv-files we use the basic read.csv() and for RDS-files we use readRDS().

It should be noted that importing data from foreign formats can be slow, but mainly noticeable for large files. For very large files, there are speedier functions for at least csv-files, and there also exist special formats to handle very large files quicker. R can also connect to a wide range of databases. We do not cover such topics here, but focus on the types of data you are more likely to encounter in forseable future as a student of social sciences.

There are also commersial software available to transfer datafiles from one format to another. One such is Stat/Transfer which solves most such issues. However, those who store data for sharing should ideally take care to store the data so that one avoids such problems. Alas, that is not always the case.

## Rows: 49,519
## Columns: 3
## $ isco08  <dbl+lbl> 3333, 7122, 4221, 4311, 6130, 7212, 5131, 5223, 6112, 4224…
## $ gndr    <dbl+lbl> 1, 1, 2, 1, 2, 1, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 1…
## $ polintr <dbl+lbl> 3, 2, 4, 3, 2, 2, 4, 3, 3, 4, 2, 2, 1, 4, 2, 1, 2, 3, 2, 3…

Exercises: Import

Ex 1. Import data in Stata format (.dta) using read_dta()

dta <- read____("data/politics.___")

Ex 2. Import data in SPSS format (.sav) using read_sav()

Ex 3. Import data in SAS format (.sas7bdat) using read_sas()

sas <- read____("data/politics.sas7bdat")

Ex 4. Import data in csv format using read.csv. (OBS! Use punctuation). To make sure all variables are interpreted as character, set colClasses = "character". The separator is semi-colon, so specify that in sep = " "

csv <- read.___("data/politics.csv", sep=";", colClasses="character")

Character, numeric, and factors

Numeric are digits on a continuous scale. In short: variables where it makes sense to calculate the mean and standard deviations. These can be stored with varying degrees of presicion, and will appear as double, numberic, float or integers. In almost every situation, these are just continuous, and how it is stored does not matter much. (At least in social sciences. If you later start doing physics, it probably matters).

Character are variables consisting of text strings, or plain text if you will. They will appear with apostrophes around it, such as “high education” or “low social class” etc. They might also have numeric values, but treated as text. For example, municipality codes are numbers, but not really numeric, and would normally be represented as text. Oslo has the code “0301”. If it had been numeric the leading zero would make it appear as 301. Might not be such a big deal? However, the city area “Gamle Oslo” has the code “01”, and the full code is a combination of those two: “030101”. If you combine 301 and 1, you need to increase the number of digits and insert an additional zero. Worse: to extract the two last digits from the full code “030101” is harder. If stored as a string, there are functions to do such things easily.

Factors are categorical variables with a numeric representation, but an attached meaning to each “level”. In this way, the variable can be used for calculations, and R will automatically create dummy-variables when appropriate. You would often neeed to convert character variables to factors when doing analyses.

Dates is a special type of numeric variable to represent date and times. We will skip these for now, but you sometimes bump into these and should be aware that they are special. The package lubridate provides a lot of functions the handle dates and times.

Potential problem with numberic values for categorical data

In the above example using when reading Statafiles, you might have noticed that all the variables are numeric. Clearly, that is not quite correct. At least isco08 and gndr are categorical. (Would you compute the average occupation type in the population??). This is how it is stored in the original files, so there is nothing wrong with the data as such. If someone is to blame, it is those persons who made the file.

There is one potential problem with storing isco08 as numeric. It is a standard coding scheme and you might take a look at Statistics Norway standard classification pages to get a grip of it here or the international page for the international standard classifications of occupation.

If the two first digits are “01”, it means “Armed forces”, but the numberic representation would then be 1. If the first digit is 1, it means “leader position”. Thus, if extracting the first digit from the full code, you might get some unexpected errors. They are avoidable also in the numeric representation, but easier avoided if it was a text string to begin with, as “0” would then be the first digit.

But this is how the world of data is: not ideal. Those who created the dataset did not have you in mind spesifically, and might have been sloppy as well. More likely: the quality control ensured that such mistakes are not actually possible for these data - even though you couldn’t necessarily know that. We’ll assume this is ok. Anyway, we need to deal with it. These data are labelled, so that should be easy.

Labelled data

We use glimpse() to take a look at the data.

glimpse(polit) 

## Rows: 49,519
## Columns: 3
## $ isco08  <dbl+lbl> 3333, 7122, 4221, 4311, 6130, 7212, 5131, 5223, 6112, 4224…
## $ gndr    <dbl+lbl> 1, 1, 2, 1, 2, 1, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 1…
## $ polintr <dbl+lbl> 3, 2, 4, 3, 2, 2, 4, 3, 3, 4, 2, 2, 1, 4, 2, 1, 2, 3, 2, 3…

Notice that for each variable it is indicated what kind of variable it is. The ‘isco08’ is marked as <dbl+lbl>. The first indicates it is a numeric variable of the kind “double” (which is just a technical detail about how data is stored: it is a numeric variable). The second indicates that the variable is “labelled”. The standard way of storing data in some software is to have labels attached to them, and this is done here. That means that the variable has a value and a label indicating what the value means.

Labelled variable type is not the same thing as a factor, although it might sound similar! A label is more like a sticker, glued to the variable, but not the variable value as such. For data management, you need to use the values. While R can handle labels, it is rarely used in R, and it will not show on most standard outputs.

The first observation has isco08-code 3333, which means “Employment agents and contractors”. For the variable gndr, the value is 1, which means “Male”, and so forth.

As you will see below, for the isco08, we will need the values. To do recoding on long text strings like this would be a true mess. The codes are meaningful in themselves, and will eases some recoding a lot. For gndr, we can just make it into a factor-variable, but for the others we should keep the original values.

Let’s make a couple of versions of these variables to be used when needed. To do so, we use mutate(). This can be done in one mutate-sentence, but in the following, mutate is repeated three times. It is not necessary, but it is done here because we are using a little trick: .after= is for putting the new variable a specific place, just after the original variable. This is done for isco08 and polintr so that these versions of the same variable is next to each other. The placement of the variable does not matter for analysis, so it is just because it can be convenient when looking at the data. Afterwards, we use mutate again, just to make the original isco08 and gndr into factor variables. Note that the order this is done matters: we have to create the factor variable of isco08 last because as.character() on a factor variable will give unexpected results.

polit <- read_dta("data/politics.dta")
polit <- polit %>% 
  mutate(isco08_code = as.character(isco08), .after=isco08) %>% 
  mutate(polintr_fct = as_factor(polintr), .after = polintr)  %>% 
  mutate(isco08 = as_factor(isco08),
         gndr_fct = as_factor(gndr))  %>% 
  relocate(gndr, .after = last_col())

glimpse(polit) 

## Rows: 49,519
## Columns: 6
## $ isco08      <fct> "Employment agents and contractors", "Floor layers and til…
## $ isco08_code <chr> "3333", "7122", "4221", "4311", "6130", "7212", "5131", "5…
## $ polintr     <dbl+lbl> 3, 2, 4, 3, 2, 2, 4, 3, 3, 4, 2, 2, 1, 4, 2, 1, 2, 3, …
## $ polintr_fct <fct> Hardly interested, Quite interested, Not at all interested…
## $ gndr_fct    <fct> Male, Male, Female, Male, Female, Male, Male, Female, Male…
## $ gndr        <dbl+lbl> 1, 1, 2, 1, 2, 1, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, 2, …

Let’s make some tables. We start by using a very basic code for making tables: table() and the corresponding addmargins() to get the totals as well.

polit %>% 
  select(polintr, gndr) %>% 
  table() %>% 
  addmargins()

Looks good! But if you look closely, and the total number of observations are less than reported in glimpse() above. The reason is missing values.

Let’s try to make tables recommended in a previous chapter. We will also rather use the factor-versions of these variables.

polit <- read_dta("data/politics.dta") %>% 
  mutate(isco08_code = as.character(isco08), .after=isco08) %>% 
  mutate(polintr_fct = as_factor(polintr), .after = polintr)  %>% 
  mutate(isco08 = as_factor(isco08),
         gndr_fct = as_factor(gndr))  %>% 
  relocate(gndr, .after = last_col())

polit %>% 
  select(polintr_fct, gndr_fct) %>% 
  tbl_summary(by = gndr_fct) %>% 
  add_overall()

A new row appears at the bottom: “Unknown”. These are not included in the percentages, though. This is because these are the missing values as shown in the documentation shown above. These values are defined as should not be used in computations.

It can be convenient to define missing values in this way, but mainly for categorical variables. If you do it for continuous variables, make sure the indicator for missing is not a potential true value as that might cause problems later.

ggplot( polit, aes(x=isco08)) +
  geom_bar()+
  theme(axis.text.x = element_text(angle = -90, size = .5, vjust = 3))

Damn! That was not very informative at all. There are just too many values. We have to do something about that. And what is that one occupation which is so much larger than anyone else?

Let’s start identifying what kind of occupation is the largest one. An earlier chapter gives you the tools to do so, so look at that and fill in the gaps below. The code should count the number of observations for each level of isco08, and then sort the data with the largest group first.

polit %>% 
  group_by(...) %>% 
  ...(count = n()) %>% 
  ...(desc(count))

Ouch. The largest occupation group is missing. NA means “not available”.

More on missing values

For some of the responsdents, there might be missing values. Either the respondendt did not answer, or perhaps the questionaire was designed so that not everybody got all the questions.

In survey data, it is common to also code reasons for not responding, or including response options that indicates that the respondendt do not want to answer. En example is the variable polintr as displayed in the documentation:

This means that only values 1 through 4 are actually valid responses to the question. The other responses can be regarded as reasons to not respond. Sometimes,

na_if

Simplify with dummy-coding

Let’s start with the variable on political interest. If we are to study how political interest vary by occupation and gender, then polintr is the outcome variable. One approach is to focus on those who are interested in politics, so, one might choose to focus on those who answered very or quite interested vs the rest. That implies simplifying to only two categories: interested vs the rest. Then those with missing values would reasonably go into the “rest” group.

In the following exercise

polit_d <- polit %>% 
  mutate(interest = as.numeric(polintr %in% c(1, 2))) 

polit_d %>% 
  select(polintr_fct, interest) %>% 
  tbl_cross()

Factor-variables

Hierarchical structured string variables

Sometimes, a categorical variable have houndreds of values. One such example is the international standard classifications of occupation where each kind of occupation has a specific code. There are many types of occupations, so there are about 800 codes. For most analyses, this is way too detailed, and you need to group into a smaller number of main categories. Statistics Norway provides an easy accessible overview of these codes here. In short, there is a 4-digit code, where the first position is a rough grouping, and the following positions implies further details within the overall group.

You could write up recoding as explained above, but that will take quite a few lines of code. It will take a lot of time, it is easy to make mistakes - and it is a nightmare to de-bug to find errors. In short: that approach is out of the question. (That has nothing to do with R, and you should not do so in any software).

Rather, we just extract the first position from the text string. The function str_sub() is made for extracting parts of texts, and by specifying from where to start and end the extraction. In this case, it is just the first position.

Using catalogues for re-coding

Sometimes, such long list of unique codes have a standard grouping that are not hierarchical. One such example is the ordc class schema as developed by the HISTCLASS-project homepage. This is a coding schema for isco-codes to classes, and a catalogue is available from the homepage. As there are about 800 unique values and they are stored in Excel-format with corresponding grouping for each code. The first lines in Excel-file looks lik this:

This file can be read into R using read_excel(), where the first line is typically read as variable names. But the first lines are not to be used, and neither are several columns.

To read it into R, we need to skip the lines not needed and change the variable names. In this example, we only need the isco-kodes and the ordc_yrk codes. Variable names should not include spaces, so we include an argument for ensuring universal valid variable names. That substitutes white space with punctuation. We also specify col_types = "text" to avoid the values being interpreted as numeric.

isco <- readxl::read_excel(path = "data/3_codes_isco88_ordc.xlsx", skip = 4, .name_repair = "universal", col_types = "text") %>% 
  select(2,9) 

## New names:
## * `ISCO-88 Title EN` -> ISCO.88.Title.EN
## * `ISCO-88 code` -> ISCO.88.code
## * `ISCO 08 Code` -> ISCO.08.Code
## * `ISCO-08 part` -> ISCO.08.part
## * `ISCO-08 Title EN` -> ISCO.08.Title.EN
## * ...

head(isco)

Now, we can merge the data with this catalogue. So that every record in the catalogue is merged to each record with the same code. To do this, we use left_join(), and store in a new object.

isco <- isco %>% 
  rename(isco08_code = ISCO.88.code)

polit2 <- left_join(polit, isco, by = "isco08_code")

head(polit2)

What happened here is that the recoding happened almost automatically by adding a new column with the new variable.

Now, you can make e.g. a cross-tabulation of social class by gender.

polit2 %>% 
  select(ORDC_YRK, gndr) %>% 
  tbl_summary(by = gndr)

## ℹ Column(s) gndr are class "haven_labelled". This is an intermediate datastructure not meant for analysis. Convert columns with `haven::as_factor()`, `labelled::to_factor()`, `labelled::unlabelled()`, and `unclass()`. "haven_labelled" value labels are ignored when columns are not converted. Failure to convert may have unintended consequences or result in error.

## • https://haven.tidyverse.org/articles/semantics.html

## • https://larmarange.github.io/labelled/articles/intro_labelled.html#unlabelled

conditional assignments

Graphics

Descriptives and basic comparisons

Regression