CLEAN TEXT IN RSTUDIO HOW TO
CLEAN TEXT IN RSTUDIO CODE

Grep commands are fairly straight forward, and much more powerful and useful for subsetting rows or columns within larger datasets.

CLEAN TEXT IN RSTUDIO HOW TO

Here is a useful cheatsheet that includes more examples of how to use GREP to find patterns in text. GREP-style commands also include a wildcard which can be used to, for example, find all words in a string that start with a certain letter, such as “P”: some_text<-c('This','Professor','is','not','so','great') We can also pass two arguments at once using the | separator as follows: gsub(' | ', '', wikipedia_scrape) # 'Class of 2018: Senior Stories of Discovery, Learning and Serving' The first argument in the gsub function names the pattern we are looking for, the second (blank) argument tells us what we want to replace that pattern with, and the third argument is the strong we want to transform. Now let’s use the gsub command to remove all s from the string gsub(' ', '', wikipedia_scrape) # 'Class of 2018: Senior Stories of Discovery, Learning and Serving ' The output tells us that there was one occurrence of “Class.” The text within quotes is the pattern we are trying to find, and the second argument is the string we want to search within.

CLEAN TEXT IN RSTUDIO CODE

For example, let’s write some code that determines whether the word “Class” is part of our string using the grepl function in base R: grepl('Class', wikipedia_scrape) # TRUE Once again, GREP-style commands search for a certain pattern. Let’s create a character string that includes such characters as follows (the meaning of the text isn’t important- this was scraped from the Duke University web page “Events” section): wikipedia_scrape<- 'Class of 2018: Senior Stories of Discovery, Learning and Serving ' At the time, I mentioned that these are html tags, or chunks of code that tell your web browser how to display something (in this case a “tab” space and a new line). In that tutorial, we scraped a Wikipedia page and discovered that there were strange characters such as and interspersed throughout the text we scraped. To demonstrate why you need to learn some GREP, let’s return to an issue we encountered in a previous tutorial on screen-scraping. If you try to run the code below and receive error messages such as invalid multibyte string, this is indicative of a character encoding issue that you will most likely need to resolve using one of the imperfect steps above.Īnother important tool for working with text is GREP, which stands for “Globally search a regular expression and print.” In laymans terms, GREP is a tool that helps you search for the presence of a string of characters that matches a pattern. Inconsistent character encoding is one of the most common pitfalls for those attempting to learn how to perform quantitative text analysis in R, but there are no easy solutions. For example, if an old version of character encoding is applied to text that contains emoji, the emoji may appear as strings of seeminly incoherent symbols and punctuation marks. Note, however, that the latter function may create “place holders” for characters that it cannot process. The Encoding and inconv functions in base R can be very useful for the latter purposes. Before we begin working with a text-based dataset, it is useful to either a) make sure every text uses the same character encoding or b) use a tool to force or coerce all text into a single character encoding. Why should you care that text can be created using different forms of character encoding? Well, if you have scraped a large amount of data from multiple websites-or multiple social media sites- you may find that your data exist in multiple types of character encoding, and this can create a big hassle.

The figure below illustrates a form of character encoding called “Latin Extended-B” which was developed for representing text in languages derived from Latin (which of course excludes a number of important languages) There are presently dozens of different types of character encoding that resulted not only from advances in computing technology-and the development of different styles for different operating systems-but also for different languages (and even new languages such as emoji). Character encoding is a style of writing text in computer code that helps programs such as web browsers figure out how to display text. One of the first things that is important to learn about quantitative text analysis is to most computer programs, texts or strings also have a numerical basis called character encoding.

Clean text in rstudio