---
title: "Web APIs"
subtitle: "Lecture 13"
author: "Dr. Colin Rundel"
footer: "Sta 323 - Spring 2025"
format:
  revealjs:
    theme: slides.scss
    transition: fade
    slide-number: true
    self-contained: true
execute:
  echo: true
  warning: true
engine: knitr
---

```{r setup}
#| message: False
#| warning: False
#| include: False
options(
  width=80
)

knitr::opts_chunk$set(
  fig.align = "center", fig.retina = 2, dpi = 150,
  out.width = "100%",
  message = TRUE
)

library(tidyverse )
library(rvest)
```


## URLs

![](imgs/url-structure.png){fig-align="center" width="100%"}


::: {.aside}
From [HTTP: The Protocol Every Web Developer Must Know](http://code.tutsplus.com/tutorials/http-the-protocol-every-web-developer-must-know-part-1--net-31177)
:::


## Query Strings

Provides named parameter(s) and value(s) that modify the behavior of the resulting page. 

<br/>

Format generally follows:

::: {.center .large}
`?arg1=value1&arg2=value2&arg3=value3`
:::

. . .

<br/>

Some quick examples,

::: {.small}
* <http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address=1600+Amphitheatre+Parkway>

* <https://swapi.dev/api/people/?search=r2>
:::


## URL encoding

This is will often be handled automatically by your web browser or other tool, but it is useful to know a bit about what is happening

* Spaces will encoded as '+' or '%20'

* Certain characters are reserved and will be replaced with the percent-encoded version within a URL

::: {.small}
| !   | #   | $   | &   | '   | (   | )   |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| %21 | %23 | %24 | %26 | %27 | %28 | %29 |
| *   | +   | ,   | /   | :   | ;   | =   |
| %2A | %2B | %2C | %2F | %3A | %3B | %3D |
| ?   | @   | [   | ]   |
| %3F | %40 | %5B | %5D |
:::

* Characters that cannot be converted to the correct charset are replaced with HTML numeric character references (e.g. a &#931; would be encoded as &amp;#931; )

## Examples



::: {.small}
```{r}
URLencode("http://lmgtfy.com/?q=hello world")
URLdecode("http://lmgtfy.com/?q=hello%20world")
```
:::

. . .

::: {.small}
```{r}
URLencode("!#$&'()*+,/:;=?@[]")
URLencode("!#$&'()*+,/:;=?@[]", reserved = TRUE)
URLencode("!#$&'()*+,/:;=?@[]", reserved = TRUE) |> 
  URLdecode()
```
:::

. . .

::: {.small}
```{r}
URLencode("Σ")
URLdecode("%CE%A3")
```
:::


# RESTful APIs

## REST

*RE*presentational *S*tate *T*ransfer 

* describes an architectural style for web services (not a standard)

* all communication via HTTP requests

* Key features: 
    - client–server architecture
    - addressible (specific URL endpoints)
    - stateless (no client information stored between requests)
    - layered / hierarchical
    - cacheability




## GitHub API

GitHub provides a REST API that allows you to interact with most of the data available on the website.

There is extensive documentation and a huge number of endpoints to use - almost anything that can be done on the website can also be done via the API.

::: {.center}
[GitHub REST API](https://docs.github.com/en/rest)
:::


# Demo 1 - GitHub API <br/> Basic access 

<br/><br/>

::: {.center}
[Get a user](https://docs.github.com/en/rest/users/users?apiVersion=2022-11-28#get-a-user)

[List organization repositories](https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-organization-repositories)
:::

# httr2

## Background

`httr2` is a package designed around the construction and handling of HTTP requests and responses. It is a rewrite of the `httr` package and includes the following features:

* Pipeable API

* Explicit request object, with support for

    * rate limiting
    
    * retries
    
    * OAuth
    
    * Secrure secret storage

* Explicit response object, with support for

    * error codes / reporting
    
    * common body encoding (e.g. json, etc.)


## Structure of an HTTP Request

<br/>

![](imgs/HTTP_Request.png){fig-align="center" width="100%"}

::: {.aside}
[Source](https://www3.ntu.edu.sg/home/ehchua/programming/webprogramming/http_basics.html)
:::

## HTTP Methods / Verbs

* *GET* - fetch a resource

* *POST* - create a new resource

* *PUT* - full update of a resource

* *PATCH* - partial update of a resource

* *DELETE* - delete a resource.

Less common verbs: *HEAD*, *TRACE*, *OPTIONS*.

::: {.aside}
Based on [HTTP: The Protocol Every Web Developer Must Know](http://code.tutsplus.com/tutorials/http-the-protocol-every-web-developer-must-know-part-1--net-31177)
:::


## httr2 request objects

A new request object is constructed via `request()` which is then modifed via `req_*()` functions

Some useful functions:

* `request()` - initialize a request object

* `req_method()` - set HTTP method

* `req_url_query()` - add query parameters to URL

* `req_url_*()` - add or modify URL

* `req_body_*()` - set body content (various formats and sources)

* `req_user_agent()` - set user-agent

* `req_dry_run()` - shows the exact request that will be made




## Structure of an HTTP Response

<br/>

![](imgs/HTTP_Response.png){fig-align="center" width="100%"}

::: {.aside}
[Source](https://www3.ntu.edu.sg/home/ehchua/programming/webprogramming/http_basics.html)
:::

## Status Codes

* 1xx: Informational Messages

* 2xx: Successful

* 3xx: Redirection

* 4xx: Client Error

* 5xx: Server Error



## httr2 response objects

Once constructed a request is made via `req_perform()` which returns a response object (the most recent response can also be retrieved via `last_response()`). Content of the response are accessed via the `resp_*()` functions

Some useful functions:

* `resp_status()` - extract HTTP status code 

* `resp_status_desc()` - return a text description of the status code

* `resp_content_type()` - extract content type and encoding

* `resp_body_*()` - extract body from a specific format (`json`, `html`, `xml`, etc.)

* `resp_headers()` - extract response headers



# Demo 2 - httr2 + GitHub