Managing Connections

Efficiently managing connections is the foundation of a robust data pipeline. ctoclient is designed to handle both simple single-server scripts and the multi-server setups common in large-scale research projects.

1. Authentication strategies

Security first. Never store passwords directly in your R scripts. If you share your code or push it to GitHub, your credentials go with it.

The .Renviron approach

The most common way to manage credentials is the .Renviron file.

Call usethis::edit_r_environ().
Add your credentials:

CTO_SERVER=myorg
CTO_USER=admin@example.com
CTO_PASS=mypassword123

Restart R for the change to take effect.
Connect:

library(ctoclient)

cto_connect(
  server   = Sys.getenv("CTO_SERVER"),
  username = Sys.getenv("CTO_USER"),
  password = Sys.getenv("CTO_PASS")
)

Note that .Renviron entries are plain NAME=value lines, not R code, and that values containing spaces or # should be quoted.

Interactive mode

If you are working locally and have not set up environment variables, omit the password and you will be prompted for it securely:

cto_connect(server = "myorg", username = "admin@example.com")

The system keyring

For the strongest option on a personal machine, keep the password in your operating system’s credential store with the keyring package. The secret never touches a file in your project:

# Once, interactively:
keyring::key_set("ctoclient", username = "admin@example.com")

# In every script from then on:
cto_connect(
  server   = "myorg",
  username = "admin@example.com",
  password = keyring::key_get("ctoclient", username = "admin@example.com")
)

2. What a session is

cto_connect() authenticates, verifies the credentials, and stores the resulting request object inside the package. Every other function picks it up on its own, which is why you never pass a connection around:

cto_connect("myorg", "admin@example.com")

# No connection argument anywhere
cto_form_ids()
cto_form_data("baseline_survey")
cto_dataset_list()

cto_is_connected() reports whether a session currently exists. It tells you that cto_connect() has been called, not that the server is reachable right now:

cto_is_connected()
#> [1] TRUE

Cookies

By default cto_connect() preserves cookies and handles the CSRF token for you. Keep it that way. A number of functions reach console endpoints rather than REST endpoints, and they abort on a session that carries no cookies:

cto_form_data()
cto_form_metadata() and cto_form_definition()
cto_form_languages(), cto_form_printable(), cto_form_stata_template() and cto_form_mail_template()
cto_metadata()

cto_form_dofile() and cto_form_docx() are in the same position indirectly, since both download the form definition first. In other words, cookies = FALSE leaves you with the dataset functions, the attachment downloads and the listing functions, and little else.

3. Working with more than one server

Some projects move data between servers, or aggregate across organizations. cto_connect() returns the session invisibly, so you can capture each one and switch between them with cto_set_connection().

conn_staging <- cto_connect("org-staging", "admin@example.com")
conn_prod    <- cto_connect("org-prod", "admin@example.com")

# The most recent connect() is the active one, so this reads from prod
data_prod <- cto_form_data("baseline_survey")

# Switch to staging and write the same data to a dataset there
cto_set_connection(conn_staging)
cto_dataset_upload("aggregated_data", file = "data/baseline.csv")

Because the active session is global, a script that switches servers should switch deliberately and close to the call that depends on it. Interleaving reads from two servers without an intervening cto_set_connection() is the easiest way to fetch from the wrong one.

cto_dataset_upload() takes a path to a CSV in its file argument. To upload a data frame you have in memory, write it out first:

tmp <- tempfile(fileext = ".csv")
readr::write_csv(data_prod, tmp)
cto_dataset_upload("aggregated_data", file = tmp)

4. Customizing the underlying request

Every session is an httr2 request object, so anything httr2 can do to a request you can do to a connection before handing it back to the package.

library(httr2)

custom <- conn_prod |>
  req_user_agent("MyResearchBot/1.0") |>
  req_retry(max_tries = 5) |>
  req_timeout(120)

cto_set_connection(custom)
cto_form_data("my_form")

This is the supported way to change retry behaviour, timeouts, proxies or headers. Two things to keep in mind: the package sets its own throttle (30 requests per minute) which you should only raise if you know your server tolerates it, and SurveyCTO rejects parallel requests from the same account with a 409, so a retry policy is more useful to you than concurrency.

5. Verbosity

Most functions report progress through the console. To silence them — in a scheduled job, say — set the option:

options(ctoclient.verbose = FALSE)

Set it back to TRUE, or unset it, to get the messages again.