Preparing the data • vizsurvey

library(vizsurvey)

Introduction

This vignette presents the data preparation required to use {vizsurvey} through the runVizsurvey_from_folder call. The goal is to enable regular, structured use of the tool within a team managing several surveys.

In this context, launching the interface manually for each database quickly becomes tedious. Moreover, computing the gaps, especially when they are broken down by survey wave or by filter, can be resource-intensive.

To make this process easier, {vizsurvey} offers several automated preparation functions. This vignette describes how they work, step by step, from structuring the files to generating the objects needed to launch the application.

Example data

EU-SILC is an annual survey coordinated by Eurostat. It is used to measure income, poverty, inequality and living conditions in European countries. Eurostat provides Public Use Files that can be used freely for testing. The files are anonymised and do not necessarily reproduce the official results. We provide two years of the EU-SILC survey for Belgium, for the household file. There is no interviewer number in these files initially, so a fictitious number NR_ITW was created for the examples here.

The files can be found in the package source files at inst/extdata/SILC, or downloaded here.

Configuring a survey

Your database file must be readable by the data.table::fread function. This function automatically recognises standard formats (CSV, TSV, etc.). Place your file in a dedicated directory, where it will be the only file of that format.

If your survey is made up of several files (for example one file per year or per country, as for the EU-SILC survey), you can place them in the same directory. All files must, however, contain the same variables as those listed in the configuration file, and be in the same format. In addition, no other file of that type must be present in the directory.

Survey configuration file

Each data directory must contain a configuration file named config.txt. This file tells {vizsurvey} how to interpret and structure your data. It must include the following elements:

name_survey = 

vars_discretes =
vars_continous =

var_wave = 
var_filter = 

var_intvwr =

name_survey: name of the survey (useful for identifying the files);
vars_discretes: variables to be treated as categorical;
vars_continous: variables to be treated as continuous;
var_wave: survey wave variable used to break down the calculations (example: year);
var_filter: filter variable allowing an additional breakdown (example: province). The calculations are also produced without any filter distinction;
var_intvwr: interviewer identifier variable.

The file can be created manually, or generated automatically with the create_config function. Here is the example for the household files of the SILC survey, with the year (HB010) as the survey wave and the province (HB020) as the filter:

create_config(
  folder_path = "inst/extdata/SILC/HFILE",
  name_survey = "SILC-H",
  var_wave    = "HB010",
  var_filter    = "HB020",
  var_intvwr   = "NR_ITW"
)

Two levels for the wave or the filter

var_wave and var_filter can contain two variables, separated by a comma. The second one is then a second level, nested in the first one: for example the year and the quarter of the survey.

var_wave = YEAR, QUARTER
var_filter = REGION, DEGURBA

create_config(
  folder_path = "inst/extdata/SILC/HFILE",
  name_survey = "SILC-H",
  var_wave    = c("YEAR", "QUARTER"),
  var_filter  = "HB020",
  var_intvwr  = "NR_ITW"
)

In this case, {vizsurvey} builds a complete key by concatenating the levels (2024 / T1), and computes the statistics twice: once for the first level alone (the year), once for the complete key (the year and the quarter). The preparation is therefore longer, roughly in proportion to the number of modalities added.

In the interface, the first level is a mandatory choice and the second one is optional (All by default), displayed in cascade below the first. Choosing only the year gives the analysis at the year level; adding the quarter restricts the analysis, and the comparisons (the other waves) are then also made at the year-and-quarter level.

The same applies to runVizsurvey_from_r:

runVizsurvey_from_r(df,
                    var_intvwr = "NR_ITW",
                    var_wave   = c("YEAR", "QUARTER"),
                    var_filter = "REGIO")

The discrete and continuous variables only need to be specified if the classify_df function misidentifies their type. This function automatically determines the type of each variable, based on its format and number of categories. The threshold for classifying a variable as categorical is set to 15 by default.

classify_df(iris)
#> # A tibble: 5 × 2
#>   variable     type      
#>   <chr>        <chr>     
#> 1 Petal.Length Continuous
#> 2 Petal.Width  Continuous
#> 3 Sepal.Length Continuous
#> 4 Sepal.Width  Continuous
#> 5 Species      Modal

Optional: interviewer summary file

In addition to the microdata, you can place an optional summary file containing one row per interviewer in the survey directory. This file gathers interviewer-level indicators that are not derived from the response distributions themselves, for example the number of completed interviews, the average interview duration, or the response rate. It must include the interviewer identifier (the same one used in var_intvwr) so that it can be matched to the microdata.

To use it, declare its filename in config.txt with the file_synthesis key, or pass it to create_config:

file_synthesis = synthesis.csv

create_config(
  folder_path    = "inst/extdata/SILC/HFILE",
  name_survey    = "SILC-H",
  var_wave       = "HB010",
  var_filter     = "HB020",
  var_intvwr     = "NR_ITW",
  file_synthesis = "synthesis.csv"
)

When such a file is present, it is automatically integrated and analysed during preparation, so that any atypical interviewer-level indicator is flagged as a potential anomaly, in the same way as the distributional gaps.

Preparing the survey

Simple structure: a single directory

Once the configuration is in place, {vizsurvey} can compute in advance all the statistics needed to monitor the survey. When you have only a single survey directory, the prepa_survey function uses this directory, containing the data and the config.txt file, to produce:

per wave, the descriptive statistics of each variable;
per wave, for each variable, the gaps of each interviewer relative to the population (within the wave);
per wave and per filter, for each variable, the gaps of each interviewer relative to the population (within the filter and the wave).

By default, prepa_survey looks for the CSV files in the directory, but the file_pattern argument lets you adapt this behaviour. A global.rds file is then generated in the same folder: it is the only file that will be used by the interactive interface.

prepa_survey(
  folder_path  = "inst/extdata/SILC/HFILE",
  file_pattern = "*.csv")

The application can then be launched simply with the runVizsurvey_from_folder function.

runVizsurvey_from_folder("inst/extdata/SILC/HFILE", depth_folder = 1)

The prepa_survey function can be called each time your survey data are updated. Its execution can be scheduled so that all the statistics on your surveys are computed daily.

Double structure: several directories

If you manage several surveys, you can create the directories side by side following this structure:

data/
  ├── ENQ1/
  │   ├── *.csv
  │   └── config.txt
  └── ENQ2/
      ├── *.csv
      └── config.txt

You can then prepare all your surveys with the prepa_surveys function. This function acts as a wrapper around prepa_survey and automatically applies the preparation to all the child directories of the specified path. In this case, you need to set depth_folder = 2.

prepa_surveys(folder_path  = "inst/extdata/SILC", depth_folder = 2)

You can then launch the interface by specifying the same depth level:

runVizsurvey_from_folder("inst/extdata/SILC", depth_folder = 2)

Triple structure: several directory levels

It is also possible to manage a full survey tree, with several hierarchical levels. The first level may correspond to the different surveys conducted and, as a second level, one directory per type of file produced by that survey. For example, the SILC survey contains four types of files, two at the household level and two at the individual level. The HBS survey contains a survey file and a file for expenditure diaries, and so on. Here is an example of the expected directory structure:

data/
  ├── ENQ1/
  │   ├── ENQ1-A/
  │   │   └── ...
  │   └── ENQ1-B/
  │       └── ...
  └── ENQ2/
      ├── ENQ2-A/
      │   └── ...
      └── ENQ2-B/
          └── ...

Each sub-directory must contain its own config.txt and its own data files. Within each sub-directory, all files must have the same data structure. You can then run the global preparation:

prepa_surveys(folder_path  = "data", depth_folder = 3)

Finally, the interface is launched with the same depth:

runVizsurvey_from_folder("data", depth_folder = 3)

If the data are updated, for example during fieldwork, you simply need to run the prepa_surveys function again to obtain an updated version of the global.rds objects, and thus a full update of the interactive interface.

Conclusion

Data preparation is an essential step before fully using {vizsurvey} within an organisation that runs surveys on a daily basis. It optimises the responsiveness of the application and guarantees that all survey managers analyse the same up-to-date files.

Once the files are structured, configured and prepared, each launch of the interface becomes immediate: all the necessary statistics are already computed and stored in the global.rds file. This approach ensures an efficient reuse of the data, whatever the number of surveys or the complexity of their tree structure. It also facilitates collaborative work within a team: each member can explore the results without having to know R.