Skip to contents

A survey’s quality rules live inside the form — constraint, required, calculate — and they are written once, in the spreadsheet, by the person who knows what a plausible answer looks like. The problem is that nothing in an analysis script runs them: the data frame you read back is whatever the devices sent, and the rules that were supposed to guard it are XML the script never opens.

rxeval runs them. This walkthrough uses the small household form shipped with the package, and a submission with problems planted on purpose.

library(rxeval)
library(dplyr)

form       <- system.file("extdata", "visit.xml", package = "rxeval")
submission <- system.file("extdata", "visit-submission.xml", package = "rxeval")

Before the field: will the form travel?

ODK forms are evaluated by two implementations — JavaRosa inside Collect and KoboCollect, Enketo inside web forms — and neither is a superset of the other. The gaps do not announce themselves: an expression Collect cannot evaluate usually yields nothing rather than an error, so the form fills in, the interview finishes, and a column comes back empty.

form_portability() reads the form the way both engines would:

form_portability(form) |>
  as_tibble()
#> # A tibble: 1 × 8
#>   path                 rule  expression construct breaks effect suggestion says 
#>   <chr>                <chr> <chr>      <chr>     <chr>  <chr>  <chr>      <chr>
#> 1 /data/youngest_ente… calc… /data/res… last()    Colle… JavaR… count() o… /dat…

The verdict worth filtering for is breaks. "both, differently" is the worst value it takes — nothing fails anywhere, and the two engines quietly compute different answers. Here the form has a milder, common case: a calculate built on last(), which Enketo evaluates and JavaRosa does not have at all.

form_portability(form) |>
  filter(breaks == "Collect / KoboCollect") |>
  select(path, construct, suggestion)
#>                     path construct            suggestion
#> 1 /data/youngest_entered    last() count() of the repeat

Every issue carries a suggestion when there is one to give: a rewrite that means one thing in both engines. Zero rows means the form travels.

After the field: what do the form’s own rules say?

submission_findings() evaluates the questionnaire’s rules against collected data and reports what does not hold:

findings <- submission_findings(form, submission)

findings |>
  as_tibble()
#> # A tibble: 3 × 3
#>   path                   kind        says                                       
#>   <chr>                  <chr>       <chr>                                      
#> 1 /data/youngest_entered calculation "/data/youngest_entered: holds \"\", but t…
#> 2 /data/residents        constraint  "/data/residents: More than 20 residents n…
#> 3 /data/respondent       required    "/data/respondent: required, and unanswere…

Three findings, three different kinds of trouble:

findings |>
  count(kind)
#>          kind n
#> 1 calculation 1
#> 2  constraint 1
#> 3    required 1
  • a constraint the device should have enforced and, on this submission, did not — 25 residents against a rule of at most 20;
  • a required question that came back unanswered;
  • a calculation whose stored value is not what the form computes — the last() issue found above, no longer a prediction but a hole in a column.

A rule the engine could not evaluate comes back as kind = "not-evaluated", never as a pass: a rule that did not run has not been satisfied, and reporting it as clean would be the one failure worth avoiding.

Over a whole round, the same call is one lapply away:

list.files("submissions", full.names = TRUE) |>
  lapply(\(s) submission_findings(form, s) |> mutate(file = basename(s))) |>
  bind_rows() |>
  count(path, kind, sort = TRUE)

Trying a rule by hand

eval_expression() answers what one expression does before it goes into the spreadsheet — against a real submission, in the same dialect the devices speak:

eval_expression("count(/data/resident[age < 18])", submission)
#> [1] "1"

eval_expression("if(/data/residents > 20, 'needs a note', 'fine')", submission)
#> [1] "needs a note"

An expression the engine cannot evaluate raises an error rather than returning a plausible value — the same rule as everywhere else in rxeval: silence is never a verdict.

Where the rules come from

Every portability rule in the engine was decided by putting the same expression to JavaRosa and to Enketo’s evaluator and reading the two answers — the method, and the receipts, are on the project site.