issue with utf-8 weird chars

[gbekes]

Great package

Used development version.
Data has a variable with variety of cities around globe, UTF-8. (Windows, RStudio), can't share, sry.

It runs through despite errors: with weird characters, e.g. html gives you
“Wiener Neustädt”

[annennenne]

Thanks again for submitting the issue.

I think I'll need a minimal example that produces the error in order to find a solution. I'm having trouble getting the error myself.

Does the following code produce the problem for you?

a <- data.frame(cities = c(rep("Copenhagen", 2), "Budapest", "Wiener Neustädt"),
 num = c(1, 2, 1, 3))
library(dataReporter)
makeDataReport(a, file = "deleteme.rmd", output = "html", replace = TRUE)

And if so, would you mind sharing the output of devtools::session_info()?

[annennenne]

For future self: We discussed the issue further via email, seems like the problem was local. I was not able to reproduce it even with the original data.

[vorpalvorpal]

I am getting the same issue. Your European city example gives the following error:

Data report generation is finished. Please wait while your output file is being rendered.
Error in sub(re, "", x, perl = TRUE) : input string 2 is invalid UTF-8
In addition: Warning messages:
1: In readLines(con, warn = FALSE) :
 invalid input found on input connection 'deleteme.rmd'
2: In xfun::read_utf8(input) :
 The file deleteme.rmd is not encoded in UTF-8. These lines contain invalid UTF-8 characters: 94, 102

deleteme.rmd shows Neust?dt in place of Neustädt if I open it in rstudio (defualts to opening Rmds as utf-8). If I choose to reopen it with ISO-8859-1 (system default encoding) Neustädt displays correctly. This is a stupid windows problem I've run up against before.

Looking at the docs for rmarkdown::render(), the rmarkdown document produced is always UTF-8 (there is an encoding argument, but it is actually ignored). However the default option for file() is encoding = getOption("encoding") which by default is "native.enc". Thus when the native encoding isn't UTF-8 it will save the UTF-8 document in the native encoding (eg. ISO-8859-1) leading to any fancy characters being mis-rendered. Because the output of rmarkdown::render() is always going to be UTF-8, the encoding set by file() should be set to explicitly be "UTF-8". So in makeDataReport.R calls to file should be:

fileConn <- file(file, "w", encoding = "UTF-8") #for main document
vListConn <- file(vListFileName, "w", encoding = "UTF-8")

I haven't tested that the change actually works, but I think that it should do.

output of devtools::session_info():

- Session info -----------------------------------------------------------------------------------------------------------
 setting value
 version R version 4.1.1 (2021-08-10)
 os Windows 10 x64 (build 15063)
 system x86_64, mingw32
 ui RStudio
 language (EN)
 collate English_Australia.1252
 ctype English_Australia.1252
 tz Australia/Sydney
 date 2022-02-01
 rstudio 2021.09.0+351 Ghost Orchid (desktop)
 pandoc 2.14.0.3 @ C:/Users/XXX/scoop/apps/rstudio/current/bin/pandoc/ (via rmarkdown)

- Packages ---------------------------------------------------------------------------------------------------------------
 ! package * version date (UTC) lib source
 askpass 1.1 2019-01-13 [1] CRAN (R 4.1.1)
 assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.1.1)
 backports 1.3.0 2021-10-27 [1] CRAN (R 4.1.1)
 base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.1.1)
 broom * 0.7.10 2021-10-31 [1] CRAN (R 4.1.1)
 bslib 0.3.1 2021-10-06 [1] CRAN (R 4.1.2)
 cachem 1.0.6 2021-08-19 [1] CRAN (R 4.1.1)
 callr 3.7.0 2021-04-20 [1] CRAN (R 4.1.1)
 cellranger 1.1.0 2016-07-27 [1] CRAN (R 4.1.1)
 checkmate 2.0.0 2020-02-06 [1] CRAN (R 4.1.1)
 CHNOSZ 1.4.1 2021-04-09 [1] CRAN (R 4.1.2)
 class 7.3-19 2021-05-03 [1] CRAN (R 4.1.1)
 classInt 0.4-3 2020-04-07 [1] CRAN (R 4.1.1)
 cli 3.1.0 2021-10-27 [1] CRAN (R 4.1.1)
 cluster 2.1.2 2021-04-17 [1] CRAN (R 4.1.1)
 codetools 0.2-18 2020-11-04 [1] CRAN (R 4.1.1)
 colorspace 2.0-2 2021-06-24 [1] CRAN (R 4.1.1)
 crayon * 1.4.2 2021-10-29 [1] CRAN (R 4.1.2)
 crosstalk 1.1.1 2021-01-12 [1] CRAN (R 4.1.1)
 curl 4.3.2 2021-06-23 [1] CRAN (R 4.1.1)
 cusumcharter 0.1.0 2021-11-15 [1] CRAN (R 4.1.2)
 data.table 1.14.2 2021-09-27 [1] CRAN (R 4.1.1)
 data.tree 1.0.0 2020-08-03 [1] CRAN (R 4.1.1)
 dataReporter * 1.0.2 2021-11-11 [1] CRAN (R 4.1.2)
 DBI 1.1.1 2021-01-15 [1] CRAN (R 4.1.1)
 dbplyr 2.1.1 2021-04-06 [1] CRAN (R 4.1.1)
 DEoptimR 1.0-10 2022-01-03 [1] CRAN (R 4.1.2)
 desc 1.4.0 2021-09-28 [1] CRAN (R 4.1.2)
 devtools 2.4.3 2021-11-30 [1] CRAN (R 4.1.2)
 diffdf * 1.0.4 2020-03-17 [1] CRAN (R 4.1.1)
 digest 0.6.28 2021-09-23 [1] CRAN (R 4.1.1)
 dplyover * 0.0.8.9002 2021-11-01 [1] Github (TimTeaFan/dplyover@f0cd984)
 dplyr * 1.0.7 2021-06-18 [1] CRAN (R 4.1.1)
 DT 0.19 2021-09-02 [1] CRAN (R 4.1.1)
 e1071 1.7-9 2021-09-16 [1] CRAN (R 4.1.1)
 editData * 0.1.8 2021-04-02 [1] CRAN (R 4.1.1)
 ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.1)
 evaluate 0.14 2019-05-28 [1] CRAN (R 4.1.1)
 exifr * 0.3.2 2021-03-20 [1] CRAN (R 4.1.1)
 fansi 0.5.0 2021-05-25 [1] CRAN (R 4.1.1)
 farver 2.1.0 2021-02-28 [1] CRAN (R 4.1.1)
 fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.1)
 flextable * 0.6.9 2021-10-07 [1] CRAN (R 4.1.1)
 forcats * 0.5.1 2021-01-27 [1] CRAN (R 4.1.1)
 foreign 0.8-81 2020-12-22 [1] CRAN (R 4.1.1)
 Formula 1.2-4 2020-10-16 [1] CRAN (R 4.1.1)
 fs 1.5.0 2020-07-31 [1] CRAN (R 4.1.1)
 gdtools 0.2.3 2021-01-06 [1] CRAN (R 4.1.1)
 generics 0.1.1 2021-10-25 [1] CRAN (R 4.1.1)
 ggforce * 0.3.3 2021-03-05 [1] CRAN (R 4.1.1)
 ggh4x * 0.2.0 2021-08-21 [1] CRAN (R 4.1.1)
 ggiraph * 0.7.10 2021-05-19 [1] CRAN (R 4.1.2)
 ggplot2 * 3.3.5 2021-06-25 [1] CRAN (R 4.1.1)
 ggrepel 0.9.1 2021-01-15 [1] CRAN (R 4.1.1)
 glue * 1.4.2 2020-08-27 [1] CRAN (R 4.1.1)
 GQAnalyzer * 0.1.0 2021-11-01 [1] Github (khaors/GQAnalyzer@d51540c)
 gridExtra 2.3 2017-09-09 [1] CRAN (R 4.1.1)
 gtable 0.3.0 2019-03-25 [1] CRAN (R 4.1.1)
 hablar * 0.3.0 2020-03-19 [1] CRAN (R 4.1.1)
 haven 2.4.3 2021-08-04 [1] CRAN (R 4.1.1)
 here * 1.0.1 2020-12-13 [1] CRAN (R 4.1.1)
 highr 0.9 2021-04-16 [1] CRAN (R 4.1.1)
 Hmisc 4.6-0 2021-10-07 [1] CRAN (R 4.1.1)
 hms 1.1.1 2021-09-26 [1] CRAN (R 4.1.1)
 htmlTable 2.3.0 2021-10-12 [1] CRAN (R 4.1.1)
 htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.1.1)
 htmlwidgets 1.5.4 2021-09-08 [1] CRAN (R 4.1.1)
 httpuv 1.6.3 2021-09-09 [1] CRAN (R 4.1.1)
 httr 1.4.2 2020-07-20 [1] CRAN (R 4.1.1)
 janitor * 2.1.0 2021-01-05 [1] CRAN (R 4.1.1)
 jpeg 0.1-9 2021-07-24 [1] CRAN (R 4.1.1)
 jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.1.1)
 jsonlite 1.7.2 2020-12-09 [1] CRAN (R 4.1.1)
 KernSmooth 2.23-20 2021-05-03 [1] CRAN (R 4.1.1)
 knitr 1.36 2021-09-29 [1] CRAN (R 4.1.1)
 labeling 0.4.2 2020-10-20 [1] CRAN (R 4.1.1)
 labelled * 2.9.0 2021-10-29 [1] CRAN (R 4.1.2)
 later 1.3.0 2021-08-18 [1] CRAN (R 4.1.1)
 lattice 0.20-44 2021-05-02 [1] CRAN (R 4.1.1)
 latticeExtra 0.6-29 2019-12-19 [1] CRAN (R 4.1.1)
 leafem 0.1.6 2021-05-24 [1] CRAN (R 4.1.1)
 leaflet * 2.0.4.1 2021-01-07 [1] CRAN (R 4.1.1)
 leafpm * 0.1.0 2019-03-13 [1] CRAN (R 4.1.1)
 librarian 1.8.1 2021-07-12 [1] CRAN (R 4.1.1)
 lifecycle 1.0.1 2021-09-24 [1] CRAN (R 4.1.1)
 lubridate * 1.8.0 2021-10-07 [1] CRAN (R 4.1.1)
 magrittr * 2.0.1 2020-11-17 [1] CRAN (R 4.1.1)
 mapedit * 0.6.0 2020-02-02 [1] CRAN (R 4.1.1)
 mapview * 2.10.0 2021-06-05 [1] CRAN (R 4.1.1)
 MASS 7.3-54 2021-05-03 [1] CRAN (R 4.1.1)
 Matrix 1.3-4 2021-06-01 [1] CRAN (R 4.1.1)
 memoise 2.0.1 2021-11-26 [1] CRAN (R 4.1.2)
 mgcv 1.8-36 2021-06-01 [1] CRAN (R 4.1.1)
 mime 0.12 2021-09-28 [1] CRAN (R 4.1.1)
 miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.1.1)
 modelr 0.1.8 2020-05-19 [1] CRAN (R 4.1.1)
 munsell 0.5.0 2018-06-12 [1] CRAN (R 4.1.1)
 nlme 3.1-152 2021-02-04 [1] CRAN (R 4.1.1)
 nnet 7.3-16 2021-05-03 [1] CRAN (R 4.1.1)
 officer * 0.4.1 2021-11-14 [1] CRAN (R 4.1.2)
 openxlsx * 4.2.4 2021-06-16 [1] CRAN (R 4.1.1)
 pander * 0.6.4 2021-06-13 [1] CRAN (R 4.1.2)
 pdftools 3.0.1 2021-05-06 [1] CRAN (R 4.1.1)
 pillar 1.6.4 2021-10-18 [1] CRAN (R 4.1.1)
 pkgbuild 1.3.1 2021-12-20 [1] CRAN (R 4.1.2)
 pkgcond * 0.1.1 2021-04-28 [1] CRAN (R 4.1.1)
 pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.1)
 pkgload 1.2.4 2021-11-30 [1] CRAN (R 4.1.2)
 plyr 1.8.6 2020-03-03 [1] CRAN (R 4.1.1)
 png 0.1-7 2013-12-03 [1] CRAN (R 4.1.1)
 polyclip 1.10-0 2019-03-14 [1] CRAN (R 4.1.1)
 pracma 2.3.3 2021-01-23 [1] CRAN (R 4.1.1)
 prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.1.1)
 processx 3.5.2 2021-04-30 [1] CRAN (R 4.1.1)
 promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.1.1)
 proxy 0.4-26 2021-06-07 [1] CRAN (R 4.1.1)
 ps 1.6.0 2021-02-28 [1] CRAN (R 4.1.1)
 purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.1.1)
 qpdf * 1.1 2019-03-07 [1] CRAN (R 4.1.1)
 R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.1)
 rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.1.1)
 raster 3.5-11 2021-12-23 [1] CRAN (R 4.1.2)
 RColorBrewer * 1.1-2 2014-12-07 [1] CRAN (R 4.1.1)
 Rcpp 1.0.7 2021-07-07 [1] CRAN (R 4.1.1)
 readr * 2.0.2 2021-09-27 [1] CRAN (R 4.1.1)
 readxl * 1.3.1 2019-03-13 [1] CRAN (R 4.1.1)
 remotes 2.4.1 2021-09-29 [1] CRAN (R 4.1.1)
 repr 1.1.3 2021-01-21 [1] CRAN (R 4.1.1)
 reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.1)
 rhandsontable * 0.3.8 2021-05-27 [1] CRAN (R 4.1.1)
 rio 0.5.27 2021-06-21 [1] CRAN (R 4.1.1)
 D rJava 1.0-5 2021-09-24 [1] CRAN (R 4.1.1)
 rlang 0.4.12 2021-10-18 [1] CRAN (R 4.1.1)
 rlist * 0.4.6.2 2021-09-03 [1] CRAN (R 4.1.1)
 rmarkdown 2.11 2021-09-14 [1] CRAN (R 4.1.1)
 robustbase 0.93-9 2021-09-27 [1] CRAN (R 4.1.2)
 rpart 4.1-15 2019-04-12 [1] CRAN (R 4.1.1)
 rprojroot 2.0.2 2020-11-15 [1] CRAN (R 4.1.1)
 rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.1.1)
 rvest 1.0.2 2021-10-16 [1] CRAN (R 4.1.1)
 sass 0.4.0 2021-05-12 [1] CRAN (R 4.1.1)
 satellite 1.0.4 2021-10-12 [1] CRAN (R 4.1.1)
 scales 1.1.1 2020-05-11 [1] CRAN (R 4.1.1)
 sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.2)
 sf * 1.0-3 2021-10-07 [1] CRAN (R 4.1.1)
 shiny * 1.7.1 2021-10-02 [1] CRAN (R 4.1.1)
 shinyjs * 2.0.0 2020-09-09 [1] CRAN (R 4.1.1)
 shinyWidgets * 0.6.2 2021-09-17 [1] CRAN (R 4.1.1)
 skimr * 2.1.3 2021-03-07 [1] CRAN (R 4.1.1)
 snakecase 0.11.0 2019-05-25 [1] CRAN (R 4.1.1)
 SOfun * 1.76 2021-11-01 [1] Github (mrdwab/SOfun@e41fa62)
 sp 1.4-6 2021-11-14 [1] CRAN (R 4.1.2)
 splitstackshape * 1.4.8 2019-04-21 [1] CRAN (R 4.1.1)
 staplr * 3.1.1 2021-01-11 [1] CRAN (R 4.1.1)
 stringdist * 0.9.8 2021-09-09 [1] CRAN (R 4.1.1)
 stringi * 1.7.5 2021-10-04 [1] CRAN (R 4.1.1)
 stringr * 1.4.0 2019-02-10 [1] CRAN (R 4.1.1)
 survival 3.2-11 2021-04-26 [1] CRAN (R 4.1.1)
 systemfonts 1.0.3 2021-10-13 [1] CRAN (R 4.1.1)
 terra 1.5-12 2022-01-13 [1] CRAN (R 4.1.1)
 tesseract * 4.1.2 2021-09-18 [1] CRAN (R 4.1.1)
 testthat 3.1.1 2021-12-03 [1] CRAN (R 4.1.2)
 tibble * 3.1.5 2021-09-30 [1] CRAN (R 4.1.1)
 tidyr * 1.1.4 2021-09-27 [1] CRAN (R 4.1.1)
 tidyselect * 1.1.1 2021-04-30 [1] CRAN (R 4.1.1)
 tidyverse * 1.3.1 2021-04-15 [1] CRAN (R 4.1.1)
 tweenr 1.0.2 2021-03-23 [1] CRAN (R 4.1.1)
 tzdb 0.1.2 2021-07-20 [1] CRAN (R 4.1.1)
 units * 0.7-2 2021-06-08 [1] CRAN (R 4.1.1)
 usethis 2.1.5 2021-12-09 [1] CRAN (R 4.1.2)
 utf8 1.2.2 2021-07-24 [1] CRAN (R 4.1.1)
 uuid * 0.1-4 2020-02-26 [1] CRAN (R 4.1.1)
 vctrs 0.3.8 2021-04-29 [1] CRAN (R 4.1.1)
 webchem * 1.1.1 2021-02-07 [1] CRAN (R 4.1.1)
 webshot 0.5.2 2019-11-22 [1] CRAN (R 4.1.1)
 whoami 1.3.0 2019-03-19 [1] CRAN (R 4.1.2)
 withr 2.4.2 2021-04-18 [1] CRAN (R 4.1.1)
 xfun 0.27 2021-10-18 [1] CRAN (R 4.1.1)
 xml2 1.3.2 2020-04-23 [1] CRAN (R 4.1.1)
 xtable 1.8-4 2019-04-21 [1] CRAN (R 4.1.1)
 yaml 2.2.1 2020-02-01 [1] CRAN (R 4.1.1)
 zeallot * 0.1.0 2018-01-28 [1] CRAN (R 4.1.1)
 zip 2.2.0 2021-05-31 [1] CRAN (R 4.1.1)
 zoo * 1.8-9 2021-03-09 [1] CRAN (R 4.1.1)

 [1] C:/Users/XXX/scoop/apps/r/4.1.1/library

 D -- DLL MD5 mismatch, broken installation.

[vorpalvorpal]

In the meantime for anyone else encountering this issue until the fix is published, it can be resolved locally by setting:

options(encoding = "UTF-8")

[annennenne]

Thank you for this thorough and excellent suggestion. We will definitely look into making these changes the next time we work on updates!