The diagnose() produces information for diagnosing the quality of the variables of data.frame or tbl_df.
diagnose(.data, ...)
# S3 method for data.frame
diagnose(.data, ...)
# S3 method for grouped_df
diagnose(.data, ...)a data.frame or a tbl_df or a grouped_df.
one or more unquoted expressions separated by commas. You can treat variable names like they are positions. Positive values select variables; negative values to drop variables. If the first expression is negative, diagnose() will automatically start with all variables. These arguments are automatically quoted and evaluated in a context where column names represent column positions. They support unquoting and splicing.
An object of tbl_df.
The scope of data quality diagnosis is information on missing values and unique value information. Data quality diagnosis can determine variables that require missing value processing. Also, the unique value information can determine the variable to be removed from the data analysis.
The information derived from the data diagnosis is as follows.:
variables : variable names
types : data type of the variable or to select a variable to be corrected or removed through data diagnosis.
integer, numeric, factor, ordered, character, etc.
missing_count : number of missing values
missing_percent : percentage of missing values
unique_count : number of unique values
unique_rate : ratio of unique values. unique_count / number of observation
See vignette("diagonosis") for an introduction to these concepts.
# \donttest{
# Diagnosis of all variables
diagnose(jobchange)
#> # A tibble: 14 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 enrollee_id char… 0 0 19158 1
#> 2 city fact… 0 0 123 0.00642
#> 3 city_dev_index nume… 0 0 93 0.00485
#> 4 gender fact… 4508 23.5 4 0.000209
#> 5 relevent_experi… fact… 0 0 2 0.000104
#> 6 enrolled_univer… fact… 386 2.01 4 0.000209
#> 7 education_level orde… 460 2.40 6 0.000313
#> 8 major_discipline fact… 2813 14.7 7 0.000365
#> 9 experience orde… 65 0.339 23 0.00120
#> 10 company_size orde… 5938 31.0 9 0.000470
#> 11 company_type fact… 6140 32.0 7 0.000365
#> 12 last_new_job orde… 423 2.21 7 0.000365
#> 13 training_hours inte… 0 0 241 0.0126
#> 14 job_chnge fact… 0 0 2 0.000104
# Select the variable to diagnose
diagnose(jobchange, gender, experience, training_hours)
#> # A tibble: 3 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 gender factor 4508 23.5 4 0.000209
#> 2 experience ordered 65 0.339 23 0.00120
#> 3 training_hours integer 0 0 241 0.0126
diagnose(jobchange, -gender, -experience, -training_hours)
#> # A tibble: 11 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 enrollee_id char… 0 0 19158 1
#> 2 city fact… 0 0 123 0.00642
#> 3 city_dev_index nume… 0 0 93 0.00485
#> 4 relevent_experi… fact… 0 0 2 0.000104
#> 5 enrolled_univer… fact… 386 2.01 4 0.000209
#> 6 education_level orde… 460 2.40 6 0.000313
#> 7 major_discipline fact… 2813 14.7 7 0.000365
#> 8 company_size orde… 5938 31.0 9 0.000470
#> 9 company_type fact… 6140 32.0 7 0.000365
#> 10 last_new_job orde… 423 2.21 7 0.000365
#> 11 job_chnge fact… 0 0 2 0.000104
diagnose(jobchange, "gender", "experience", "training_hours")
#> # A tibble: 3 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 gender factor 4508 23.5 4 0.000209
#> 2 experience ordered 65 0.339 23 0.00120
#> 3 training_hours integer 0 0 241 0.0126
diagnose(jobchange, 4, 9, 13)
#> # A tibble: 3 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 gender factor 4508 23.5 4 0.000209
#> 2 experience ordered 65 0.339 23 0.00120
#> 3 training_hours integer 0 0 241 0.0126
# Using pipes ---------------------------------
library(dplyr)
# Diagnosis of all variables
jobchange %>%
diagnose()
#> # A tibble: 14 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 enrollee_id char… 0 0 19158 1
#> 2 city fact… 0 0 123 0.00642
#> 3 city_dev_index nume… 0 0 93 0.00485
#> 4 gender fact… 4508 23.5 4 0.000209
#> 5 relevent_experi… fact… 0 0 2 0.000104
#> 6 enrolled_univer… fact… 386 2.01 4 0.000209
#> 7 education_level orde… 460 2.40 6 0.000313
#> 8 major_discipline fact… 2813 14.7 7 0.000365
#> 9 experience orde… 65 0.339 23 0.00120
#> 10 company_size orde… 5938 31.0 9 0.000470
#> 11 company_type fact… 6140 32.0 7 0.000365
#> 12 last_new_job orde… 423 2.21 7 0.000365
#> 13 training_hours inte… 0 0 241 0.0126
#> 14 job_chnge fact… 0 0 2 0.000104
# Positive values select variables
jobchange %>%
diagnose(gender, experience, training_hours)
#> # A tibble: 3 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 gender factor 4508 23.5 4 0.000209
#> 2 experience ordered 65 0.339 23 0.00120
#> 3 training_hours integer 0 0 241 0.0126
# Negative values to drop variables
jobchange %>%
diagnose(-gender, -experience, -training_hours)
#> # A tibble: 11 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 enrollee_id char… 0 0 19158 1
#> 2 city fact… 0 0 123 0.00642
#> 3 city_dev_index nume… 0 0 93 0.00485
#> 4 relevent_experi… fact… 0 0 2 0.000104
#> 5 enrolled_univer… fact… 386 2.01 4 0.000209
#> 6 education_level orde… 460 2.40 6 0.000313
#> 7 major_discipline fact… 2813 14.7 7 0.000365
#> 8 company_size orde… 5938 31.0 9 0.000470
#> 9 company_type fact… 6140 32.0 7 0.000365
#> 10 last_new_job orde… 423 2.21 7 0.000365
#> 11 job_chnge fact… 0 0 2 0.000104
# Positions values select variables
jobchange %>%
diagnose(4, 9, 13)
#> # A tibble: 3 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 gender factor 4508 23.5 4 0.000209
#> 2 experience ordered 65 0.339 23 0.00120
#> 3 training_hours integer 0 0 241 0.0126
# Negative values to drop variables
jobchange %>%
diagnose(-8, -9, -10)
#> # A tibble: 11 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 enrollee_id char… 0 0 19158 1
#> 2 city fact… 0 0 123 0.00642
#> 3 city_dev_index nume… 0 0 93 0.00485
#> 4 gender fact… 4508 23.5 4 0.000209
#> 5 relevent_experi… fact… 0 0 2 0.000104
#> 6 enrolled_univer… fact… 386 2.01 4 0.000209
#> 7 education_level orde… 460 2.40 6 0.000313
#> 8 company_type fact… 6140 32.0 7 0.000365
#> 9 last_new_job orde… 423 2.21 7 0.000365
#> 10 training_hours inte… 0 0 241 0.0126
#> 11 job_chnge fact… 0 0 2 0.000104
# Using pipes & dplyr -------------------------
# Diagnosis of missing variables
jobchange %>%
diagnose() %>%
filter(missing_count > 0)
#> # A tibble: 8 × 6
#> variables types missing_count missing_percent unique_count unique_rate
#> <chr> <chr> <int> <dbl> <int> <dbl>
#> 1 gender fact… 4508 23.5 4 0.000209
#> 2 enrolled_univers… fact… 386 2.01 4 0.000209
#> 3 education_level orde… 460 2.40 6 0.000313
#> 4 major_discipline fact… 2813 14.7 7 0.000365
#> 5 experience orde… 65 0.339 23 0.00120
#> 6 company_size orde… 5938 31.0 9 0.000470
#> 7 company_type fact… 6140 32.0 7 0.000365
#> 8 last_new_job orde… 423 2.21 7 0.000365
# Using group_by ------------------------------
# Calculate the diagnosis of all variables by 'job_chnge' using group_by()
jobchange %>%
group_by(job_chnge) %>%
diagnose()
#> # A tibble: 28 × 8
#> variables types job_chnge data_count missing_count missing_percent
#> <chr> <chr> <fct> <int> <dbl> <dbl>
#> 1 enrollee_id chara… No 14381 0 0
#> 2 enrollee_id chara… Yes 4777 0 0
#> 3 city factor No 14381 0 0
#> 4 city factor Yes 4777 0 0
#> 5 city_dev_index numer… No 14381 0 0
#> 6 city_dev_index numer… Yes 4777 0 0
#> 7 gender factor No 14381 3119 21.7
#> 8 gender factor Yes 4777 1389 29.1
#> 9 relevent_experience factor No 14381 0 0
#> 10 relevent_experience factor Yes 4777 0 0
#> # ℹ 18 more rows
#> # ℹ 2 more variables: unique_count <int>, unique_rate <dbl>
# }