Introduce dlookr

Choonghyun Ryu

2025-09-01

Source: vignettes/Introduce.Rmd

Preface

After you have acquired the data, you should do the following:

  • Diagnose data quality.
    • If there is a problem with data quality,
    • The data must be corrected or re-acquired.
  • Explore data to understand the data and find scenarios for performing the analysis.
  • Derive new variables or perform variable transformations.

The dlookr package makes these steps fast and easy:

  • Performs a data diagnosis or automatically generates a data diagnosis report.
  • Discover data in various ways and automatically generate EDA(exploratory data analysis) reports.
  • Impute missing values and outliers, resolve skewed data, and binaries continuous variables into categorical variables. And generates an automated report to support it.

dlookr increases synergy with dplyr. Particularly in data exploration and data wrangling, it increases the efficiency of the tidyverse package group.

Supported data structures

Data diagnosis supports the following data structures.

  • data frame: data.frame class.
  • data table: tbl_df class.
  • table of DBMS: table of the DBMS through tbl_dbi
    • Use dplyr as the back-end interface for any DBI-compatible database.

List of supported tasks of data analytics

Diagnose Data

Overall Diagnose Data

Tasks Descriptions Functions Support DBI
describe overview of data Inquire basic information to understand the data in general overview()
summary overview object summary described overview of data summary.overview()
plot overview object plot described overview of data plot.overview()
diagnose data quality of variables The scope of data quality diagnosis is information on missing values and unique value information diagnose() x
diagnose data quality of categorical variables frequency, ratio, rank by levels of each variables diagnose_category() x
diagnose data quality of numerical variables descriptive statistics, number of zero, minus, outliers diagnose_numeric() x
diagnose data quality for outlier number of outliers, ratio, mean of outliers, mean with outliers, mean without outliers diagnose_outlier() x
plot outliers information of numerical data box plot and histogram whith outliers, without outliers plot_outlier.data.frame() x
plot outliers information of numerical data by target variable box plot and density plot whith outliers, without outliers plot_outlier.target_df() x
diagnose combination of categorical variables Check for sparse cases of level combinations of categorical variables diagnose_sparese()

Visualize Missing Values

Tasks Descriptions Functions Support DBI
pareto chart for missing value visualize the Pareto chart for variables with a missing value. plot_na_pareto()
combination chart for missing value visualize the distribution of missing value by combining variables. plot_na_hclust()
plot the combination variables that is include missing value visualize the combinations of missing value across cases plot_na_intersect()

Reporting

Types Descriptions Functions Support DBI
report the information of data diagnosis into a PDF file report the information for diagnosing the data quality diagnose_report() x
reporting the information of data diagnosis into HTML file report the information for diagnosing the quality of the data diagnose_report() x
reporting the information of data diagnosis into HTML file dynamic report the information for diagnosing the quality of the data diagnose_web_report() x
reporting the information of data diagnosis into PDF and HTML files paged report the information for diagnosing the quality of the data diagnose_paged_report() x

EDA

Univariate EDA

Types Tasks Descriptions Functions Support DBI
categorical summaries frequency tables univar_category()
categorical summaries chi-squared test summary.univar_category()
categorical visualize bar charts plot.univar_category()
categorical visualize bar charts plot_bar_category()
numerical summaries descriptive statistics describe() x
numerical summaries descriptive statistics univar_numeric()
numerical summaries descriptive statistics of standardized variable summary.univar_numeric()
numerical visualize histogram, box plot plot.univar_numeric()
numerical visualize Q-Q plots plot_qq_numeric()
numerical visualize box plot plot_box_numeric()
numerical visualize histogram plot_hist_numeric()

Bivariate EDA

Types Tasks Descriptions Functions Support DBI
categorical summaries frequency tables cross cases compare_category()
categorical summaries contingency tables, chi-squared test summary.compare_category()
categorical visualize mosaics plot plot.compare_category()
numerical summaries correlation coefficient, linear model summaries compare_numeric()
numerical summaries correlation coefficient, linear model summaries with threshold summary.compare_numeric()
numerical visualize scatter plot with marginal box plot plot.compare_numeric()
numerical Correlate correlation coefficient correlate() x
numerical Correlate summaries with correlation matrix summary.correlate() x
numerical Correlate visualization of a correlation matrix plot.correlate() x
both PPS PPS(Predictive Power Score) pps() x
both PPS summaries with PPS summary.pps() x
both PPS visualization of a PPS matrix plot.pps() x

Normality Test

Types Tasks Descriptions Functions Support DBI
numerical summaries Shapiro-Wilk normality test normality() x
numerical summaries normality diagnosis plot (histogram, Q-Q plots) plot_normality() x

Relationship between target variable and predictors

Target Variable Predictor Descriptions Functions Support DBI
categorical categorical contingency tables relate() x
categorical categorical mosaics plot plot.relate() x
categorical numerical descriptive statistic for each levels and total observation relate() x
categorical numerical density plot plot.relate() x
categorical categorical bar charts plot_bar_category()
numerical categorical ANOVA test relate() x
numerical categorical scatter plot plot.relate() x
numerical numerical simple linear model relate() x
numerical numerical box plot plot.relate() x
categorical numerical Q-Q plots plot_qq_numeric()
categorical numerical box plot plot_box_numeric()
categorical numerical histogram plot_hist_numeric()

Reporting

Types Descriptions Functions Support DBI
reporting the information of EDA into PDF file reporting the information of EDA eda_report() x
reporting the information of EDA into HTML file reporting the information of EDA eda_report() x
reporting the information of EDA into PDF file dynamic reporting the information of EDA eda_web_report() x
reporting the information of EDA into HTML file paged reporting the information of EDA eda_paged_report() x

Transform Data

Find Variables

Types Descriptions Functions Support DBI
missing values find the variable that contains the missing value in the object that inherits the data.frame find_na()
outliers find the numerical variable that contains outliers in the object that inherits the data.frame find_outliers()
skewed variable find the numerical variable that is the skewed variable that inherits the data.frame find_skewness()

Imputation

Types Descriptions Functions Support DBI
missing values missing values are imputed with some representative values and statistical methods. imputate_na()
outliers outliers are imputed with some representative values and statistical methods. imputate_outlier()
summaries calculate descriptive statistics of the original and imputed values. summary.imputation()
visualize the imputation of a numerical variable is a density plot, and the imputation of a categorical variable is a bar plot. plot.imputation()

Binning

Types Descriptions Functions Support DBI
binning converts a numeric variable to a categorization variable binning()
summaries calculate frequency and relative frequency for each levels(bins) summary.bins()
visualize visualize two plots on a single screen. The plot at the top is a histogram representing the frequency of the level. The plot at the bottom is a bar chart representing the frequency of the level. plot.bins()
optimal binning categorizes a numeric characteristic into bins for ulterior usage in scoring modeling binning_by()
summaries summary metrics to evaluate the performance of binomial classification model summary.optimal_bins()
visualize generates plots for understand distribution, bad rate, and weight of evidence after running binning_by() plot.optimal_bins()
infogain binning categorizes a numeric characteristic into bins for multi-class variables using recursive information gain ratio maximization binning_rgr()
visualize generates plots for understanding distribution and distribution by target variable after running binning_rgr() plot.infogain_bins()
evaluate calculates metrics to evaluate the performance of binned variable for binomial classification model performance_bin()
summaries summary metrics to evaluate the performance of binomial classification model after performance_bin() summary.performance_bin()
visualize It generates plots to understand frequency, WoE by bins using performance_bin after running binning_by() plot.performance_bin()
visualize extract bins from “bins” and “optimal_bins” objects extract.bins()

Diagnose Binned Variable

Types Descriptions Functions Support DBI
diagnosis performs diagnose performance that calculates metrics to evaluate the performance of binned variable for binomial classification model performance_bin()
summaries summary method for “performance_bin”. summary metrics to evaluate the performance of the binomial classification model summary.performance_bin()
visualize visualize for understanding frequency, WoE by bins using performance_bin and something else plot.performance_bin()

Transformation

Types Descriptions Functions Support DBI
transformation performs variable transformation for standardization and resolving skewness of numerical variables transform()
summaries compares the distribution of data before and after data transformation summary.transform()
visualize visualize two kinds of a plot by attribute of the ‘transform’ class. The transformation of a numerical variable is a density plot plot.transform()

Reporting

Types Descriptions Functions Support DBI
reporting the information of transformation into PDF reporting the information of transformation transformation_report()
reporting the information of transformation into HTML reporting the information of transformation transformation_report()
reporting the transformation information into PDF dynamic reporting the transformation information transformation_web_report()
reporting the information of transformation into HTML paged reporting the information of transformation transformation_paged_report()

Miscellaneous

Statistics

Types Descriptions Functions Support DBI
statistics calculate the entropy entropy()
statistics calculate the skewness of the data skewness()
statistics calculate the kurtosis of the data kurtosis()
statistics calculate the Jensen-Shannon divergence between two probability distributions jsd()
statistics calculate the Kullback-Leibler divergence between two probability distributions kld()
statistics calculate the Cramer’s V statistic between two categorical(discrete) variables cramer()
statistics calculate the Theil’s U statistic between two categorical(discrete) variables theil()
statistics finding percentile of a numerical variable. get_percentile()
statistics transform a numeric vector using several methods like “log”, “sqrt”, “log+1”, “log+a”, “1/x”, “x^2”, “x^3”, “Box-Cox”, “Yeo-Johnson” get_transform()
statistics calculate the Cramer’s V statistic cramer()
statistics calculate the Theil’s U statistic theil()

Programming

Types Descriptions Functions Support DBI
programming extracts variable information having a certain class from an object inheriting data.frame find_class()
programming gets class of variables in data.frame or tbl_df get_class()
programming retrieves the column information of the DBMS table through the tbl_bdi object of dplyr get_column_info()
programming finding the user machine’s OS. get_os()
programming import Google fonts import_google_font()