From hourly readings to daily sensor patterns

Analysis overviewone shared calendar, five separate fits

Question

What daily patterns are shared across the sensors, and which components track weather, time or the reference analysers?

The supplied analysis consists of the data file, a Python script, its saved results and interactive plots. These pages reorganise that material and show the saved PCA and UMAP results unchanged.

Input and output

Input: each sensor’s hourly response, its raw output signal rather than a gas concentration. Output, for each sensor: a table of 357 days × 24 hours, PCA scores and loadings, UMAP coordinates, and correlations with daily covariates (weather, calendar and reference-gas averages).

Source details: Dataset →

Parse and select complete daysfrom hourly rows to a shared calendar

Step	Choice	Result
Read	Semicolon separator; decimal comma	9,471 rows and 17 columns before cleanup
Clean	Drop 114 blank rows and 2 empty unnamed columns	9,357 hourly records; 15 columns
Missing	Treat −200 as absent data	Keep absence distinct from a low response
Segment	Require 24 hourly rows with nonmissing sensor values	389 full dates before completeness screening
Align	Intersect complete dates across all five sensors	357 complete days kept; 32 full dates excluded

Missing sensor readings are not filled in. The first and last dates in the file are partial days. Daily averages of the reference gases use the hours with readings on each day, so some rest on fewer than 24 hours.

Inspect the five design matricessame row = same date; column = hour

Each sensor’s readings become one table, its design matrix: one row per complete day (357 days, 11 March 2004 to 3 April 2005) and one column per hour (0–23), so each cell is the sensor’s response in one hour. The 357 days are 91.3% of the 391 calendar dates in the file. Each sensor on its own has the same 357 complete days, so row i is the same date in all five tables.

Values stay in each sensor’s own response scale. Read across a row for one day’s 24-hour profile, and down a column for how one hour changes from day to day. Sort rows by reorders all five tables in the same way; the Direction button reverses the order.

Sort rows by

Direction

low responsehigh response

Each table has its own colour scale, from its 2nd to its 98th percentile, because the sensors share no common unit: a colour means low or high for that sensor.

Hover over any table to see that row’s date, its position, its sort value and every sensor’s daily mean; a marker shows the same row in all five tables.

PCA and UMAP settingstwo views of the same raw daily profiles

What this step does. Each complete day is described by 24 numbers, one per hour. PCA (principal component analysis) rotates these numbers onto new axes, ordered by how much of the day-to-day variation each one captures. The axes are the principal components PC1, PC2 and so on; a day’s position along an axis is its score, and the weight the axis gives each hour is its loading. UMAP is a nonlinear method that keeps each point’s nearest neighbours close together: it places every day on a 2D map near the days with the most similar 24-hour profiles. Both methods are fitted separately for each sensor.

Method	Recorded configuration	Interpretation
PCA	10 components; full SVD; mean-centred columns; no standardisation	Explained variance and loadings describe each sensor’s own daily patterns
UMAP	2 dimensions; n_neighbors=15; min_dist=0.1; Euclidean; random_state=42	Fitted directly to the 24-hour profiles, not to PCA scores
Fit scope	One fit per sensor on all 357 complete days	The Period menu shows a subset of days from the same fit
Elbow	Maximum perpendicular distance to the PC1–PC10 chord	Marks where the scree curve bends most, as a rule of thumb for how many components matter

Sensor	PC1	PC2	PC3	Elbow	Cumulative share at elbow
PT08.S1	65.3%	10.1%	7.7%	PC2	75.3%
PT08.S2	62.3%	9.9%	9.1%	PC2	72.2%
PT08.S3	61.3%	17.5%	7.9%	PC3	86.8%
PT08.S4	77.6%	8.2%	4.4%	PC2	85.8%
PT08.S5	63.3%	11.0%	8.4%	PC2	74.3%

For four sensors the elbow is at PC2. S3 is the exception: its PC2 holds 17.5% of the variance, and its elbow is at PC3.

PCA subtracts each hourly column’s mean but does not rescale the columns, and UMAP works on the raw profiles. Days therefore differ both in their overall level and in the size of their daily swings, and both count.

How to read the plotsvariance, loadings, scores and linked days

Start with the scree plot. Its bars show the share of variation each component explains, and the line adds them up. The shaded column marks the elbow, where the curve bends most.
Use the loadings to name a pattern. A loading curve shows how much weight a component gives each hour. A mostly flat curve on one side of zero means the component tracks the day’s overall level; a curve that changes sign contrasts some hours with others, for example morning against evening.
Read scores as days. Each point in the pairplot is one complete day, placed by its scores on two components. Use Colour points by to colour the days by weather, calendar or reference-gas readings and see which of them line up with a component.
Compare PCA with UMAP. PCA keeps the directions along which days differ most overall; UMAP keeps each day close to the days with the most similar profiles. In the UMAP plot, look at which days sit together, and colour them as above.
Trace one day across the plots. Hover over a point, or pick a date under “Inspect a day”: the same day is highlighted in the pairplot and the UMAP plot, and its raw 24-hour profile appears under “Selected day — raw 24-hour profile”. The table “every complete day” lists each day’s scores and summaries.

Inspect projections, loadings and associationssaved PCA and UMAP results, sensor by sensor

Colour points by

Period

Inspect a day

Hover a point, or select a date to inspect the same day across the plots.

Compare the components with covariateshow the correlations are computed

Quantity	How it is computed
Pearson r / Spearman ρ	Correlation of each day’s PC1–PC4 scores with daily weather, the weekend flag, reference-gas averages and the retained-day index; from −1 to +1
Calendar η	How much a component’s scores differ between months or between days of the week; from 0 to 1, without sign
p-value	Uses a lag-1 effective sample size, clipped between 3 and the available number of days
Retained-day index	The day’s position, 0–356, among the 357 complete days in date order

Check the evidencewhat was verified for these pages

Check	Result
Imported source files	All 6 match their recorded SHA-256 values
Daily profiles	All five 357 × 24 matrices match the CSV exactly
Date alignment	All five sensors share the same order of complete days
PCA explained variance	Independently recomputed; agrees within the saved 5-decimal precision
UMAP	The saved coordinates are shown as supplied; UMAP was not refitted

What the analysis tells usseven findings from the saved results

S4’s main pattern follows humidity and temperature

S4’s PC1 explains 77.6% of the variation between its daily profiles. Its scores are correlated with absolute humidity, temperature and retained-day order, while the correlation with the NOx reference is weak (p ≥ 0.05).

S4 PC1: AH +0.747 · T +0.699 · time order -0.652 · NOx -0.146

Benzene and S2 are tightly coupled

For almost every distinct S2 value, the benzene-labelled column C6H6(GT) takes a single value, so knowing S2 almost fixes the benzene reading.

1,078 / 1,245 distinct S2 values map to exactly one benzene value · maximum within-value spread 0.1 µg/m³ · no decreases beyond that rounding step

PC1 explains 61–78% of each sensor’s day-to-day variation

For every sensor, PC1 explains 61.3–77.6% of the variation between daily profiles. The loading curves separate the day’s overall level from contrasts between hours, which PC2 and later components capture.

PC1: S1 65.3% · S2 62.3% · S3 61.3% · S4 77.6% · S5 63.3% · heuristic elbow: PC2 for four sensors, PC3 for S3

The weekly calendar is especially visible in S1 and S3

PC2 is strongly correlated with the weekend flag for S1 and S3, and less strongly for S4.

PC2 / weekend: S1 -0.641 · S3 -0.647 · S4 -0.469 · maximum PC2 / day-of-week η 0.684

The sensors change differently through the year

S4’s PC1 is strongly correlated with retained-day order, S3’s less so, and S1’s hardly at all.

PC1 / retained-day index: S4 -0.652 · S3 -0.369 · S1 -0.023

S1 and S5 follow the CO reference

The PC1 scores of S1 and S5 correlate strongly with the daily CO reference average, and those of S5 also with NOx.

PC1 / CO: S1 +0.818 · S5 +0.856 · S5 PC1 / NOx +0.764

Some columns overlap or cover only part of the year

AH closely follows a transform of T and RH; benzene and S2 are tightly coupled; NMHC readings exist only for an early window. Reference-gas readings are missing on some days, so correlations with them use fewer days.

AH / Magnus transform r = 0.999908 · NMHC: 914 observed hours · CO, NOx and NO₂ jointly missing: 1,226 hours

Glossary25 terms, in the context of this analysis

The data

Term	Meaning here
Metal-oxide (MOX) sensor	A heated semiconducting oxide whose electrical response changes with its environment. The five PT08.S1–S5 columns are these sensors’ responses, not gas concentrations.
Reference analyser	An instrument that reports a gas concentration for comparison with the sensors; its columns are labelled (GT) in this file.
−200	The code the file uses for a missing reading. The hour’s row is still present; the analysis replaces −200 with a missing value before computing means, distances or correlations.
Cross-sensitivity	Response to gases or environmental conditions other than the gas a sensor is designed to target (its nominal target).
Drift	A change over time in a sensor’s response to the same conditions.

The design matrix

Term	Meaning here
Design matrix	The input table for a model: rows are observations and columns are features. Here it has 357 dates × 24 hourly responses, separately for each sensor.
Complete day (day segment)	A calendar date with all 24 hourly readings present for every sensor. The analysis keeps 357 of the file’s 391 dates; the retained-day index numbers them 0–356 in date order.
Shared row index	Row i refers to the same date in all five sensor matrices. Each channel has the same 357 complete dates, so their intersection excludes no additional dates.
Mean-centring	Subtracting each hourly column’s mean before PCA. The columns are not rescaled by their standard deviation, and days are not normalised.

Dimensionality reduction

Term	Meaning here
PCA	Principal component analysis: rotates the data onto new axes ordered by how much of the variation each one captures. Here each axis is a weighted combination of the 24 hours.
Principal component (PC)	One of those axes: PC1 captures the most variation, PC2 the most of what remains, and so on. Each sensor has its own fit. The direction of an axis is arbitrary, so the sign of a score, and of its correlations, can be reversed without changing the result.
Score / projection	A day’s coordinate along a component, as shown in the pairplots. Zero indicates the mean of the fitted days along that direction.
Loading	A component’s weight for each hour. A fairly flat curve on one side of zero indicates an overall response level; a curve that changes sign across the day indicates a contrast between hours.
Explained variance ratio	The share of a sensor’s total day-to-day variation captured by one component.
Scree plot	Explained variance plotted against component number. The scree plots here show ten components and their running total.
Elbow / knee	The point where the scree curve bends most. Here it is the component whose point lies farthest from the straight line joining the PC1 and PC10 points.
UMAP	Uniform Manifold Approximation and Projection: a nonlinear method that keeps each point’s nearest neighbours close together. Here each point is a day, placed near the days with the most similar 24-hour profiles.
Fitted once, filtered after	PCA and UMAP are fitted on all 357 dates. Period controls hide points from these fixed fits, preserving coordinates across selections.

Statistics

Term	Meaning here
Pearson r	Linear correlation, from −1 to +1. For a PCA score, its sign follows the arbitrary direction of the component.
Spearman ρ	Correlation of ranks. It captures monotone relationships that need not be straight lines.
Point-biserial correlation	Pearson correlation when one variable is binary. Here the weekend flag is 1 on Saturday or Sunday and 0 otherwise.
Correlation ratio η	The square root of between-group variance divided by total variance. It is unsigned and runs from 0 to 1; used here for month and day of week.
Effective sample size	How many independent observations the days are worth once the similarity of neighbouring days is allowed for: n × (1 − r₁ˣr₁ʸ) / (1 + r₁ˣr₁ʸ), clipped to [3, n], where r₁ˣ and r₁ʸ are the lag-1 autocorrelations of the two series. The p-values use it.
Autocorrelation	Correlation of a series with itself shifted by one or more steps. When neighbouring days are alike, 357 days carry less information than 357 independent observations.
ns	Marks a correlation whose two-sided p-value is 0.05 or more.