How to develop near-infrared spectroscopy calibrations in the 21st Century? / Wie werden Nahinfrarotspektroskopie Kalibrierungen im 21. Jahrhundert entwickelt?

The Problem

Calibration modeling is a complex and very important part of NIR spectroscopy, especially for quantitative analysis. If the model is badly designed the best instrument precision and highest data quality does not help getting good and robust measurement results. And NIR Spectroscopy requires periodically recalibration and validation.

How are NIR models built today?

In a typical usage in industry, a single person is responsible to develop the models (see survey). He or she uses a Chemometric software that has a click-and-wait working process to adjust all the possible settings for the used algorithms in dialogs and wait for calculations and graphics and then to think about the next modeling steps and the time is limited to do so. Do we expect to find the best use-able or optimal model that way? How to develop near-infrared spectroscopy calibrations in the 21st Century?

Our Solution

Why not put all the knowledge a good model builder is using into software and let the machines do the possibilities of calculations and presenting the result? Designing the software that way, that the domain knowledge is built-in, not just only the algorithms for machine learning and make it possible to scale the calculations to multi-core computers and up to cloud servers. Extend the Chemometric Software with the Domain Knowledge and make as much computer power available as needed.

As it was since the beginning

User  → Chemometric Software → one Computer → some results to choose from

==> User's time needed to click-and-wait for creating results

Our Solution

User → (Domain Knowledge → automatized Chemometric Software) → many Computers → the best models

==> User's time used to study the best models and reasoning about his product / process

Note that the “Domain Knowledge” here does perfectly support the User's product and process knowledge to get the things done right and efficient.

Scaling at three layers

  • Knowledge : use the domain knowledge to drive the Chemometric Software
  • Chemometric Software : support many machine learning algorithms and data pre-processings and make it automatic
  • Computer : support multi-core calculations and scale it to the cloud
The hard part in doing this, is of course the aggregation of the needed domain knowledge and transform it into software. The Domain Knowledge for building Chemometric NIR Spectroscopic models is well known and it's huge and spreads multiple disciplines. Knowledge-driven software for computing helps to find the gold needle in the haystacks. It's all about scaling that makes it possible. See Proof of Concept.

New possibilities

  • NIR users can get help working more efficient and getting better models.
  • New types of applications for NIR can be discovered.
  • Evaluation of NIR Applications to replace conventional analytical methods.
  • Hopeless calibrations development efforts can be re-started.
  • Higher model accuracy and robustness can be delivered.
  • Automate the experimental data part of your application study.
  • Person independent optimization will show new solutions, because it's not limited by a single mindset => combining all the aggregated knowledge and its combinations.
  • Software independent optimization will show new solutions, because none of vendor specific limitations and missing algorithms are present => combining all open available algorithms and there permutations.
  • Computing service is included.

Contact us for trial

Your NIR data is modeled by thousands of different useful calibration models and you get the best of them! That was not possible before in such a easy and fast way! Start Calibrate See How it works

NIR Calibration Modeling (Part 2)NIR Kalibrationsentwicklung (Teil 2)NIR calibrazione Modeling (parte 2)

( to part 1 )

All the below categories are implemented by using multiple different algorithms and formulas which leads to many different calibrations.

Steps in modeling
  • Data Cleaning - (bad data, missing values, duplicate elimination, spectral quality / intensity / noise, input value typing errors, …)
  • Initial Calibration set up - selection of calibration, validation and test samples
  • Wavelengths selection
  • Data preprocessing, pretreatments
  • Method calculation
  • Choosing the number of Principal Components / Latent Variables
  • Validation of calibration model / Statistics of performance - (accuracy, precision, linearity, repeatability, range, distribution, robustness / stability, sensitivity, simplicity, etc.)
  • Outlier examination and removal


The problem of choosing the optimal number of factors to find the optimum between underfitting and overfitting is solved by having multiple methods and protocols implemented leading to multiple calibrations.

The evaluation and the selection of the best calibration is based on many individual statistical values including the most popular RMSEP, SEP, Bias, SEC, R2 and PCs etc.

Results and Reporting

A detailed calibration report is provided detailing the best available calibration containing all calibration parameter settings and statistics of prediction performance of the calibration set, the validation set and the test set. A visual expression of the calibration is provided with the most importance plots.

Our service works with any quantitative NIR spectra data set in the standard JCAMP-DX format and uses mainly PLS and PCR to be compatible with other chemometric calibration software.

( zu Teil 1 )

Alle folgenden Kategorien werden durch die Verwendung mehrerer verschiedener Algorithmen und Formeln umgesetzt, was zu vielen unterschiedlichen Kalibrierungen führt.

Arbeitsschritte bei der Modell Erstellung
  • Daten Bereinigung - (schlechte Daten, fehlende Werte, Duplikateliminierung, spektrale Qualität / Intensität / Rauschen, Eingabewert Tippfehler, ...)
  • Initial Kalibrierung einrichten - Auswahl der Kalibrierungs-, Validierungs- und Test-Sets
  • Wellenlängen Auswahl
  • Datenvorverarbeitung, Datenvorbehandlungen
  • Method Berechnung
  • Die Wahl der Anzahl der Hauptkomponenten / Latente Variablen / Faktoren
  • Validierung des Kalibrierungs Modell / Performance Statistiken - (Genauigkeit, Präzision, Linearität, Wiederholbarkeit, Reichweite, Verteilung, Robustheit / Stabilität, Empfindlichkeit, Einfachheit, etc.)
  • Ausreißer Untersuchung und Beseitigung


Das Problem der Wahl der optimalen Anzahl von Faktoren, um das Optimum zwischen Unterfittung und Überfittung zu finden, wird gelöst durch mehrere implementierte Methoden und Protokollen, was zu mehreren Kalibrierungen führt.

Die Auswertung und die Auswahl der besten Kalibrierung basiert auf vielen einzelnen statistischen Werten, einschließlich der beliebtesten RMSEP, SEP, Bias, SEC, R2 und PCs usw.

Ergebnisse und Berichte

Eine detailliertes Kalibrierprotokoll wird bereitgestellt, das die beste verfügbare Kalibrierung detailliert mit allen Kalibrierparameter Einstellungen und Statistiken der Vorhersage Leistung des Kalibrier-Sets, des Validierungs-Sets und des Test-Sets beinhaltet. Eine visuelle Betrachtung der Kalibrierung wird mit den wichtigsten Grafiken zur Verfügung gestellt.

Unser Service funktioniert mit jedem quantitative NIR-Spektren Daten Satz im Standardformat JCAMP-DX-Format und verwendet hauptsächlich PLS und PCR um kompatibel zu sein mit anderen chemometrischen Kalibrierungssoftwaren.

( to part 1 )

All the below categories are implemented by using multiple different algorithms and formulas which leads to many different calibrations.

Steps in modeling
  • Data Cleaning - (bad data, missing values, duplicate elimination, spectral quality / intensity / noise, input value typing errors, …)
  • Initial Calibration set up - selection of calibration, validation and test samples
  • Wavelengths selection
  • Data preprocessing, pretreatments
  • Method calculation
  • Choosing the number of Principal Components / Latent Variables
  • Validation of calibration model / Statistics of performance - (accuracy, precision, linearity, repeatability, range, distribution, robustness / stability, sensitivity, simplicity, etc.)
  • Outlier examination and removal


The problem of choosing the optimal number of factors to find the optimum between underfitting and overfitting is solved by having multiple methods and protocols implemented leading to multiple calibrations.

The evaluation and the selection of the best calibration is based on many individual statistical values including the most popular RMSEP, SEP, Bias, SEC, R2 and PCs etc.

Results and Reporting

A detailed calibration report is provided detailing the best available calibration containing all calibration parameter settings and statistics of prediction performance of the calibration set, the validation set and the test set. A visual expression of the calibration is provided with the most importance plots.

Our service works with any quantitative NIR spectra data set in the standard JCAMP-DX format and uses mainly PLS and PCR to be compatible with other chemometric calibration software.

Chemometrics / Chemometrik

ChemoMetric

Definitions

  • Chemometrics is the science of extracting information from chemical systems by data-driven means. It is a highly interfacial discipline, using methods frequently employed in core data-analytic disciplines such as multivariate statistics, applied mathematics, and computer science, in order to address problems in chemistry, biochemistry, medicine, biology and chemical engineering. In this way, it mirrors several other interfacial ‘-metrics’ such as psychometrics and econometrics.

    – Wikipedia (2012)
  •  Chemometrics is the science of relating measurements made on a chemical system or process to the state of the system via application of mathematical or statistical methods. Chemometric research spans a wide area of different methods which can be applied in chemistry. There are techniques for collecting good data (optimization of experimental parameters, design of experiments, calibration, signal processing) and for getting information from these data (statistics, pattern recognition, modeling, structure-property-relationship estimations). Chemometrics tries to build a bridge between the methods and their application in chemistry.

    – The International Chemometrics Society (ICS)
  • Chemometrics is what chemometricians do.
    – Anonymous
  • Chemometrics is the application of mathematical and statistical techniques in chemistry.
  • Chemometrics is the application of mathematical or statistical methods to chemical data.

Links

ChemoMetric

Definition

  • Unter Chemometrik oder auch Chemometrie versteht man die chemische Teildisziplin, die sich mit der Anwendung mathematischer und statistischer Methoden beschäftigt, um zum einen in optimaler Weise chemische Verfahren und Experimente zu planen, zu entwickeln oder auszuwählen. Zum anderen kann mit chemometrischen Methoden ein Maximum an chemischen Informationen aus experimentellen Messdaten extrahiert werden. Beispielsweise sind Spektren der Nahinfrarotspektroskopie nur mittels der Chemometrie auswertbar. – Wikipedia (2012)
  • Chemometrik ist das, was Chemometriker tun. – Anonym
  • Chemometrie ist die Anwendung von mathematischen und statistischen Techniken in der Chemie.

Links

Chemometrie Programme / Software