Spectroscopy and Chemometrics News Weekly #5+6, 2017


Chemometrics

Non-Destructive Sensor-Based Prediction of Maturity and Optimum Harvest Date of Sweet Cherry Fruit | sensors LINK


IDC unveils its Top 10 Predictions for global Robotics Industry Industry40 Robotics LINK


Spectroscopy

Global Molecular Spectroscopy Market is expected to reach USD 6.712 billion till 2024. htt… LINK!


Near Infrared

Assessing pre-harvest sprouting in cereals using near-infrared spectroscopy-based metabolomics LINK


Rapid screening of commercial extra virgin olive oil products for authenticity: Performance of a handheld NIR device LINK


Hyperspectral

Imec () launches TDI, multispectral and hyperspectral sensors | imaging HSI LINK


Near-infrared hyperspectral imaging of lamination and finishing processes in textile technology LINK


Spectral Imaging

Viavi Solutions and ESPROS Photonics Corporation Debut New Miniaturized Spectral Sensor and Multispectral Sensor LINK


Equipment

Meta-lenses bring benchtop performance to small, hand-held spectrometer – Science Daily LINK



Scan anywhere with Neospectra Spectrometer Case powered by XPNDBLS PhotonicsWest … LINK!


Agriculture

World feed production exceeds 1 billion MT LINK


Chemometric soil analysis on the determination of specific bands for the detection of magnesium & potassium by … LINK


Other

This app uses spectral analysis to analyze objects and their makeup HawkSpex LINK


Research details developments in the multivariate analysis software industry | MVA LINK

“The worlds first ever spectroscopy enabled iPhone!” Check out our video to see it in action: LINK


Investments in AI will triple in 2017. ($47 billion by 2020 per ) CIO CMO | LINK


Some aspects of fetal development have long puzzled scientists, but new molecular technologies are shining a light: https:/… LINK!


CalibrationModel.com

Spectroscopy and Chemometrics News Weekly 3+4, 2017 | Spectroscopy NIRS MVDA… LINK


Spektroskopie und Chemometrie Neuigkeiten Wöchentlich 3+4, 2017 | NIRS Spektroskopie Chemometrie Multivariate LINK


Spettroscopia e Chemiometria Weekly News 3+4, 2017 | NIRS Spettroscopia Chemiometria news LINK


WHITE PAPER: A novel knowledge-based Chemometric Software Framework for quantitative NIRS Calibration Modeling LINK



Improve Accuracy of fast non-destructive NIR Measurements by Optimal Calibration | spectroscopy sensor modeling LINK


NIRS as a secondary method requires extensive calibration on an ongoing basis | foodindustry Digitalization IoT LINK


Services for Optimization of Chemometric Application Methods of Near-Infrared Spectroscopy | Quality Control NIRS LINK


► Timesaving NIRS Calibration ► near-infrared spectroscopy | protein fat moisture sensor measurement scanning LINK





Procedures for NIR calibration – Creation of NIRS spectroscopy calibration curves

Do you know the effect that you prefer to try out their favorite data pretreatments in combination and often try the same wavelength selections based spectra of the visualized?

You try as six to ten combinations until one of them selects his favorite calibration model, to then continue to optimize. Since then suddenly fall to outliers, because it goes in depth, so is familiar with the data, we know now the spectra of numbers of outliers and is familiar with the extreme values.

Now, the focus is on the major components (principal components, Latent Variables, factors) and makes sure not to over-fit and under-fit not to. The whole takes a few hours and finally one is content with the model found.

So what would happen if you all in the beginning tried variants found outliers removed and re-evaluated and compared? The results would be better than that of the previous model choice? One does not try out? Because it is cumbersome and takes hours again?

We have developed a software which simplifies this so that also the number of model variations can be increased as desired. The variants generation is automated with an intelligent control system, as well as the optimization and comparing the models and finally the final selection of the best calibration model.

Our software includes all the usual known data pretreatment methods (data pre-processing) and can combine them useful. Since many Preteatments are directly dependent on the wavelength selection, such as the normalization the determined within a wavelength range of the scaling factors to normalize the spectra so that pretreatments with the wavelength ranges may be combined. So a variety of settings sensible model comes together that are all calculated and optimized. For the automatic selection of the relevant wavelength ranges, different methods are used, which are based on the spectral intensities. Thus, for example, regions with total absorption is not used, and often interfering water bands removed or retained.

Over all the calculated model variations as a summary outlier analysis can be made. Are there any new outliers (hidden outlier) discovered, all previous models can be automatically recalculated, optimized and compared without these outliers.

From this great number of calculated models with the statistical quality reviews (prediction performance) the optimum calibration can now be selected. For this purpose, not simply sorting by the prediction error (prediction error, SEP RMSEP) or the coefficient of determination (coefficient of determination r2), but by several statistical and test values are used jointly toward the final assessment of optimal calibration.

Thus we have created a platform that allows the highly automated work what a man can never do with a commercial software.

We therefore offer the largest number of matched to your application problem modeling calculations and choose the best calibration for you!

This means that our results are faster, more accurate, robust and objective basis (person independent) and quite easy for you to apply.

You have the full control of the models supplied by us, because we provide a clearly structured and detailed blueprint of the complete calibration, with all settings and parameters, with all necessary statistical characteristics and graphics.

Using this blueprint, you can adjust the quantitative calibration model itself in the software you use, understand and compare. You have everything under control form model creation, model validation and model refinement.

Your privacy is very important to us. The NIR data that you briefly provide us for the custom calibration development will remain of course your property. Your NIR data will be deleted after the job with us.

Interested, then do not hesitate to contact us.

NIR Spectroscopy Calibration Report for quantitative predictive models

When you send your quantitative NIR spectra data to our NIR Calibration Model Service, you get a detailed calibration report (calibration protocol) of the found optimal calibration settings, so you are able to see all insights and easily re-build the model in your NIR/Chemometric software.

Here is a part of our calibration report, that exactly describes the data used in the calibration set (CSet), the validation set (VSet) and the test set (TSet). The numbers are the number ids of the spectra in your delivered NIR data file.


The calibration method settings and parameters are
Waveselection : the variable selection or wavenumber selection or wavelength selection
Pretreatments : the spectral data pre-processing
PCs : the number of  Principal Components (PC) or Latent Variables (LV)
Method : the modeling method algorithm used, e.g. PLS

Then the statistical analysis of the PLS model by the different sets (CSet, VSet, Tset).

Calibration Report

Statistical analysis of calibration, validation and test results : 1 Name, 2 Unit, 3 N : number of spectra, 4 N : number of samples, 5 Average spectra count per sample, 6 Reference values, 7 Min, 8 Mean, 9 Median, 10 Max, 11 Standard deviation, 12 Skewness : left (-) or right (+) lack of symmetry, 13 Kurtosis : flat (-) or peaked (+) shape, 14 Model statistics, 15 RPD, 16 R², 17 RMSEC, RMSEP, RMSET : root mean square of prediction errors, 18 SEC, SEP, SET : standard error (bias corrected), 19 Bias, 20 Skewness of prediction errors, 21 Kurtosis of prediction errors, 22 Intercept, 23 Slope, 24 Intercept (reverse), 25 Slope (reverse), 26 Sample Prediction Repeatability Error, 27 Sample Prediction Repeatability Error (of Missing data MSet)

This shows how we deliver the optimal settings. With the statistical values, the NIR model predicted values of all spectra and additional plots you are able to compare with your re-built model to verify that the models perform nearly equally.

Uniqueness


It is easy
1. send your NIR data (we do not collect, share or sell your data)
2. receive your optimal model blueprint
3. build it, validate it, use it

We have a Chemometric software not to do chemometrics,
we have the solution to build an optimal model for your data
so you can get better NIR measurement results.

So we don’t name it a “Chemometric Software”.
It’s a service named, as that what it delivers, a Calibration Model.
It gives you an optimal chemometric model for your NIR data.
That is what you want to achieve.

So don’t bother about Chemometrics and endless helpless possibilities
and spend your time with clicking and waiting for a chemometric software,
when you can get an optimal model for your data as a service!

There is no lock-in.
Because there is no software to install.

There is no black-box.
Because the model is delivered as a detailed and complete blueprint in human readable form.

You stay independent.
Because you can always choose:
- You can still do it as you have done it before.
- You will experience that with the service you will get the better models faster and is inexpensive.

Let’s have a try
please contact us, so we can help you!

How to develop near-infrared spectroscopy calibrations in the 21st Century?


The Problem

Calibration modeling is a complex and very important part of NIR spectroscopy, especially for quantitative analysis. If the model is badly designed the best instrument precision and highest data quality does not help getting good and robust measurement results. And NIR Spectroscopy requires periodically recalibration and validation.


How are NIR models built today?

In a typical usage in industry, a single person is responsible to develop the models (see survey). He or she uses a Chemometric software that has a click-and-wait working process to adjust all the possible settings for the used algorithms in dialogs and wait for calculations and graphics and then to think about the next modeling steps and the time is limited to do so. Do we expect to find the best use-able or optimal model that way? How to develop near-infrared spectroscopy calibrations in the 21st Century?


Our Solution

Why not put all the knowledge a good model builder is using into software and let the machines do the possibilities of calculations and presenting the result? Designing the software that way, that the domain knowledge is built-in, not just only the algorithms for machine learning and make it possible to scale the calculations to multi-core computers and up to cloud servers. Extend the Chemometric Software with the Domain Knowledge and make as much computer power available as needed.

As it was since the beginning

User  → Chemometric Software → one Computer → some results to choose from

==> User’s time needed to click-and-wait for creating results

Our Solution

User → (Domain Knowledge → automatized Chemometric Software) → many Computers → the best models

==> User’s time used to study the best models and reasoning about his product / process

Note that the “Domain Knowledge” here does perfectly support the User’s product and process knowledge to get the things done right and efficient.


Scaling at three layers

  • Knowledge : use the domain knowledge to drive the Chemometric Software
  • Chemometric Software : support many machine learning algorithms and data pre-processings and make it automatic
  • Computer : support multi-core calculations and scale it to the cloud

The hard part in doing this, is of course the aggregation of the needed domain knowledge and transform it into software. The Domain Knowledge for building Chemometric NIR Spectroscopic models is well known and it’s huge and spreads multiple disciplines. Knowledge-driven software for computing helps to find the gold needle in the haystacks. It’s all about scaling that makes it possible. See Proof of Concept.


New possibilities

  • NIR users can get help working more efficient and getting better models.
  • New types of applications for NIR can be discovered.
  • Evaluation of NIR Applications to replace conventional analytical methods.
  • Hopeless calibrations development efforts can be re-started.
  • Higher model accuracy and robustness can be delivered.
  • Automate the experimental data part of your application study.
  • Person independent optimization will show new solutions, because it’s not limited by a single mindset => combining all the aggregated knowledge and its combinations.
  • Software independent optimization will show new solutions, because none of vendor specific limitations and missing algorithms are present => combining all open available algorithms and there permutations.
  • Computing service is included.

Contact us for trial

Your NIR data is modeled by thousands of different useful calibration models and you get the best of them! That was not possible before in such a easy and fast way! See How it works

Proof of Concept

Chemometric software competitions (aka shootouts) are a good way to check algorithms, software and knowledge against all other experts in the field.

Imagine that the prediction results can be produced with any kind of software and newest algorithms.

And we just use PLS right to generate models that can be used on all NIR software systems, because PLS is a quasi standard, supported in all major chemometrics software.

Our software framework reached very good results, got gold (rank #1) and silver (rank #2) during well known international NIR Chemometric software shootouts* so far, the competitions are held bi-annual.

Rank / competitors  Competition / Conference  Year
 #1 / 1  **  Kaji / ANSIG  2014
 #1 / 150  Kaji / ANSIG  2012
 #2 / ???  IDRC / IDRC  2012
The Kaji Competition

A set of NIR spectral data will be available for downloading from the ANISG website and contestants will be asked to find and explain a “best” chemometric model to robustly predict samples of the same type.
A panel will select the three “best” entries based on the predicted results and spectroscopic explanation of the products and attributes of interest. 

http://www.anisg.com.au/the-kaji-competition


The IDRC Competition

The Software Shootout has been a staple of the IDRC. It is a competition amongst participants of the conference that aims at determining the person who developed the best model and obtained the lowest prediction error for a particular problem.
Every IDRC, a new challenge is proposed to participants. The challenge consists of a data set with calibration, test and a validation set.
Participants are given target values for the calibration and test sets but must do their best to develop a model that will predict the validation set as accurately and precisely as possible. Challenges from all sorts of fields of NIRS have been used (agriculture, biomedical, pharmaceutical, soil, …).

IDRC


*) The author was unable to present the results at the conferences, so this ranking was not official but confirmed by the shootout organizers. Thanks go to Benoit Igne, IDRC 2012 shootout organizer and Steve Holroyd, Kaji Competition organizer at ANISG Conference 2012.  

Conclusion

Our chemometric software framework can significantly reduce the time spent for NIR method development and fine optimization. The time saving can be achieved through highly automated experiments and the usage of cloud computing. Calibrations are built and evaluated using automated good practices protocols resulting in useful, precise and robust Calibrations. The high number of experiments enables a deep screening of the solution domain to find the optimum calibration settings, something currently unavailable in standard chemometric software.

**) We were the only participator that got the 4 competition tasks (4-times more than usual) completed in that short time and submitted the fully documented results. After the competition, the information was given, that the data was originated from forages and the constituents were dry matter, organic matter digestibility, protein and ash. Thanks go to Daniel Cozzolino, Kaji 2014 Competition organizer.

NIR Calibration Modeling (Part 2)

( to part 1 )

All the below categories are implemented by using multiple different algorithms and formulas which leads to many different calibrations.

Steps in modeling
  • Data Cleaning – (bad data, missing values, duplicate elimination, spectral quality / intensity / noise, input value typing errors, …)
  • Initial Calibration set up – selection of calibration, validation and test samples
  • Wavelengths selection
  • Data preprocessing, pretreatments
  • Method calculation
  • Choosing the number of Principal Components / Latent Variables
  • Validation of calibration model / Statistics of performance – (accuracy, precision, linearity, repeatability, range, distribution, robustness / stability, sensitivity, simplicity, etc.)
  • Outlier examination and removal


The problem of choosing the optimal number of factors to find the optimum between underfitting and overfitting is solved by having multiple methods and protocols implemented leading to multiple calibrations.

The evaluation and the selection of the best calibration is based on many individual statistical values including the most popular RMSEP, SEP, Bias, SEC, R2 and PCs etc.

Results and Reporting

A detailed calibration report is provided detailing the best available calibration containing all calibration parameter settings and statistics of prediction performance of the calibration set, the validation set and the test set. A visual expression of the calibration is provided with the most importance plots.

Our service works with any quantitative NIR spectra data set in the standard JCAMP-DX format and uses mainly PLS and PCR to be compatible with other chemometric calibration software.

Chemometrics

ChemoMetric

Definitions

  • Chemometrics is the science of extracting information from chemical systems by data-driven means. It is a highly interfacial discipline, using methods frequently employed in core data-analytic disciplines such as multivariate statistics, applied mathematics, and computer science, in order to address problems in chemistry, biochemistry, medicine, biology and chemical engineering. In this way, it mirrors several other interfacial ‘-metrics’ such as psychometrics and econometrics.
    Wikipedia (2012)


  •  Chemometrics is the science of relating measurements made on a chemical system or process to the state of the system via application of mathematical or statistical methods. Chemometric research spans a wide area of different methods which can be applied in chemistry. There are techniques for collecting good data (optimization of experimental parameters, design of experiments, calibration, signal processing) and for getting information from these data (statistics, pattern recognition, modeling, structure-property-relationship estimations). Chemometrics tries to build a bridge between the methods and their application in chemistry.
    The International Chemometrics Society (ICS)


  • Chemometrics is what chemometricians do.
    - Anonymous

  • Chemometrics is the application of mathematical and statistical techniques in chemistry.

  • Chemometrics is the application of mathematical or statistical methods to chemical data.



Links