Our AI application SURUS explained

Our Science team has been working hard on developing SURUS, our very own artificial intelligence application. Sounds fancy, but what does that actually mean, an artificial intelligence application? What is SURUS and why are we developing it? I’ll give you some more information on the ins and outs of SURUS.

1. Background & Objective: why did we develop SURUS?

Every day, healthcare professionals use scientific evidence to make decisions, like which dose to give to a patient or which brand of medicine to offer. But the amount of available information is huge, and keeps expanding every day, making it impossible to keep track. That’s why researchers and healthcare professionals use literature reviews. Literature reviews are summaries of existing scientific literature, and give an overview of the current state of the art. Thanks to literature reviews, it’s easier to keep track of all of the available information. Very useful for healthcare professionals in making their decisions, but also for researchers, as they can easily expand on existing research instead of spending all their time trying to keep track of it. 

Unfortunately, conducting a literature review is very labour intensive and takes a lot of time. So we wondered: can we speed up this process, without losing quality? We started working on a model that uses artificial intelligence to automate (parts of) conducting literature research. SURUS was born!

What will SURUS be doing?

Researchers use something called the PICOS framework in conducting their literature reviews. PICOS stands for Patient, Intervention, Comparison, Outcome, and Study Design, and it helps in selecting scientific articles to include and exclude in a literature review. 

Let’s say you’re writing a literature review and you want to know what the current literature says about the effect of insulin on patients with diabetes. Your PICOS could look like this:

Using these criteria, you can search through your database, selecting all relevant articles and excluding the irrelevant ones. SURUS also uses PICOS to search through a database, just like a human researcher would. Except SURUS can do this in a matter of minutes, instead of weeks or even months it takes a researcher to manually do the same job. 

Is SURUS taking over a researcher’s job? No, it’s just a very useful tool to immensely speed up one part of it: the literature selection. Now researchers can immediately start with the fun part: analysing the literature!

1. Annotation & Dataset

How do we teach SURUS what to do? SURUS uses artificial intelligence, which means that we can train him. We do this by showing examples of what we want him to do. In our case: reading scientific texts, and using PICOS to determine whether or not they should be included in the literature review. So we want to write a literature review about diabetes? Then SURUS should select articles about diabetes, but not about rheumatism. 

So we show SURUS a database containing lots and lots of articles, where we manually labelled the contents. We use a technique called Named Entity Recognition to compose labels in seven categories, which are shown below.

SURUS then analyses these labelled texts and tries to identify patterns. These patterns can then be applied to new, unlabeled, texts. It’s like an exam for SURUS: we showed him the course materials, and now we want to know if he can fill in the blanks. We then give SURUS feedback on his exam, so he can learn from his mistakes, and perform better the next time he encounters an unlabelled text.

1. The SURUS model: what’s inside of SURUS?

Maybe this all still sounds a bit vague to you: artificial intelligence model, pattern analysis, labelling techniques… In previous blog posts, we already explained a little more about artificial intelligence, named entity recognition, and semantic dependency parsing, some of the technologies we use for SURUS. 

To give you some more insight into the engine behind SURUS: SURUS actually consists of two separate models. We’ve optimised these models for our specific objective, and combined them to work together. The models are based on an existing model called BERT, each with its own focus (Named Entity Recognition and segment classification). Together, they make sure SURUS knows which labels can be found in the texts in our database, giving SURUS all of the information needed to automate literature selection!

1. Results: how well does SURUS perform?

We are constantly working on developing and improving SURUS. This means we are also always keeping an eye on SURUS’ performance. To measure SURUS’ performance, we look at the F1-score, a measure of accuracy. By calculating the F1-score, we can gain insight into how accurately SURUS can predict the different labels within scientific articles.

The F1 score is expressed as a number between 0 and 1, where 0 is bad and 1 is good. SURUS has an F1 score of 0.93 for the NER model, and 0.91 for the segmentation model!

1. Conclusion: what does the future look like for SURUS?

SURUS is already pretty good at predicting labels in scientific articles. That means we can use SURUS for automating PICOS-based literature selection! To put this to the test, we’ve compared SURUS’ performance to that of human researchers, by looking at an example research question. Experienced researchers and SURUS used the same research question to search for relevant articles in a scientific database, using PICOS.

Results showed that SURUS excluded 89.4% of irrelevant articles, without loss of relevant articles!

Our next step is to develop and design an attractive user interface, so that everyone will be able to easily use the SURUS search and selection engine!

Does that mean SURUS is finished? Definitely not. Automating literature selection is just a small portion of what we can do with artificial intelligence and SURUS. That’s why we keep working on exploring new opportunities. 

Have you got a problem that SURUS could help solve? Don’t hesitate to contact us, we’re happy to discuss potential collaborations.

We’ve had the chance to present all of the information above to other researchers, at various scientific conferences. The accompanying poster provides all of the info at a glance, and can be downloaded by clicking the button below.