3.1. Graph Convolutional Networks

A graph convolutional network (GCN) is a deep learning model designed to operate on graph-structured data rather than Euclidean data such as sequences. The structure of a GCN enables pattern recognition in non-Euclidean data, which consist of nodes, their features, and the edges that define their interconnectivity. In this study, the convolution layer proposed in ¹⁹ is employed, as shown in Eq. (1).

The expression performs graph convolution over \(l\) layers, starting from \(H^{(0)}\) which represents the initial node features. Let \(l\) denote a convolution layer, and \(H^{\left(l\right)}\in\mathbb{R}^{N\times D}\) represents the activation matrix at the \(l\)-th layer, where \(N\) is the number of nodes and \(D\) is the feature dimension. A graph exhibited as an adjacency matrix \(A\) is modified by adding the identity matrix to obtain \(\tilde{A}\), where the adjacency matrix \(A\) resembles \(\tilde{A}\), yet contains a self-loop per node. The degree matrix \(D\), defined as \({\tilde{D}}_{ii}= \sum_{j}{\tilde{A}}_{ij}\), is used to normalize the adjacency structure and prevent nodes with many neighbours from dominating the aggregated features. The trainable weight matrix \(W^{\left(l\right)}\) was optimized using back propagation during training. The ELU activation function, denoted as \(\sigma(\cdot)\), is employed to introduce non-linearity into the model.

3.2. Recurrent Neural Networks

RNNs are a class of deep-learning algorithms designed to enable neural networks to learn from sequential data, such as text, speech, and images, by retaining internal memory of previous inputs ²⁰. The key feature of RNNs is their ability to process information across time steps using recurrent connections. This allows the network to maintain and update its internal state based on the new information encountered in subsequent time steps ²¹. The architecture of an RNN module, which can be conceptualized as folding and unfolding over time, is shown in Fig. 1.

At each time step, RNNs receive an input, denoted as \(x_t\), which is then processed by the hidden state \(h_t\) using a weight matrix \(W_x\). The recurrent nature of RNNs is evident in the update of the hidden state, where the current hidden state is influenced by the previous hidden state, \(h_{t-1}\), through a recurrence relation, as described in Eq. (2). The function \(f\) represents a nonlinear activation function, such as tanh or ReLU, and \(W_h\) is the weight matrix associated with the hidden state, while \(b\) denotes the bias term. After updating the hidden state, the RNN generates an output at each time step, as shown in Eq. (3), where \(y_t\) represents the output at time \(t\), \(W_y\) is the weight matrix for the output layer, and \(c\) is the corresponding bias term.

3.3. Long Short-Term Memory

Long short-term memory (LSTM) is a variant of the RNN algorithm designed to address problems encountered by vanilla RNNs when learning long-term dependencies, particularly the issues of vanishing and exploding gradients. The LSTM relies on the gating mechanism consisting of three gates: the input gate, forget gate, and output gate, while preserving the recurrent nature through the propagation of hidden and cell states across time steps ²². The LSTM determines which information to retain or discard using the sigmoid neural net layer and element-wise multiplication operations at each gate. An illustration of the LSTM architecture is shown in Fig. 2. The forget gate in Eq. (4) selects the information to be discarded. The input gates in Eqs. (5) and (6) determine which information is updated and added to the cell state. The dynamics of the cell state are given by Eq. (7). The output gate in Eqs. (8) and (9) generates the output based on the previous hidden state and the current cell state ²¹.

Here, \(f_t\) is the forget gate at time step \(t\), \(h_{t-1}\) is the hidden state from the preceding time step, \(x_t\) is the input at the current time step, \(\sigma\) represents the sigmoid activation function, \(i_t\) is the input gate, \(\hat{C}_t\) is the candidate cell state, \(C_t\) is the actual cell state, \(C_{t-1}\) is the cell state from the previous time step, and \(o_t\) is the output gate. \(W_f\), \(W_i\), \(W_c\), and \(W_o\) are the weight matrices associated with each gate, while \(b_f\), \(b_i\), \(b_c\), and \(b_o\) are the bias terms for the respective gates.

3.4. Gated Recurrent Unit

The gated recurrent unit (GRU) is an RNN variant that reduces computational complexity by replacing LSTM’s forget gate, input gate, and cell state with the update gate and reset gate ²³. The gate controls the amount of information that must be discarded. The GRU manages the absence of a cell state using an update gate that controls the memory retention in the hidden state ²⁴. An illustration of the GRU architecture is shown in Fig. 3.

As shown in Eqs. (10) and (11), the reset gate \(r_t\) and update gate \(z_t\) are functionally analogous to the forget and input gates in LSTM, respectively. The hidden state of the GRU is computed as shown in Eq. (12), where \(\tilde{h}_t\) represents the candidate activation, along \(h_t\) defined in Eq. (13).

4.1. Dataset

In line with the model development in this study, the model predicts four roles based on various benchmark datasets: MultiArith ²⁵, AQuA ²⁶, AddSub ²⁷, BigBench Date Understanding (BDU) ²⁸, StrategyQA ²⁹, CommonsenseQA (CsQA) ³⁰, Big-Bench Tracking-Shuffled-Objects (BTSO) ²⁸, and MedQA ³¹. In addition, the ontology data from OccO were extracted using SPARQL queries, as OccO is represented in the web ontology language format. The extracted datasets comprised definitions of roles, skills, and abilities. These components were consolidated into a unified dataset yielding 13 data points (role definition, six skills, and six abilities) for each role derived from the ontology. Table 1 presents a detailed breakdown of the dataset quantities, while Table 2 illustrates the augmentation of the extracted data. The skills and abilities were unified by concatenating their names with their definitions using a space character. For instance, a skill named “active learning” is followed by a space and a skill definition. This approach was implemented to streamline the subsequent data pre-processing stage.

4.2. Data Preprocessing

Based on the results of the data exploration at this stage, several elements in the data samples were eliminated and data transformation was performed. The elimination process targeted data containing numbers, dates, names, punctuation, and stopwords. This approach ensured that the model focuses on the learning patterns of important words representing relevant roles. Text data transformations, such as case folding and lemmatization, were applied to obtain words with lowercase characters and their root forms, eliminating prefixes and suffixes. The final step of this stage involved data annotation for the four predefined roles in both ontology and non-ontology scenarios. All ontology data were associated with the corresponding labels. Consequently, there were 13 labeled ontology data points per role.

4.3. RNN-Based Models Feature Extraction

At this stage, words in the sample data were transformed into vectors using pre-trained GloVe word embeddings containing 400k vocabulary ³². The specific GloVe model used has a dimensionality of 100. Since the three models developed are RNN-based, an embedding matrix containing the features of each word is required. This embedding matrix enables the model to learn from text data through an embedding layer.

The initial step involved calculating word occurrences in the dataset. Let \(D = \{d_1, d_2,\dots, d_n\}\) be a set of documents, where each document \(d_i\) is a sequence of words \(d_i = \{w_1, w_2,\dots, w_m\}\). The word frequency function \(f(w)\) defined in Eq. (14) computes the number of occurrences of word \((w)\) across the entire dataset. Here, \(d_{i,j}\) denotes the \(i\)-th word in the \(j\)-th document, and the Kronecker delta function used in Eq. (14) is defined in Eq. (15).

The second step involved mapping of words to integer indices, denoted by \(\varphi\): \(W \to N\), where \(W\) is the set of all unique words and \(N\) is the set of integers. The function \(\varphi(w)\) assigns a unique index to each word based on its frequency, starting with the <PAD> token initialized to 0, followed by the vocabulary in the data. The <PAD> token is used to handle documents that have fewer words than the longest document in the dataset.

The third step was the construction of an embedding matrix using word features extracted from GloVe. Let \(E\) denote the embedding matrix with dimensions \(V \times d\), where \(V\) is the vocabulary size, and \(d\) is the embedding dimension. Each row of this matrix corresponds to the feature vector of a word in the vocabulary and is represented in \(\mathbb{R}^d\), denoted as \(e_w\) for word \(w\). Words not represented by a vector in GloVe are assigned a zero vector of dimension \(d\). The embedding for each word \(w\) in the embedding matrix \(E\) is defined as shown in Eq. (16).

The final step was to prepare the data to fit the RNN-based model input, which requires sequential data. Sequential data were obtained by converting the text into a list of indices. The function \(\varphi(w)\) resolved the issue by assigning the same unique index, allowing the resulting indices to match the word embeddings contained in \(E\). The documents were labeled according to their origins, as summarized in Table 1. Consequently, there were four distinct labels across the documents.

4.4. Graph Data Creation

This phase is exclusive to the GCN and GraphConv models, which require input data in the form of graphs. The cleaned text data are converted into graphs through three substages: data feasibility checking, text-to-graph transformation, and embedding assignment, as well as labeling.

Data feasibility checking: Each document in the corpus was checked to determine whether it was one word in length. The purpose of this process is to avoid the existence of a trivial graph, that is, a graph that possesses only one node. A trivial graph cannot represent co-occurrence relationships between words. Thus, we identified a potential issue when removing documents that contained only a single word.

Text-to-graph transformation: We employed this transformation using the text2graphapi state-of-the-art Python API, which streamlines text into a graph transformation. The primary operations involved were the co-occurrence matrix, window technique, and adjacency matrix. These operations were applied to each document to establish the local context relationships of the words within the graph-structured data. The co-occurrence matrix \(M \in \mathbb{R}^{|V|\times|V|}\) for each document in the corpus \(D = \{d_1, d_2,\dots, d_n\}\) contains \(M_{i,j}\) entries, each representing the number of context words \(v_j\) appearing within a window of the target word \(v_i\). Vocabulary denoted as \(D = \{v_1, v_2,\dots, v_m\}\) is a set of unique words in a document. If each document \(d_i = \{w_1, w_2,\dots, w_T\}\) contains a given word \(w_t\) where \(w_t \in V\), the entry \(M_{i,j}\) is defined by Eq. (17). The expression calculates the co-occurrence of context words \(v_j\) occurring within the context window \(C(w_t)\) of a target word \(w_t\). As shown in Eq. (18), it defines the context words within a window of radius \(c\) around a target word to include a set of words up to \(c\) positions to the right and left of \(w_t\), where \(w_t\) in \(1 \leq t \leq T_{j}\). We selected \(c\) as 2 to preserve the local context and bonds.

Once the co-occurrence matrix was obtained, each entry was mapped to \(M\) by \(A_{i,j}\) forming an adjacency matrix \(A\). The ultimate goal is to derive an adjacency matrix to represent graph data in an undirected and unweighted manner. Consequently, the maximum number of occurrences of the context word surrounding the target word was one. An undirected, unweighted graph was selected to ensure that the model focused on learning the local context of the target word bidirectionally, without interference from word-context weighting. The resulting adjacency matrix represents word linkages, with binary values zero or one indicating the presence or absence of a word-to-word relationship.

Word-embedding assignments and labeling: At this stage, the embedding of each word is traced using GloVe. If the embedding of a word is found, a 100-dimensional vector is assigned. Otherwise, zero vector \(z\in\mathbb{R}^{100}\) is assigned. The word-embedding trace is defined in Eq. (20). Suppose \(e\) is the embedding of a word \(w_t\) in a document \(d_i\). The data labeling for the document is similar to that described in the previous section, based on document origin, as shown in Table 1.

4.5. Padding Sequences

RNN-based models require fixed-length inputs. However, the documents in our dataset vary significantly in length. We addressed this discrepancy using sequence padding, a preprocessing step that standardizes all sequences to a uniform length for efficient batch processing. This involves truncating longer sequences or padding shorter sequences with zeros until they match a predetermined size. This value is set according to the maximum document length. Sequences shorter than this threshold were extended by zero-padding, whereas those exceeding it were truncated to the specified length.

4.6. Evaluation

Our study used a multiclass confusion matrix to evaluate the model’s performance in predicting each label, and the receiver operating characteristic-area under the curve (ROC–AUC) was used as an additional metric to assess the model performance across all classification thresholds.

Because our dataset is approximately balanced, we selected macro-averaged precision (Eq. (21)), recall (Eq. (22)), and F1 measure (Eq. (23)). We calculated the classifier’s overall correctness in Eq. (24) to measure accuracy. A true positive (TP) is the number of correctly predicted positive classes. A false positive (FP) occurs when an instance that does not belong to a class is incorrectly predicted as belonging to that class. False negatives (FN) are positive instances incorrectly predicted as other classes, and true negatives (TN) refer to correctly predicted instances of the negative class.

4.7. Experimental Setup

We performed hyperparameter tuning using Optuna. This is a framework for optimizing hyperparameter search. The primary targets are listed in Table 3. They were combined within a specified range during training and evaluated on models trained using a combination of data from benchmark and ontology datasets. Once the best hyperparameter was found, it was used to assess the impact of ontology on the model’s learning process. We designed experiments in which the models were trained using ontology-enriched and ontology-absent data. The learning rate was explored on a logarithmic scale. To automatically stop unpromising trials early during the hyperparameter optimization, we employed a MedianPruner, conducting 100 trials with a maximum of 150 epochs per trial.

We assessed the impact of ontology involvement in the model development by comparing the two scenarios, as shown in Table 4. In the training phase, we set the number of model epochs to 300 based on the results of hyperparameter search results recorded in the log. The categorical cross-entropy (CE) loss function is shown in Eq. (25), where \(K\) represents the total number of classes, \(y_j\) is the true label for \(j\)-th class, and \(\hat{f}_j\left(x\right)\) is the predicted probability for the \(j\)-th class given input \(x\). The Adam optimizer was used to optimize the model weights in the learning process. In addition, the embedding layer of each model was frozen to prevent overfitting.

5.1. Feature Extraction for RNN-Based Model

Following the research workflow, we determined that the maximum sequence length of the dataset from training and testing was 151. Accordingly, the documents shorter than 151 words were padded with zeros in the padding sequences according to the difference in length. We identified 7,526 unique words in both the training and test sets. We used GloVe with a 100-dimensional vector to convert the words into vectors based on their unique global representations. In the GloVe corpus, which contains a vocabulary of 400k words, 6,507 of the 7,526 words are embedded as vectors, while the remaining 1,019 were out-of-vocabulary (OOV). In this case, OOV words were embedded as zero vectors; thus, their vector representations were considered void. Consequently, the embedding matrix has dimensions \(\textrm{7,526} \times 100\), corresponding to 7,526 words, with each word represented by a 100-dimensional vector.

5.2. Graph Data Creation

We analyzed the constructed graph data to gain insights into the distribution of nodes as building blocks of the documents. Furthermore, imbalances in the data distribution may contribute to poor model performance. Based on 3,411 graphs, from the training and test sets, the distribution of nodes with respect to document count, as shown in Fig. 5, is right-skewed. This was typically caused by the relationship mean \(>\) median \(>\) mode, with the scores \(22.57\), 14, and \(0.97\), respectively. The number of nodes within the range of 0 to 151 exhibited a high relative distribution, indicated by a standard deviation score close to the mean, which was \(21.8\). Despite the presence of outliers in Fig. 5, we retained them as they did not significantly impact knowledge acquisition during training. The OOV aspects and the number of available word embeddings were consistent with those described in the previous section.

5.3. Best Configuration

Regarding the hyperparameters tuned using Optuna, the best-performing hyperparameter logs are listed in Table 4 for each documented model. The recorded hyperparameters were tested under two scenarios, resulting in the model performance described in the following subsection.

5.4. Model Convergence and Performance

As shown in Fig. 6, the RNN demonstrated the most rapid convergence among the recurrence-based models, attaining stability at 110 epochs, whereas the LSTM and GRU required 220 and 200 epochs, respectively. The presence of the ontology affected the convergence of the models, as the loss record during training in the ontology-involved training setting fluctuated more than that in the ontology-excluding setting. GRU was the most impervious among the recurrence-based models. However, as in the GNN-based models, the GCN converges at smaller epochs, suggesting a more robust capability because the plots across the training settings are congruent. The performances of the models listed in Table 5 support these phenomena by comparing the performances of the GCN and recurrence-based models. Recurrence-based models, particularly RNN, experienced a significant decrease in performance when the ontology was excluded.

The baseline RNN model achieved an accuracy of \(0.77\) and an AUC of \(0.89\) once the ontology was injected, representing a significant increase from the previous values of \(0.49\) and \(0.87\). Conversely, the LSTM and GRU models demonstrated superior performance, attaining an accuracy of \(0.92\) with respective AUC values of \(0.96\) and \(0.97\). This represents an accuracy increase of 8% and 6% for the LSTM and GRU models, respectively, and an AUC increase of 1% and 2%, respectively. The GCN achieved a lower performance than the GRU in the ontology training setting. Regarding the convergence plot, the GCN attained a stable performance across the training settings by yielding a performance gap of only \(0.01\).

This case also occurred with the false positive rate value against the true positive rate value of the model, as shown in Fig. 7, where the RNN scored the lowest AUC compared to the other models in any setting. From a holistic perspective, ontology-void training yielded models with lower AUC. However, the difference in the AUC based on the training settings was not significant. At a fine-grained level, the classifier’s performance across the four classes is illustrated in Fig. 8. The RNN trained without ontology information was unable to classify samples belonging to the recorder class. The LSTM and GRU perfectly classified the samples belonging to the recorder and nurse classes in both settings. Ultimately, all models encountered similar challenges when predicting mathematician and education teacher classes; however, the GCN performed better than the recurrence-based models.

5.5. Model Complexity

We examined model complexity using multiply-accumulates (MACs) and floating-point operations (FLOPs) to assess the computational cost. The number of hidden and architectural layers affects the complexity of the models, as documented in Table 6. The GRU exhibits the highest complexity, compared with the RNN and LSTM models. This result is attributed to the optimal GRU configuration, which used 181 hidden layers and two architecture layers. Both the RNN and LSTM have the same number of architecture layers, which is one to perform optimally; however, the RNN has more hidden layers than the LSTM. Although the RNN has more hidden layers than the LSTM, the LSTM model is more computationally expensive due to its more complex architecture, which includes gating mechanisms absent in the RNN. The GCN with the best hyperparameter configuration, using 250 hidden units in a single layer, is the most computationally efficient and possesses a simpler architecture than recurrent-based models.