• 
    

    
    

      99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看

      ?

      TSCL-SQL:Two-Stage Curriculum Learning Framework for Text-to-SQL

      2023-09-22 14:30:46YINFengCHENGLuyi程路易WANGQiuyue王秋月WANGZhijun王志軍DUMingXUBo
      關鍵詞:王志軍路易

      YIN Feng(尹 楓), CHENG Luyi (程路易), WANG Qiuyue(王秋月), WANG Zhijun(王志軍), DU Ming(杜 明), XU Bo(徐 波)

      School of Computer Science and Technology, Donghua University, Shanghai 201620, China

      Abstract:Text-to-SQL is the task of translating a natural language query into a structured query language. Existing text-to-SQL approaches focus on improving the model’s architecture while ignoring the relationship between queries and table schemas and the differences in difficulty between examples in the dataset. To tackle these challenges, a two-stage curriculum learning framework for text-to-SQL(TSCL-SQL) is proposed in this paper. To exploit the relationship between the queries and the table schemas, a schema identification pre-training task is proposed to make the model choose the correct table schema from a set of candidates for a specific query. To leverage the differences in difficulty between examples, curriculum learning is applied to the text-to-SQL task, accompanied by an automatic curriculum learning solution, including a difficulty scorer and a training scheduler. Experiments show that the framework proposed in this paper is effective.

      Key words:text-to-SQL; curriculum learning; semantic parsing

      Introduction

      Text-to-SQL is the task of mapping a natural language query to a structured query language, which enables general users to query relational databases with natural languages. Limited by the scale of the dataset, early work can only complete the task on a single database with a few tables[1]. Recently, the release of the WikiSQL[2]dataset, which consists of more than 20 000 tables and about 80 000 natural language queries, presents a new challenge. The model is required to be generalized to unseen table schemas[2]and different kinds of queries.

      To tackle this challenge, existing text-to-SQL approaches cast the problem as a slot-filling[3]task. Xuetal.[4]utilized a multi-task model to fill the predicted values into a pre-defined grammar template. Heetal.[5]and Lyuetal.[6]further improved the model architecture and achieved better performance. However, the current text-to-SQL models still suffer from two challenges.

      The first challenge is that current approaches do not leverage the differences in difficulty between examples in the dataset. As shown in Fig.1(a), a simple query is related to fewer columns in the table, and the names of all related columns are mentioned in the query. In the simple example, the Winner and Runner-up columns are directly mentioned in the query. A complex query is shown in Fig.1(b). It is related to more columns, and some of the columns’ names are not mentioned in the query. In this complex example, the query is related to the Goals, Matches, Average and Team columns. However, the Team column is not mentioned in the query. The model must infer the column name from potential cell values. It makes sense that we can use the differences in difficulty to guide the training process.

      The second challenge is that current approaches do not utilize the relationship between queries and table schemas. As shown in Fig.1, a column name might be mentioned directly or indirectly in the query. The model is required to ground these potential mentions to the table schema. However, existing methods only consider the query’s corresponding table schema, which makes it difficult for the model to learn query-schema alignment.

      To address these shortcomings, a two-stage curriculum learning framework for text-to-SQL is proposed. Specifically, to leverage the differences in difficulty between examples, curriculum learning[7]is applied to the text-to-SQL task and an automatic curriculum learning solution is designed, including a difficulty scorer and a training scheduler. To exploit the relationship between queries and table schemas, a schema identification pre-training task is proposed to make the model choose the correct table schema from a set of candidates for a specific query. Experiments, including comprehensive ablation studies conducted on the WikiSQL dataset would demonstrate the effectiveness of the proposed method.

      Fig.1 Examples of text-to-SQL task:(a) simple example; (b) complex example

      1 Framework Overview

      In this section, the text-to-SQL problem is formulated and the two-stage curriculum learning framework for the text-to-SQL problem is introduced.

      1.1 Problem formulation

      Given a natural language queryQand a table schemaS=, the text-to-SQL task aims to output the corresponding SQL query. The table schema consists of the names of the columnsC={c1,c2, ,cn} and their corresponding typesT={t1,t2, ,tn}.

      1.2 Overall architecture

      As shown in Fig.2, the TSCL-SQL framework split the training process of the text-to-SQL task into two stages. Firstly, the query-schema alignment model was built at the pre-training stage. Specifically, a schema identification task was designed to retrieve the table schema for a specific natural language query. Based on the cross-encoder[8]with an in-batch negative[9]sampling strategy, the model chose the most similar table schema from the candidates for a specific query. Secondly, the curriculum learning was adopted, and the training process of the text-to-SQL task was re-designed with a difficulty scorer and a training scheduler at the curriculum learning stage. The difficulty scorer scored the difficulty of each training sample. The training scheduler organized training samples according to the score, from simple to complex, and split them into buckets to guide the optimization process.

      Fig.2 TSCL-SQL framework:(a) pre-training stage; (b) curriculum learning stage

      2 Pre-training Stage

      The objective of the pre-training stage is to enhance the encoder for the text-to-SQL task by establishing a strong alignment between the natural language query and the table schema. In order to build the alignment, a novel schema identification task is proposed to retrieve the relevant table schema for a given query. To facilitate this task, a dataset specifically designed for schema identification is constructed based on the WikiSQL dataset. The schema identification task is completed using a cross-encoder approach with an in-batch negative sampling strategy, effectively leveraging the power of the model to accurately identify and match query-table schema pairs.

      2.1 Dataset construction

      As shown in Table 1, the dataset mainly consists of the query and the table schema’s meta information. Since Wikipedia is the data source of the WikiSQL dataset, the corresponding table ID, article title, and section title from Wikipedia are concatenated as descriptions for each table schema. Figure 3 shows an example of the data source.

      Table 1 Information of schema identification dataset

      Fig.3 Data source of descriptions for a table schema

      2.2 Query-schema alignment model

      The query-schema alignment model aims to build a better encoder representation for the text-to-SQL task. A retrieval-based schema identification task of selecting the most similar table schema from a set of candidates for the given query is proposed. Figure 4 shows the architecture of the query-schema alignment model. It took the query and the description of the table schema as input and output a score representing the semantic consistency between the query and the table schema. The one with the highest score was chosen as the query’s corresponding table schema.

      sim(Qi,Di)=Linear(red(Encoder

      ([CLS]Qi[SEP]Di[SEP]))),

      (1)

      whereEncoder(·) represents the encoder of a pre-trained language model based on transformers[10];red(·) is a function that takes the representation of the first token from the last layer of the encoder;Linear(·) is a fully connected layer; [CLS] and [SEP] are special tokens.

      Fig.4 Architecture of query-schema alignment model

      (2)

      3 Curriculum Learning Stage

      The curriculum learning stage aims to use a curriculum learning framework to train a text-to-SQL model. A curriculum learning framework for the text-to-SQL task is introduced. Then, the implementation of two core components of the framework is described in detail.

      3.1 Curriculum learning framework

      The curriculum learning framework consists of a difficulty scorer and a training scheduler. The difficulty of each training sample is measured by an automatic difficulty scorer to avoid the uncertainty of hand-crafted rules and consider more about the feedback from the model. The overall process is as follows.

      Firstly, the difficulty scorer scores the samples and sorts them from easy to complex. Secondly, the training scheduler initializes an empty subset of the training set as a training stage starts. Sorted samples are split into buckets. For each training stage, a new bucket is added to the subset according to the difficulty. If the training on the subset is converged, the scheduler moves to the next stage until all the buckets are trained. Finally, the full training set is used for training for another few epochs.

      3.2 Difficulty scorer

      The difficulty scorer aims to score every training sample based on its difficulty. Due to the lack of existing information about which training samples are more difficult, instead of hand-craft rules, the model’s training loss is used as a measurement of the difficulties. A higher loss indicates that the sample is more complex and difficult.

      (3)

      Fig.5 Template used for slot-filling task

      3.3 Training scheduler

      The training scheduler aims to arrange the scored training samples for curriculum learning. As shown in Fig.6, the scheduler first sorts the training samples from easy to difficult and splits them intoNbuckets. Then it starts training with the easiest one. If the training process reaches convergence or a pre-defined number of epochs, a more difficult bucket will be merged. The scheduler will shuffle the data in the bucket and start training. After all the bucket is merged, it will train for several extra epochs on the complete training set.

      Fig.6 Training scheduler

      4 Experiments

      4.1 Dataset and metrics

      The proposed framework is evaluated on the WikiSQL dataset. It consists of tables from Wikipedia, natural language queries and their corresponding SQL statements. The basic characteristic of the WikiSQL dataset is shown in Table 2.

      Table 2 Basic characteristics of WikiSQL dataset

      Specifically, the natural language queries and their corresponding SQL statements are stored with JavaScript object notation. The tables are managed with SQLite database. Figure 7 shows an example of the training set.

      Fig.7 Example of training sample

      In Fig.7, table_id represents the corresponding table of a query; question is the natural language query, sql is the annotated SQL statement; agg and sel represent the column name and the aggregate function of the SELECT statement, respectively; conds are triplets (column-name, operator, value) of the WHERE statement.

      Logic form accuracyAccland execution accuracyAcceare used to evaluate the performance. Logic form accuracy considers whether the predicted SQL statement matches the ground truth. Execution accuracy considers if the execution result of the predicted SQL statement is the same as the execution result of the ground truth one. The formulas are as follows.

      (4)

      (5)

      (6)

      (7)

      whereNdenotes the size of a specific split of the dataset;SQL′ andSQLdenote the predicted SQL statement and the ground truth one, respectively;Y′ andYrepresent the execution result of the predicted SQL statement and the ground truth one, respectively.

      4.2 Execution-guided decoding

      Execution-guided decoding[11]uses beam search to expand the search space of candidate SQL statements and uses the execution result to filter the candidates. The main idea of execution-guided decoding is as follows. If the execution result of the predicted SQL statement is empty or the SQL parser cannot parse the statement, it is believed that the predicted SQL statement is wrong and another SQL statement will be selected from the candidates. In the following experiments, execution-guided decoding is used to further improve the performance.

      4.3 Parameter settings

      All the experiments were conducted on an entry-level server. Hardware and software configurations are shown in Table 3.

      Table 3 Hardware and software configurations

      Due to the limitation of the hardware, the implementation is based on RoBERTabase[13]. At the pre-training stage, the encoder was trained for three epochs. The initial learning rate was 3×10-6. At the curriculum learning stage, the model was first trained on the full training set for two epochs to get the difficulty scorer. Then the scored training samples were split into four buckets. After that the model was trained for three epochs for each training stage until all buckets were trained. Finally, the model was further trained on the full training set until converge.

      4.4 Baselines

      In order to establish a comprehensive performance comparison, multiple baseline methods for the text-to-SQL task are implemented and evaluated.

      1) HydraNet:HydraNet[6]proposes a hybrid ranking network, which casts the text-to-SQL problem as a column-wise ranking and decoding problem. It uses RoBERTa[12]as the encoder.

      2) HydraNet+Pt:the query-schema alignment pre-training method is implemented on that of the original HydraNet.

      3) HydraNet+CL:curriculum learning is applied to the original HydraNet.

      4) TSCL-SQL:the proposed method utilizes both query-schema alignment pre-training and curriculum learning.

      4.5 Results and analyses

      The results are shown in Tables 4 and 5, which demonstrate the framework’s performance under two scenarios, namely with execution-guided decoding (EG) and without EG.

      1) When EG is not applied, the logic form accuracy and the execution accuracy of the re-produced HydraNet model on the test set are 80.8% and 86.4%, respectively. The proposed model, TSCL-SQL, improves performance by 1.5% and 1.4%, respectively.

      2) When EG is applied, although the execution accuracy on the test set is already 91.0%, TSCL-SQL still improves the logic form accuracy and the execution accuracy by 0.9% and 0.5%, respectively.

      3) Ablation studies are conducted to investigate the effectiveness of the pre-training stage and the curriculum learning stage. If the pre-training stage is removed, the logic form accuracy and the execution accuracy will drop 0.5% and 0.6%, respectively, on the test set when EG is not applied. When EG is applied, there is still a slight decrease on the logic form and the execution accuracy if the pre-training stage is removed. It demonstrates that the pre-training stage would help the model initialize a better representation.

      Table 4 Performance of TSCL-SQL framework without EG

      Table 5 Performance of TSCL-SQL framework with EG

      Tables 6 and 7 show the performance comparison on all sub-tasks. TSCL-SQL achieves a performance improvement of 0.4% on the S-AGG sub-task compared to the baseline on the test set. On the other sub-tasks, the performance is still comparable. Therefore, TSCL-SQL is effective.

      Table 6 Development accuracy and test accuracy of various sub-tasks on Wiki SQL dataset without EG

      Table 7 Development accuracy and test accuracy of various sub-tasks on Wiki SQL dataset with EG

      Through analysis, it is found that both the pre-training stage and the curriculum learning stage are important. The pre-training stage provides a better representation for downstream tasks. The curriculum learning stage lets the model learn from easy tasks to complex tasks. It is beneficial for the model to approach the global minimum gradually and smoothly.

      5 Conclusions

      In this paper, a two-stage curriculum learning framework for text-to-SQL (TSCL-SQL) is proposed. At the pre-training stage, a schema identification pre-training task is proposed to build an alignment between queries and schemas. At the curriculum learning stage, an automatic curriculum learning solution is proposed for the text-to-SQL task. Experimental results demonstrate the effectiveness of the framework proposed in this paper.

      猜你喜歡
      王志軍路易
      國畫:慕思春雨
      當代作家(2024年3期)2024-06-29 22:18:20
      Effect of interface anisotropy on tilted growth of eutectics:A phase field study
      臭臭的路易
      王志軍 油畫作品
      真情本無語
      故事會(2018年3期)2018-02-07 15:28:22
      每個小孩都能說出天使一樣的句子
      最佳維權(quán)
      最佳維權(quán)
      愛你(2016年19期)2016-04-12 07:25:00
      對腎病患者的臨終關懷(短篇小說)
      南方文學(2015年3期)2015-07-15 08:03:15
      Numerical Simulation on New Perforator
      龙游县| 芷江| 红原县| 金秀| 瑞金市| 海门市| 平邑县| 兴山县| 唐海县| 天台县| 枣阳市| 招远市| 都江堰市| 陈巴尔虎旗| 西昌市| 辉县市| 兴义市| 白朗县| 石阡县| 通化市| 姜堰市| 磐石市| 武夷山市| 徐州市| 常山县| 凭祥市| 南充市| 南丹县| 黄冈市| 邹城市| 密云县| 台中县| 武汉市| 蓬莱市| 沙洋县| 富源县| 新巴尔虎左旗| 卢湾区| 北碚区| 清涧县| 潞西市|