Java Example Code

Development Description

This example applies only to MRS OpenTSDB.

• Prerequisites

  A datasource connection has been created and bound to a queue on the DLI management console.

  Note

  Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.

• Code implementation

  1. Import dependencies.

     • Maven dependency involved

       <dependency>
         <groupId>org.apache.spark</groupId>
         <artifactId>spark-sql_2.11</artifactId>
         <version>2.3.2</version>
       </dependency>
       

     • Import dependency packages.

       import org.apache.spark.sql.SparkSession;
       

  2. Create a session.

     sparkSession = SparkSession.builder().appName("datasource-opentsdb").getOrCreate();
     

• Connecting to data sources through SQL APIs

  • Create a table to connect to an MRS OpenTSDB data source and set connection parameters.

    sparkSession.sql("create table opentsdb_new_test using opentsdb options('Host'='10.0.0.171:4242','metric'='ctopentsdb','tags'='city,location')");
    

    Note

    For details about the Host, metric, and tags parameters, see Table 1.

  • Insert data.

    sparkSession.sql("insert into opentsdb_new_test values('Penglai', 'abc', '2021-06-30 18:00:00', 30.0)");
    

  • Query data.

    sparkSession.sql("select * from opentsdb_new_test").show();
    

• Submitting a Spark job

  1. Generate a JAR package based on the code file and upload the package to DLI.

  2. In the Spark job editor, select the corresponding dependency module and execute the Spark job.

     Note

     • If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the Module to sys.datasource.opentsdb when you submit a job.

     • If the Spark version is 3.1.1, you do not need to select a module. Configure Spark parameters (--conf).

       spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/opentsdb/*

       spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/opentsdb/*

Complete Example Code

• Maven dependency involved

  <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-sql_2.11</artifactId>
              <version>2.3.2</version>
  </dependency>
  

• Connecting to data sources through SQL APIs

  import org.apache.spark.sql.SparkSession;
  
  public class java_mrs_opentsdb {
  
      private static SparkSession sparkSession = null;
  
      public static void main(String[] args) {
          //create a SparkSession session
          sparkSession = SparkSession.builder().appName("datasource-opentsdb").getOrCreate();
  
          sparkSession.sql("create table opentsdb_new_test using opentsdb options('Host'='10.0.0.171:4242','metric'='ctopentsdb','tags'='city,location')");
  
          //*****************************SQL module***********************************
          sparkSession.sql("insert into opentsdb_new_test values('Penglai', 'abc', '2021-06-30 18:00:00', 30.0)");
          System.out.println("Penglai new timestamp");
          sparkSession.sql("select * from opentsdb_new_test").show();
  
         sparkSession.close();
  
      }
  }
  

last updated: 2024-07-02 00:32