Skip to content

hackernews-reader.mdx

Latest commit

 

History

History
101 lines (75 loc) · 2.37 KB

hackernews-reader.mdx

File metadata and controls

101 lines (75 loc) · 2.37 KB
sidebar_position 52
title Create a Hackernews Reader
description Data pipeline that periodically reads articles from Hackernews and publishes them on a topic.

Hackernews Reader helps you build an XML reader that ingests hackernews articles, converts them to json, divides them into records, and publishes each record to a topic. This guide uses the following connector:

  • http-source: to read periodically from a hackernews, parse the XML result into json records, and publish the result to a topic.

Prerequisites

Step-by-Step

  1. Create http-source configuration file
  2. Download smartmodules
  3. Start Connector
  4. Check Results

Create http-source configuration file

Create an HTTP source connector configuration file called hackernews.yaml :

:::info All versions are marked with x.y.z. To find the latest version, run:

  • fluvio hub connector list
  • fluvio hub smartmodule list :::
apiVersion: 0.1.0
meta:
  version: x.y.z
  name: hackernews
  type: http-source
  topic: hackernews
http:
  method: GET
  endpoint: 'https://hnrss.org/newest'
  interval: 600s
transforms:
  - uses: infinyon-labs/rss-json@x.y.z
  - uses: infinyon/jolt@x.y.z
    with:
      spec:
      - operation: shift
        spec:
          items: ""
  - uses: infinyon-labs/array-map-json@x.y.z

Download smartmodules

Download the smartmodules used by the connectors to your cluster:

$ fluvio hub sm download infinyon/jolt@x.y.z
$ fluvio hub sm download infinyon-labs/rss-json@x.y.z
$ fluvio hub sm download infinyon-labs/array-map-json@x.y.z

Start Connector

$ fluvio cloud connector create -c hackernews.yaml

Check Results

Connector logs:

$ fluvio cloud connector log hackernews

Records produced:

$ fluvio consume hackernews -T 10

Reference