HackerNews
This page shows how to build a Curiosity Workspace using HackerNews, a popular website for sharing and discussing tech news. HackerNews provides an API for accessing its data, such as stories, comments, users and polls. We will use this API as our data source and show how to connect it to the Curiosity graph database. We will also show how to create a custom front-end for querying and exploring the data in the workspace.
Data Source
HackerNews API is a RESTful web service that returns JSON data. It has several endpoints for different types of data, such as:
/v0/topstories: returns a list of IDs of the top stories
/v0/item/{id}: returns an item (story, comment, poll etc.) by its ID
/v0/user/{id}: returns a user by their ID
The API documentation can be found here: https://github.com/HackerNews/API
Data Model
To model the data in the Curiosity graph database, we need to define the entities and relationships that we want to capture from the data source. For this sample workspace, we will use the following entities and relationships:
Story: represents a story posted on HackerNews. It has properties such as id, title, url, score and time.
Comment: represents a comment posted on HackerNews. It has properties such as text and time, and is linked in the graph to a parent Story or a parent Comment.
User: represents a user on HackerNews. It has properties such as id, karma and created (the time they joined).
SubmissionType: represents which type of post this is (AskHN, ShowHN, Job Offers).
Status: represents the status of the post (dead, deleted or placeholder).
Data Connector
To connect the data source to the Curiosity graph database, we need to write some code that can fetch the data from the API and transform it into entities and relationships that match our data model. We will use C# as our programming language and use requests library for making HTTP requests.
The code for the data connector can be found here.
The code performs the following steps:
Fetches the IDs of top stories from /v0/topstories endpoint and finds the latest ID
For each story ID:
Fetches the story details from /v0/item/{id} endpoint
Creates a Story node with properties from story details (or equivalent data type)
Recursively fetches comments from /v0/item/{id} endpoint for each child ID in story details
Fetches author details from /v0/user/{author} endpoint
Creates a User node with properties from author details
Creates the relationships between Stories, Comments, Users, etc..
Inserts all entities and relationships into Curiosity graph database using the Curiosity Library package.
Custom Interface
To create a custom interface for navigating the HackerNews dataset in the workspace, we need to write some code that can communicate with Curiosity API and display results in an interactive way. We will use again C# as our front-end framework and use H5 compiler to translate the front-end into JavaScript.
The code for custom front-end can be found here. The code has the following structure:
App.cs: Register routes and configure the sidebar for the workspace.
Spaces: Main views for navigating stories and users from HackerNews.
Views: Renderers for each data-type included in the dataset.
Please refer to the Custom Interfaces documentation for more information.
Last updated