Curiosity for Developers
  • Overview
  • Getting Started
    • Introduction
    • System Overview
      • Workspace
      • Connectors
      • Front End
    • Requirements
    • Installation
      • Deploying on Windows
        • Download Curiosity Workspace for Windows
      • Deploying on Docker
        • Deploying using Docker Desktop App
        • Docker Hub
      • Deploying on Kubernetes
      • Deploying on OpenShift
      • Configuration
    • Configure your Workspace
    • Connecting to a Workspace
      • Download App
    • Built-in Templates
  • Security
    • Introduction
    • Hosting
    • Encryption
    • Users and Access
      • User Invitations
      • Single Sign-On (SSO)
        • Google Sign-In
        • Microsoft / Azure AD
        • Okta
        • Auth0
    • Permissions Management
    • Auditing
    • Teams management
    • Configuring Backup
      • Restoring a backup
    • Activate a workspace license
  • Data Sources
    • Introduction
    • User Apps
    • Workspace Integrations
    • API Integrations
      • Introduction
      • Data Modeling
      • Writing a Connector
      • Access Control
      • API Tokens
      • API Overview
      • Tips
    • Supported File Types
    • Curiosity CLI
      • Installation
      • Authentication
      • Commands
  • Search
    • Introduction
    • Languages
    • Synonyms
    • Ranking
    • Filters
    • Search Permissions and Access Control
  • Endpoints
    • Introduction
    • Creating an endpoint
    • Calling an endpoint
    • Endpoint Tokens
    • Endpoints API
  • Interfaces
    • Introduction
    • Local Development
    • Deploying a new interface
    • Routing
    • Node Renderers
    • Sidebar
    • Views
  • Artificial Intelligence
    • Introduction
    • Embeddings Search
    • AI Assistant
      • Enabling AI Assistant
    • Large Language Models
      • LLMs Models Configuration
      • Self-Hosted Models
    • Image Search
    • Audio and Video Search
  • Sample Workspaces
    • Introduction
    • HackerNews
    • Aviation Incidents
    • Covid Papers
    • NASA Public Library
    • Suggest a Recipe
  • Basic Concepts
    • Graph database
    • Search Engine
  • Troubleshooting
    • FAQs
      • How long does it take to set up?
      • How does Curiosity keep my data safe?
      • Can we get Curiosity on-premises?
      • Can I connect custom data?
      • How does Workspace pricing work?
      • Which LLM does Curiosity use?
      • What's special about Curiosity?
      • How are access permissions handled?
      • What enterprise tools can I connect?
      • How to access a workspace?
      • How do I hard refresh my browser?
      • How do I report bugs?
      • How do I solve connectivity issues?
      • How do I contact support?
  • Policies
    • Terms of Service
    • Privacy Policy
Powered by GitBook
On this page
  • Data Source
  • Data Model
  • Data Connector
  • Custom Interface
  1. Sample Workspaces

HackerNews

PreviousIntroductionNextAviation Incidents

Last updated 10 months ago

This page shows how to build a Curiosity Workspace using HackerNews, a popular website for sharing and discussing tech news. HackerNews provides an API for accessing its data, such as stories, comments, users and polls. We will use this API as our data source and show how to connect it to the Curiosity graph database. We will also show how to create a custom front-end for querying and exploring the data in the workspace.

Data Source

HackerNews API is a RESTful web service that returns JSON data. It has several endpoints for different types of data, such as:

  • /v0/topstories: returns a list of IDs of the top stories

  • /v0/item/{id}: returns an item (story, comment, poll etc.) by its ID

  • /v0/user/{id}: returns a user by their ID

The API documentation can be found here:

Data Model

To model the data in the Curiosity graph database, we need to define the entities and relationships that we want to capture from the data source. For this sample workspace, we will use the following entities and relationships:

  • Story: represents a story posted on HackerNews. It has properties such as id, title, url, score and time.

  • Comment: represents a comment posted on HackerNews. It has properties such as text and time, and is linked in the graph to a parent Story or a parent Comment.

  • User: represents a user on HackerNews. It has properties such as id, karma and created (the time they joined).

  • SubmissionType: represents which type of post this is (AskHN, ShowHN, Job Offers).

  • Status: represents the status of the post (dead, deleted or placeholder).

Data Connector

To connect the data source to the Curiosity graph database, we need to write some code that can fetch the data from the API and transform it into entities and relationships that match our data model. We will use C# as our programming language and use requests library for making HTTP requests.

The code performs the following steps:

  1. Fetches the IDs of top stories from /v0/topstories endpoint and finds the latest ID

  2. For each story ID:

    • Fetches the story details from /v0/item/{id} endpoint

    • Creates a Story node with properties from story details (or equivalent data type)

    • Recursively fetches comments from /v0/item/{id} endpoint for each child ID in story details

    • Fetches author details from /v0/user/{author} endpoint

    • Creates a User node with properties from author details

    • Creates the relationships between Stories, Comments, Users, etc..

Custom Interface

The code for the data connector can be found .

Inserts all entities and relationships into Curiosity graph database using the .

To create a custom interface for navigating the HackerNews dataset in the workspace, we need to write some code that can communicate with Curiosity API and display results in an interactive way. We will use again C# as our front-end framework and use to translate the front-end into JavaScript.

The code for custom front-end can be found . The code has the following structure:

: Register routes and configure the sidebar for the workspace.

: Main views for navigating stories and users from HackerNews.

: Renderers for each data-type included in the dataset.

Please refer to the for more information.

https://github.com/HackerNews/API
here
Curiosity Library package
H5 compiler
here
App.cs
Spaces
Views
Custom Interfaces documentation