news-scraper-google

A Kotlin/JVM library for fetching Google News RSS headlines and resolving a selected headline into full article content from the original publisher page.

Features

• Fetch top Google News headlines
• Fetch headlines by section such as BUSINESS, TECHNOLOGY, or SPORTS
• Search Google News by keyword
• Resolve Google News wrapper links to the original publisher URL
• Extract full article content, author, and image URL when the publisher page is accessible
• Run both unit tests and a live smoke test against real Google News data

Requirements

• JDK 21
• Gradle wrapper included in this repository
• Internet access for real Google News usage and the live smoke test

Installation

Via JitPack

1. Add JitPack to your repositories:

Gradle Kotlin DSL:

repositories {
    mavenCentral()
    maven(url = "https://jitpack.io")
}

2. Add the dependency:

dependencies {
    implementation("com.github.ashu-choudhury:news-scraper-google:VERSION")
}

Replace VERSION with a Git tag, release tag, or commit hash published from this repository.

Example:

dependencies {
    implementation("com.github.ashu-choudhury:news-scraper-google:1.0.0")
}

Local Build

Build the library locally:

.\gradlew build

The generated artifacts are placed in build/libs/.

JitPack Publishing Notes

• Repository: https://github.com/ashu-choudhury/news-scraper-google
• Group: com.github.ashu-choudhury
• Artifact: news-scraper-google
• Version: use a Git tag, release tag, or commit hash

Quick Start

Fetch top headlines

import com.ashuchoudhury.newsscraper.NewsScraper

fun main() {
    val scraper = NewsScraper()
    val headlines = scraper.getTopHeadlines(
        language = "en",
        country = "US",
        maxResults = 5
    )

    headlines.forEach {
        println(it.title)
        println(it.link)
        println(it.source)
        println()
    }
}

Fetch by section

import com.ashuchoudhury.newsscraper.NewsScraper
import com.ashuchoudhury.newsscraper.model.GoogleNewsSection

fun main() {
    val scraper = NewsScraper()
    val businessNews = scraper.getBySection(
        section = GoogleNewsSection.BUSINESS,
        language = "en",
        country = "US",
        maxResults = 10
    )

    businessNews.forEach { println(it.title) }
}

Search Google News

import com.ashuchoudhury.newsscraper.NewsScraper

fun main() {
    val scraper = NewsScraper()
    val results = scraper.search(
        query = "OpenAI",
        language = "en",
        country = "US",
        maxResults = 10
    )

    results.forEach { println(it.title) }
}

Get full article content from a headline

import com.ashuchoudhury.newsscraper.NewsScraper

fun main() {
    val scraper = NewsScraper()
    val headlines = scraper.getTopHeadlines(maxResults = 5)

    val fullArticle = scraper.getFullNewsItem(headlines.first())

    println(fullArticle.headline.title)
    println(fullArticle.resolvedUrl)
    println(fullArticle.author)
    println(fullArticle.content)
}

Public API

Main entry point

NewsScraper

• getTopHeadlines(language, country, maxResults)
• getBySection(section, language, country, maxResults)
• search(query, language, country, maxResults)
• getNews(request)
• getFullNewsItem(newsItem)

Models

NewsItem

• title
• link
• source
• description
• publishedAt

FullNewsItem

• headline
• resolvedUrl
• content
• author
• imageUrl

NewsRequest

• language
• country
• section
• query
• maxResults

GoogleNewsSection

• WORLD
• NATION
• BUSINESS
• TECHNOLOGY
• ENTERTAINMENT
• SPORTS
• SCIENCE
• HEALTH

Live Testing With Real Google News

This repository includes a real smoke test that uses live Google News and real publisher pages.

Run it with:

.\gradlew runRealSmokeTest --console plain

What it does:

• Fetches live top headlines from Google News
• Fetches live section headlines
• Attempts full-content extraction on real publisher pages
• Prints both success cases and publisher blocking failures

Testing

Run unit tests:

.\gradlew test --console plain

Run full build:

.\gradlew build --console plain

Important Limitations

• Google News RSS links often point to Google wrapper URLs, so this library resolves them before loading the publisher page.
• Some publishers block scraping or return 403, 401, or paywall/interstitial pages.
• Full-content extraction is best-effort and depends on the HTML structure of the destination publisher.
• The live smoke test may succeed for one publisher and fail for another on the same day.
• Google News feed terms and publisher site terms should be respected by library users.

Project Structure

src/main/kotlin/com/ashuchoudhury/newsscraper/
  api/
  core/
  exception/
  model/
  parser/
  tools/

src/test/kotlin/com/ashuchoudhury/newsscraper/

Development

Developer setup and maintenance notes are in DEVELOPER.md.

Contribution guidelines are in CONTRIBUTING.md.

Security reporting guidance is in SECURITY.md.

License

This repository includes an MIT license in LICENSE.