Stories with Tag data scraping

• Show HN: I scrape Steam data every month and it's yours to download for free

  permalink

  Posted: 2025-02-24 11:43:42

  Verbose tl;dr

  GGInsights offers free monthly dumps of scraped Steam data, including game details, pricing, reviews, and tags. This data is available in various formats like CSV, JSON, and Parquet, designed for easy analysis and use in personal projects, market research, or academic studies. The project aims to provide accessible and up-to-date Steam information to a broad audience.

  A data enthusiast and software engineer, operating under the moniker "GG Insights," has undertaken a significant project involving the monthly scraping and public release of data from the Steam gaming platform. This freely available dataset, accessible via the website gginsights.io, offers a wealth of information regarding games available on Steam, providing potential value to a wide array of individuals, from game developers and market analysts to researchers and curious gamers. The project aims to empower others with comprehensive and up-to-date Steam data, removing the technical hurdles associated with acquiring and processing such information on their own.

  The provided data encompasses various facets of each game listed on Steam, including but not limited to, the game's title, associated tags or genres, pricing details, release date, and the number of reviews it has garnered. This allows for diverse analyses, such as tracking trends in game development, examining the correlation between pricing and popularity, and understanding the overall landscape of the Steam marketplace. The data is meticulously collected on a monthly basis, ensuring a relatively contemporary snapshot of the platform's offerings and mitigating the risk of utilizing outdated information. This regular update cycle facilitates the observation of dynamic changes in the Steam ecosystem, permitting the identification of emerging trends and shifts in consumer preferences.

  The website, gginsights.io, acts as the central repository for this curated data, presenting it in a structured and downloadable format. This simplifies the process of accessing and integrating the information into personal projects, research initiatives, or market analyses. By eliminating the need for individual scraping efforts, GG Insights empowers others to focus on utilizing the data for their specific purposes, be it academic exploration, market research, or personal projects. This initiative effectively democratizes access to valuable Steam data, placing a powerful tool in the hands of anyone interested in exploring the complexities of the digital gaming market.

  • Steam
  • data scraping
  • Data Analysis
  • game data
  • Video Games
  • Dataset
  • Free Data
  • Open Data
  • CSV
  • Download
  • Monthly Updates
  • Gaming Industry
  • Market Research

  Summary of Comments ( 36 )
  https://news.ycombinator.com/item?id=43158425

  Verbose tl;dr

  HN users generally praised the project for its transparency, usefulness, and the public accessibility of the data. Several commenters suggested potential applications for the data, including market analysis, game recommendation systems, and tracking the rise and fall of game popularity. Some offered constructive criticism, suggesting the inclusion of additional data points like regional pricing or historical player counts. One commenter pointed out a minor discrepancy in the reported total number of games. A few users expressed interest in using the data for personal projects. The overall sentiment was positive, with many thanking the creator for sharing their work.

  The Hacker News post "Show HN: I scrape Steam data every month and it's yours to download for free" generated a fair number of comments, mostly focusing on the legality and ethics of scraping, the potential usefulness of the data, and suggestions for the project.

  Several commenters raised concerns about the legality of scraping Steam data, particularly given Steam's terms of service. They pointed out the potential for Steam to take action against the scraping activity or even against users of the data. One commenter suggested checking the robots.txt and respecting rate limits to mitigate some of these risks. Another pointed out the potential legal grey area, noting that court cases regarding scraping have had mixed outcomes.

  The usefulness of the provided data was also a topic of discussion. Some users questioned the value of monthly snapshots, suggesting that more frequent updates would be more beneficial for certain types of analysis, such as tracking game popularity or pricing changes. Others suggested potential use cases, such as identifying trending games or analyzing the effectiveness of marketing strategies. One commenter even proposed integrating the data with existing game discovery tools.

  Many commenters offered constructive feedback and suggestions for the project. These included:

  • Providing more granular data: Suggestions included details on player counts, playtime, and reviews.
  • Offering different data formats: Commenters mentioned the preference for formats like CSV or JSON over the provided Parquet format due to its broader accessibility and ease of use for analysis.
  • Improving data documentation: Users requested clearer documentation on the data schema and included variables.
  • Exploring alternative data sources: One commenter suggested using the publicly available Steam API, though acknowledging its limitations compared to comprehensive scraping.
  • Adding data visualizations: Visualizations of key trends and insights were suggested to enhance the data's usability and appeal.
  • Monetization strategies: While the data is currently offered for free, some commenters offered potential monetization strategies, such as premium tiers with more frequent updates or additional features.

  A few comments expressed appreciation for the project and the free availability of the data, while others questioned the motivation behind the project and the long-term sustainability of providing the data for free. Overall, the discussion highlighted the complex issues surrounding web scraping, the diverse potential applications of readily available data, and the importance of community feedback in shaping data-driven projects.

• League of Legends data scraping the hard and tedious way for fun

  permalink

  Posted: 2025-02-12 11:11:38

  Verbose tl;dr

  The author details their complex and manual process of scraping League of Legends match data, driven by a desire to analyze their own gameplay. Lacking a readily available API for detailed match timelines, they resorted to intercepting and decoding network traffic between the game client and Riot's servers. This involved using a proxy server to capture the WebSocket data, meticulously identifying the relevant JSON messages containing game events, and writing custom parsing scripts in Python. The process was complicated by Riot's obfuscation techniques and frequent changes to the game, requiring ongoing adaptation and reverse-engineering. Ultimately, the author succeeded in extracting the data, but acknowledges the fragility and unsustainability of this method.

  This blog post chronicles the author's intricate journey into the realm of data scraping, specifically targeting information from the popular online game League of Legends. Motivated by a personal desire to analyze game data beyond the limitations of the readily available Riot Games API, the author embarks on a challenging but ultimately rewarding expedition into the depths of web scraping.

  The post begins by outlining the author's initial attempts to extract data using conventional methods like the official API and community-developed tools. Finding these approaches lacking in the specific data points they sought, the author details the pivot towards a more hands-on, and significantly more complex, strategy: directly parsing the HTML structure of the League of Legends website. This approach presented a formidable challenge due to the dynamic nature of the site’s content, which is heavily reliant on JavaScript for loading and displaying information.

  The author meticulously describes the process of reverse-engineering the website's functionality. This involved carefully inspecting network requests, dissecting JavaScript code, and understanding how the game client interacts with the server to fetch and render data. The post highlights the complexity of this undertaking, emphasizing the numerous obstacles encountered, including navigating obfuscated code, dealing with asynchronous loading patterns, and interpreting complex data structures.

  The core of the author’s solution involved leveraging browser automation tools, specifically Selenium and Chromium, to simulate user interaction with the website. This allowed the author to trigger the JavaScript execution necessary to populate the page with the desired data, which could then be extracted by parsing the rendered HTML. The post delves into the specifics of using Selenium, outlining the steps involved in automating navigation to specific match history pages, handling login procedures, and waiting for dynamic content to fully load.

  The author further elaborates on the intricacies of data extraction, detailing the use of regular expressions and other parsing techniques to isolate relevant information from the complex HTML structure. The post acknowledges the fragility of this approach, noting its susceptibility to changes in the website's layout and the potential need for frequent adjustments to the scraping logic.

  Finally, the post concludes with a reflection on the lessons learned and the overall success of the project. While acknowledging the arduous and time-consuming nature of this method, the author emphasizes the valuable experience gained in understanding web technologies and the satisfaction of obtaining the desired data. The post implicitly suggests that this direct scraping approach, while complex, provides a powerful alternative when conventional methods fall short in providing access to specific data points.

  • League of Legends
  • data scraping
  • web scraping
  • game data
  • Riot API
  • Python
  • Tutorial
  • Data Analysis
  • Esports
  • Gaming

  Summary of Comments ( 26 )
  https://news.ycombinator.com/item?id=43024173

  Verbose tl;dr

  HN commenters generally praised the author's dedication and ingenuity in scraping League of Legends data despite the challenges. Several pointed out the inherent difficulty of scraping data from games, especially live service ones like LoL, due to frequent updates and anti-scraping measures. Some suggested alternative approaches like using the official Riot Games API, though the author explained their limitations for his specific needs. Others shared their own experiences and struggles with similar projects, highlighting the common pain points of maintaining scrapers. A few commenters expressed interest in the data itself and potential applications for analysis and research. The overall sentiment was one of appreciation for the author's persistence and the technical details shared.

  The Hacker News post "League of Legends data scraping the hard and tedious way for fun" has generated a modest discussion with a few interesting comments. The comments mostly revolve around alternative approaches to data scraping, specifically for League of Legends, and the challenges faced when relying on unofficial APIs.

  One commenter points out that the Riot API, while official, can be quite limiting and slow. They suggest exploring community-driven projects like the "Champion.gg for Desktop" project, which uses undocumented APIs and has faced its share of challenges with Riot's changes. This commenter highlights the trade-off between using official, albeit limited APIs and venturing into unofficial ones that offer richer data but risk breaking with game updates.

  Another commenter mentions their personal experience with scraping League of Legends data. They specifically mention difficulties encountered when dealing with the dynamic loading of elements on the League of Legends client, making traditional scraping methods tricky. They underscore the complexity involved in keeping up with the constantly evolving structure of the client.

  A third comment provides a direct link to the "Champion.gg for Desktop" GitHub repository mentioned earlier in the discussion. This allows other users to readily explore the project and potentially contribute or learn from its implementation.

  The discussion also briefly touches on the broader topic of web scraping ethics and legality, with one user cautiously mentioning potential terms of service violations. However, this aspect isn't explored in great detail.

  Overall, the comments on the Hacker News post provide valuable insights into the challenges and considerations involved in scraping data from online games like League of Legends. They showcase the trade-offs between utilizing official APIs and resorting to unofficial methods, emphasizing the complexities that arise from dynamic content loading and constant updates from game developers. While not a lengthy or highly active discussion, the existing comments provide practical perspectives and relevant resources for anyone interested in similar data scraping endeavors.