News collection Airflow pipeline
This app is an Airflow pipeline focused on collecting news. It serves as a personal news collection tool and a learning platform: processing is initially done with a stronger LLM, results are saved, and the plan is to later attempt training a small language model (SLM) on the saved data.
Shared over time
Each entry reflects what was shared on that date. Details may have changed since then.
News collection Airflow pipeline
This app is an Airflow pipeline focused on collecting news. It serves as a personal news collection tool and a learning platform: processing is initially done with a stronger LLM, results are saved, and the plan is to later attempt training a small language model (SLM) on the saved data.
Audience: The author, for personal use and as a learning platform.
Posted by @Havoc · Original comment ↗
“Building my an airflow pipeline focused on collecting news. Partly because I want one, but also because it makes for a good learning platform. e.g. initially do some of the processing with a stronger llm...save result and then attempt to train a SLM or similar”